Introduction

Historically, Awarding Bodies¹ and the national regulator² for England monitored standards of qualifications between Awarding Bodies, over time and between cognate qualifications at the same level, and this work continues.  A key reason for conducting such work is to avoid inequalities and inequities which would be created by the existence of easier routes to access further study or jobs. 

Ideally standards are compared in terms of candidates’ performance and in terms of the demand of the qualifications.  When comparing new qualifications there is sometimes a lack of performance evidence to form a robust sample from which generalisable research results can be drawn.  In such cases comparability studies could focus on specification³ content demands. However, studies restricted to one aspect of comparability (whether it be performance or demand) are limited in scope.

One structured approach to comparing demands of qualifications is for experts to rate them on a scale of cognitive demands known as CRAS. CRAS was developed using academic qualifications in several school subjects (Hughes et al, 1998). An issue deriving from its provenance may be that CRAS is not suitable for use with vocational qualifications which are different in nature and purpose to academic qualifications.

   Research question

In the present study the research question is whether CRAS is suitable for use in comparability studies about the assessment tasks and specification content of vocational qualifications.

To investigate this question the CRAS frame of reference was contrasted with frames of reference used in other studies about the assessment tasks and specification content of vocational qualifications.

   CRAS

The five aspects of the CRAS frame of reference given below are taken from Pollitt et al. (2007).

“Complexity: The number of components or operations or ideas and the links between them.” For example, using a single idea is less demanding than synthesising several ideas.

“Resources:The use of data and information.”  For example, using all and only the information provided is less demanding than selecting the appropriate data.

“Abstractness: The extent to which the student deals with ideas rather than concrete objects or phenomena.” For example, work which deals with concrete objects is less demanding than mostly abstract work.

“Task strategy: The extent to which the student devises (or selects) and maintains a strategy for tackling the question.”  For example, when a strategy is provided this is less demanding than when a strategy needs to be devised by the student.

“Response strategy: The extent to which students have to organise their response.”  For example, giving the student a small number of possible responses to choose between is less demanding than them having to organise their own response.

The text in quotation marks is from Pollitt et al. (2007:186)

   Foot Notes

1 Testing agencies.

2 Currently the national regulator of the Awarding Bodies is Ofqual.

3 “A complete description - including optional and mandatory aspects - of the content, assessment arrangements and performance requirements for a qualification. A subject specification forms the basis of a course leading to an award or certificate.” (QCDA, undated)

Arlett, S. (2003) 'A comparability study in VCE Health and Social Care units 3, 4 and 6. A review of the examination requirements and a report on the cross moderation exercise. A study based on the summer 2002 examinations.' Organised by the Assessment and Qualification Alliance on behalf of the Joint Council for General Qualifications. Coles, M. and Matthews, A. (1995) 'Fitness for purpose: a means of comparing qualifications. A report to Sir Ron Dearing to be considered as part of his review of 16 -19 education'. Hughes, S., Pollitt, A. and Ahmed, A., (1998) 'The development of a tool for gauging the demands of GCSE and A-level examination questions'. Paper presented at the British Educational Research Association Annual Conference, The Queen's University of Belfast. Pollitt, A., Ahmed, A. and Crisp, V. (2007) 'The demand of examination syllabuses and question papers', 166-206. In P Newton, J Baird, H Goldstein, H Patrick and P Tymms (Eds.) 'Techniques for monitoring the comparability of examination standards'. QCA: London. QCA (2006) 'Comparability study of assessment practice: Personal license holder qualifications', QCA/06/2709 http://www.ofqual.gov.uk/files/personal_licence_holder_quals_comparability_study.pdf [Accessed September 2009 ] QCDA (undated) Glossary http://testsandexams.qcda.gov.uk/15862.aspx#S [Accessed January 2010]