A Method for Comparing the Demands of Specifications

Author(s):

Jackie Greatorex(presenting / submitting)Sanjana Mehta(presenting)

Conference:

ECER 2012

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 13 C, Current Issues in Developing Assessments

Parallel Paper Session

Time:

2012-09-21

11:00-12:30

Room:

FCT - Seminario 2

Chair:

Monica Rosén

Contribution

Background

The comparability of qualifications receives much attention in the UK. One purpose of awarding bodies is to ensure the comparability of various pathways to further study and work. There are established methods for comparability research (Newton et al., 2007). Greatorex (2011) explains that such research often compares:

· The demands of assessment tasks e.g. examination questions

· The quality of learners’ performance e.g. as illustrated by their responses in scripts

· Prior/concurrent measures of attainment.

Comparability research is usually post hoc. In contrast, this research explores the feasibility of building research into qualification development.

Objectives

· To compare the demands of established units and new units.

· To identify the relevance of affective, cognitive, interpersonal, metacognitive and psychomotor taxonomies to each unit in the specifications.

Theoretical Framework

Some taxonomies, e.g. taxonomy of educational objectives (Bloom, 1956), indicate what is more or less demanding. They are based on extensive research and might offer a robust framework for comparing demand. Therefore, we compared demands using taxonomies which applied to adolescent, adult, academic, work based and practical learning. The taxonomies were:

· Affective (Hauenstein, 1998)

· Cognitive (Hauenstein, 1998)

· Interpersonal (Rackham & Morgan, 1977)

· Metacognitive (Howell & Caros, 2006)

· Psychomotor (Hauenstein, 1998).

Method

Method Units from an established specification were compared with units from one new specification. The specifications were from the same subject and level in the qualifications system. Six experts used each taxonomy to judge which unit in a pair was more demanding and gave a reason for the decision. Each pair comprised an established unit and a new unit. Five or six experts agreeing which unit was more demanding and giving a taxonomy-related reason was indicative of a demands-related difference. The specifications were qualitatively analysed for the presence/absence of demands related to each taxonomy.

Expected Outcomes

Findings Standards reside at qualification level, so differences in unit demands may not mean there are differences at specification level. There were 11 demands-related differences between units. For instance, a new unit was more metacognitively demanding than an established unit and both units had many demands related to the metacognitive taxonomy. The awarding body might consider decreasing the metacognitive demands of the new unit using the experts’ taxonomy-related reasons. Results like this help to target units in development at the same level of demands as established units.

References

References Greatorex, J. (2011) Comparing different types of qualifications and alternative comparator, Research Matters: A Cambridge Assessment Publication. Special Issue 2: Comparability, 34-41. Hauenstein, A.D. (1998). A conceptual framework for educational objectives: A holistic approach to traditional taxonomies. Lanham, MD: University Press of America. Howell, K. & Caros, J. (2006). Taxonomy of Metacognitive activities: Advanced/strategic reading. Available at http://www.wce.wwu.edu/Depts/SPED/Forms/Howell%20-Taxonomy%20%20of%20Strategic%20Reading.pdf accessed on 16 August 2010. Newton, P., Baird, J., Goldstein, H., Patrick, H. and Tymms, P. (Eds.), (2007), Techniques for monitoring the comparability of examination standards. London: Qualifications and Curriculum Authority. Rackham, N. & Morgan, T. (1977). Behaviour Analysis in Training, Maidenhead: McGraw-Hill.

Author Information

Jackie Greatorex (presenting / submitting)

Cambridge Assessment

Research Department

Cambridge

Sanjana Mehta (presenting)