Why is it so difficult to assess students’ research reports? A model of the challenges facing teachers and external assessors

Author(s):

Irenka Suto(presenting / submitting)Rita Nadas

Conference:

ECER 2012

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 13 C, Current Issues in Developing Assessments

Parallel Paper Session

Time:

2012-09-21

11:00-12:30

Room:

FCT - Seminario 2

Chair:

Monica Rosén

Contribution

Since for many people, education and workplaces in the 21^st Century have become international, multicultural and inter-connected, new skills are needed for progression (Silva, 2009). Although conceptualisations of ‘21^st Century’ skills are many and varied, most include critical thinking, problem-solving, creativity, communication, collaboration, information literacy, and life skills (ATC21S, 2011). Moreover, self-directed study skills, intellectual curiosity, and independent research skills are also valued highly in academic courses within European and many other universities, where freedom of intellectual endeavor is often a fundamental value.

To support the development of skills of this kind among 16 to 19 year olds, diverse educational approaches have been advocated. Some educationalists prefer to embed skills development within lessons in traditional school subjects, such as mathematics and history. An alternative approach, which is currently popular in the UK, is to provide opportunities for project work with a specific focus on independent research and inquiry. In addition to developing skills, research projects provide students with opportunities to investigate a specialist area of study in greater depth, to cross boundaries with an inter-disciplinary enquiry, or to explore a novel non-school subject such as archaeology or cosmology. The resulting reports are often more substantial than those written within subject-based lessons; some are 5000 words long. When applying to university, students can use project reports to demonstrate motivation for their intended course of study and to distinguish themselves from competing applicants.

This paper has two main objectives. The first is to provide an overview of some major forms of independent research carried out by 16 to 19 year olds in the UK and internationally. We conduct a more detailed comparison of two forms: the Extended Project Qualification, available in the UK, and the Extended Essay that forms a compulsory component of the International Baccalaureate Diploma. The significant differences in their aims, structures, scopes, and assessments are not widely understood. It is important for students and teachers to be conscious of the differences in order to make informed decisions about what is most suitable for them. University admissions tutors and employers also need to understand the differences in order to weigh up the experiences and achievements of applicants fairly.

The paper’s second objective is to explore the concerns that surround the assessment of research projects. Previous research indicates that assessors find it much harder to agree on the merits of long pieces of extended prose than on short responses to examination questions (Black, Suto and Bramley, 2011). To answer the question of why it is so difficult to assess research reports, we present a new model of the challenges facing teachers and external assessors. The model draws upon theory conceptualising assessment accuracy as the product of two main factors: (i) the demands of the assessment task, and (ii) the personal expertise of the assessor (Suto and Nádas, 2008). This application of an established theoretical perspective to a topic in popular discourse aims to clarify where and how the assessment of research projects can be improved.

Method

We begin by reviewing traditional and recent conceptualisations of the skills that are most frequently developed through independent research by 16 to 19 year olds. We draw from an extensive international literature, including a current international research collaboration exploring so-called 21st Century skills (ATC21S, 2012), and skills groupings developed by the Lisbon Council (2007) of the European Union. To illustrate the perspective that many skills are not actually new, we map them onto the cognitive domain of Bloom et al.’s (1956) taxonomy of educational objectives. Similarly, the overview and comparison of alternative research forms draws upon published literature. Additionally, however, curricular materials (including assessment criteria and frameworks) are analysed systematically and recent examples of research activities in British schools and colleges are presented. The dimensions of comparison include: the skills promoted, the extent of topic choice, the languages in which reports may be written, the structure of written outcomes, time requirements, and the assessment approach adopted. Finally, the model of challenges facing project assessors is presented. The contributions of theoretical influences such as Laming’s (2004) theory of human judgement, and the communities of practice discourse (Wenger, 20110) are explained, in addition to assessment accuracy theory (Suto and Nádas, 2008).

Expected Outcomes

We conclude that whilst providing students with commendable opportunities to investigate topics they are passionate about and develop new interests, there are substantial differences in the aims, structures, scopes, and assessments of alternative research forms which are often assumed to be similar. For example, whilst the IB Extended Essay promotes traditional academic writing, the Extended Project Qualification focuses more on project management. The assessment of research poses multiple diverse challenges. Those relating to task demands include: distinguishing skills and process from subject content, the variety and interdisciplinary nature of topics, the complexity of students’ ideas, the high cognitive workload caused by extensive reading, the multiple interpretations that descriptive assessment criteria yield, and holistic judgement. Challenges relating to assessors’ personal expertise include: a lack of project-marking experience, limited training and subject knowledge, and a weak wider community of practice, among internal assessors (teachers) in particular. Although the challenges articulated vary in their impacts, all deserve further investigation. Many generalise to other contexts in which long written reports and essays are assessed, in countries with either external or internal assessment systems. Examples include final school examinations requiring extended prose (Hungarian Ministry of Education, 2012), and coursework and essays written by university students.

References

Assessment and Teaching of 21st Century Skills (ATC21S) (2012) Official website. http://atc21s.org. Black, B., Suto, I. & Bramley, T. (2011) The interrelations of features of questions, mark schemes and examinee responses and their impact upon marker agreement. Assessment in Education: Principles, Policy & Practice, 18:3, 295-318 Bloom, B.S., Engelhart, M. D., Furst, E. J., Hill, W. H. & Krathwohl, D. R. (1956) Taxonomy of educational objectives: the classification of educational goals; Handbook I: Cognitive Domain (London: Longman). Hungarian Ministry of Education (2012). Detailed requirements for the middle and higher level final school examinations in Hungarian language and literature. http://www.oh.gov.hu/letolt/okev/doc/erettsegi_40_2002_201201/magyar_nyelv_es_irodalom_vk.pdf Laming, D. (2004) Human judgement: The eye of the beholder. London: Thomson. Lisbon Council (2007) Skills for the future. Brussels: Lisbon Council. Silva, E. (2009) Measuring skills for 21st-century learning. Phi Delta Kappa, 90 (09), 630-634. Suto, W.M.I. & Nádas, R. (2008) What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers. Research Papers in Education, 23(4), 477-497. Wenger, E. (2011) Communities of Practice [online] Available at: www.ewenger.com Accessed on 26 January 2012.

Author Information

Irenka Suto (presenting / submitting)

Cambridge Assessment

Research Division

Cambridge

Rita Nadas