Research questions, objectives and theoretical framework|

Research questions
This paper is based on my - project, HELVYRD (Competence based assessment for learning within the framework of vocational didactics) is part of the Norwegian research project  MECVET (Measuring Competence in Vocational Education and Training) that will  develop a model-based instrument for measuring vocational competence where the original prototype is the German KOMET (see Rauner et al, 2013). The focus of my study is to compare the judgments of raters that assess the test tasks in MECVET with judgments made in the ratings of  trade examination and written exams of apprentices (electricians) in the Norwegian VET-system. What are the criteria they use, and to what extent are their judgment holistic and have vocational validity?

This study will compare the raters’ judgments within each assessment system and between the systems. The reason for comparing these settings is that they differ in terms high stake (examinations) and low stake (research study) and the type of format (“pen and pencil” essay-based tasks, short-questions/answers and performance tasks). How does holistic and vocation-based judgment vary across these settings?

Why to pick several systems is to look at this in a more holistic way and look at the similarities and differences in each system and related to one trade. How do this affect the raters and the way they assess?  

Theoretical framework

The HELVYRD-project brings together different theoretical frameworks and areas of research. It is addressing issues that are at the core of assessment theory (Black et al, 2000) for example the referencing of competence and criterion based assessment (Wolf, 1995, 2001), the possibility of  holistic judgment, in competence based  VET education (Dobsen, 2010), task-assignment construction for high  level of  vocational validity (Haasler & Erpenbeck, 2008; Johnson, 2008), the institutionalization of model-based assessment systems compared with curriculum-based assessment systems and the combination of  different assessment systems (Baartman et al, 2007, Gipps, 1999).

The German KOMET assessement instrument is based on a model from which are derived 8 dimensions or criteria (functionality, presentation, effectiveness, sustainability, business and work process, environmental, creativity, social acceptability) and for each task assignment a solution space is developed by experts in the trade. Thus the raters are trained to adhere to these guidelines whereas raters in the Norwegian trade examinations are assumed to have a focus on official competence goals, trade-specific criteria and the work contexts of the apprentices. In the study the main focus will be on comparison on the judgmental sources of the raters in these contrasting contexts since this objective is important for a validation of the model based KOMET-instrument in relation to an authentic institution like the regular trade examination. The HELVYRD project will provide data for international comparisons since the KOMET instrument has been piloted in Germany, China, South Africa (Rauner, 2012). In parts of Germany it may be integrated in the VET-system since it accords with recent changes rooted in the Lernfeld-reform (Fischer, 2011). The HELVYRD-project is also relevant for our understanding of competence-based examinations in VET. Although some studies have been done in this field, the research on the judgmental basis of raters is almost non-existant in the international VET-literature (MacLean & Wilson, 2009).

References Baartman, L.K.J. et al (2007) Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational research review, 2, 114-129. Black, P. (2000). Research and the Development of Educational Assessment. Oxford Review of Education. Vol.26, Nos. 3&4, pp407-419. Dobsen, S (2010). Nasjonale og internasjonale utfordringer innen elev- og lærlingvurdering. I Dobsen, S. & Engh, R. r. (2010). Vurdering for læring i fag: Høyskoleforlaget. Fischer, A. (2011) Das Lernfeldkonzept als Forschungsanlass und Diskursthema in der Berufs- und Wirtschaftspädagogik – Leuphana Notizen. bwp@ Spezial 5, 1-16 Gipps, C. (1999). Chapter 10: Socio-cultural aspects on assessment. Review of Research in Education, Vol. 24, s. 355‐392. Haasler, B. & Erpenbeck, J (2008): Assessing vocational competences. In Rauner, F. & MacLean, R. (2008) Handbook of technical and vocational education and training research. Springer Media Johnson, M. (2008) Assessing at the borderline: Judging a vocationally related portfolio holistically. Issues in educational research, 18, 1, 26-44 Maclean, R. & D. Wilson (Eds.) International handbook of education for the changing world of work. Springer Science+Business Media B.V. Rauner, F. et al (2012): Project Report Comet-Pilot Test South Africa, i:BB Universität Bremen Rauner, F. et al (2013). Competence Development and Assessment in TVET (COMET). Theoretical Framwork and Empirical results: Springer Science and Business Media. Wolf, A. (1995). Competence-based assessment. Buckingham: Open University Press. Wolf, A. (2001) Competence-based assessment. In Raven, J. & J. Stephenson (eds) Competence in the learning society. NY: Peter Lang