International comparative studies of competence in VET: Issues of equivalence in test construction.

Author(s):

Leif Christian Lahn(presenting / submitting)Hæge Nore

Conference:

ECER 2015

Network:

02. Vocational Education and Training (VETNET)

Format:

Paper

Session Information

02 SES 03 B, Transitions: VET Skills and Competencies

Paper Session

Time:

2015-09-08

17:15-18:45

Room:

324. [Main]

Chair:

n.n. n.n.

Discussant:

Lorenz Lassnigg

Contribution

For many years competency-based assessment has been recognized as a promising strategy for the development of assessment tools and systems in VET (Wolf, 2002). It favours the use of performance methods to so called “objective” tests (standardized scoring) – the argument being that the former give higher priority to occupational validity compared with curricular validity. However there is little consensus among VET-researchers about the type of procedures that will accord with such an objective (Becker et al, 2010). Given such a framework, a number of strategies have been taken by researchers to propose and design new type of assessment tools. Most of them could be subsumed under the category “assessment for learning” but initiatives have also been taken to develop instruments for large-scale assessment and international comparative studies of competence in VET. In this paper we will discuss to what extent equivalence can be established in these projects. We will provide data from the feasibility study PISA-VET or VET-LSA (Baethge et al., 2009) and the project KOMET (Rauner et al, 2009) both coordinated by German universities. Our focus here is on the construction of tasks and their theoretical and methodological foundation.

In research on measurement across cultures, a number of biases are listed as methodological and practical challenges when constructing instruments for valid international comparisons (Van de Vijver & Leung, 1997). Given that equivalence is the inverse of bias, we could distinguish between different types of equivalences (Church, 2010). Conceptual equivalence is referring to variation in the definition of a construct like competence. Linguistic equivalence is about the accuracy of instrument translations whereas instrument equivalence refers to degree of familiarity with item formats. Sample equivalence is crucial but often difficult to achieve. Administrative equivalence favours provisions for the standardisation of the test situation etc. Item equivalence could here be the opposite of differential item functioning (DIF) which occurs when individuals at about the same level of performance make an unexpected solution to an item possibly due to cultural norms or translation biases. This typology has mainly been used in cross-cultural research on attitudes and with questionnaires as favourite instrument. Since our theoretical and methodological framing differs somewhat from this main track, we will use the distinctions above less rigorously. We will also add that rater equivalence is not often addressed in the psychometric literature mainly since rating is automatic or semi-automatic but has to be seriously considered when ratings involve the use of professional judgment to assess realistic complex tasks.

The aim VET-LSA was to identify comparable occupational profiles and learning outcomes at the end of initial VET. It also proposed guidelines for subsequent phases in the development of an assessment instrument. The item format should be a realistic task, taking place in a computer-simulated work environment. In order to strengthen the international comparison generalized descriptions of occupations and work activities were selected from the US data base O*net and then validated for their relevance to VET and national characteristics in 4 workshops with experts from 8 participating European countries. Competences defined as internal conditions for performance (domains, declarative/procedural knowledge etc) were guiding the construction of items and selection of existing scales.

Compared with the VET-LSA project that actively draws on standard work profiles and psychometric elements from different sources, the KOMET model is a construction based on theoretical perspectives and the use of “expert-trade workshops” (Becker et al.,2010). The output is authentic open task assignments for each trade and protocols for rating based on judgmental consensus and criterion-related guidance. The MECVET-project on which this paper is based, is a piloting of the KOMET-model in Norwegian VET.

Method

Our methodology for test construction was framed by the two models that are the subject for comparison. The VET-LSA was piloted in Norway and seven other countries (Olsen, 2009), and our data from this test development process is partly based on documentary evidence and field notes from participation in the national and international workshops by one of the authors of this paper. The VET-LSA project included vocations from three sectors - industrial/technical, commercial/administrative and care/health. Unfortunately, the pilot terminated before the construction of test tasks mainly due to a lack of conceptual equivalence – a finding to which we will return. The piloting of the KOMET-model outside Germany (China, South Africa, Norway and other countries) was done mainly by translating tasks, criteria, solutions spaces and the procedures for test development into the language of the respective countries. Thus, both the validation of test tasks and the implementation of the instrument were done separately and not in joint work shops like in the VET-LSA project. In Norway, the German material was translated and validated in expert workshops with members of examination boards and VET- teachers in three trades (health care workers, electricians and industrial mechanics). The interaction in these sessions was video-taped and analyzed. The piloting of the instrument was designed as a three-year follow-up study of a cohort (N=150) in the Norwegian sequential-dual model (testing second year in school, first year as apprentice and second year as apprentice) covering the three vocations mentioned above. The administration of the open-ended tasks was combined with the completion of a questionnaire on motivation, work conditions and learning environments. Similar processes took place in all the four countries that participated in the study of the KOMET-model, but there are interesting differences that illustrate the challenges in developing an instrument for international comparisons of VET-competences. In this paper, we will address these in terms of the typology of equivalences introduced above. Our analysis will be based on primary data from the Norwegian implementation (documentary, systematic observation, test results and qualitative interviews) and secondary data from the German and South African KOMET-projects (documentary and test results) supplemented with thematic interviews of national coordinators/researchers in these two countries and documentary data from China. Our focus here will be on the construction of test tasks, assessment criteria, solution-spaces, test administration and the training of raters.

Expected Outcomes

In terms of construct equivalence the VET-LSA concluded that the raters often evaluated the domain-specific tasks as quite abstract although some consensus was achieved concerning their relevance and centrality to cross-national occupational profiles (Baethge & Arends, 2009). The MECVET/KOMET-results from Norway confirmed this international feasibility study by pointing out that construct equivalence was especially low for electricians. Contrary to expectations about cultural and linguistic similarities both construct equivalence and translation (linguistic) equivalence were higher between the German/South African/Chinese test tasks compared with the equivalence of the German/Norwegian material (in technical occupations). Issues of sampling equivalence due to differences of occupational profiles in Germany and Norway were most pronounced for electricians and less so for industrial mechanics. For health care workers (Norwegian designation) their position in the classification of occupations differed, but the construct and linguistic equivalence was high between Germany and Norway. For this group the Norwegian translations of the complex open test tasks received high scores in terms of curricular and occupational validity. The realistic open test task format was reported to generate biases among students/apprentices in China that were used to factual based tests (Rauner et al., 2011), but did not seem to generate specific issues in the other countries. However, a major challenge in the Norwegian pilot proved to be low motivation among students and apprentices which corroborate the county-specific findings from the OECD AHELO-project (Hovdhaugen et al.,2013) that attributed such a pattern to the low prestige of educational testing in Norway. A more serious difference with Germany that jeopardizes administrative equivalence with Norway is the fact the KOMET-instrument were far better integrated in the German VET-systems and its emphasis on “learning areas” than in Norwegian VET where participation in the testing was purely voluntary.

References

Baethge, M & Arends, L (2009): Feasability Study VET-LSA. A comparative analysis of occupational profiles and VET programmes in 8 European countries – International report. Federal Ministry of Education and Research, Germany:Vocational Training Research volume 8, Bielefeld: Bertelsmann Verlag Becker, M.; Fischer, M. & Spöttl, G. (Hrsg.) (2010): Von der Arbeitsanalyse zur Diagnose beruflicher Kompetenzen: Methoden und methodologische Beiträge aus der Berufsbildungsforschung, Frankfurt am Main: Peter Lang Church, A.T. (2010) Measurement Issues in Cross-cultural Research. In Walford, G. et al. (eds) The Sage Handbook of Measurement. Los Angeles: Sage. Hovdhaugen, E., Opheim, V., Sjaastad, J. & Sweetman, R. (2013) AHELO mulighetsstudie. Oppsummering av erfaringene ved å gjennomføre OECDs AHELO mulighetsstudie i Norge (in Norwegian and summary in English). Oslo: NIFU report 11/2013. Olsen, O.J. (2009): Feasability Study VET-LSA. National report from Norway. Department of Sociology, University of Bergen, Bergen May 13th 2009 Rauner, F; Haasler, B; Heinemann, L; Grollmann, P. (2009): Messen beruflicher Kompetenzen, Band 1: Grundlagen und Konzeption des KOMET-Projektes, Berlin LIT Verlag. Rauner, F., Heinemann, L. & Maurer, A. (2011) COMET China: Implementation of learning areas: testing a competence model as a basis for test tasks and learning tasks (pilot project large-scale competence diagnostics in Beijing VET schools and VET colleges). University of Bremen: Working Paler of FG I:BB. van de Vijver, F.J. & Leung, K. (1997) Methods and data analysis for cross-cultural research. Thousand Oaks, CA: Sage. Wolf, A. (2001) Competence-based assessment. In Raven, J. & J. Stephenson (eds) Competence in the learning society. NY: Peter Lang.

Author Information

Leif Christian Lahn (presenting / submitting)

UNIVERSITY OF OSLO

INSTITUTE OF EDUCATION

OSLO

Hæge Nore