International Comparative Competence Assessment

Author(s):

Lars Heinemann(presenting / submitting)Ursel Hauschildt(presenting)Felix Rauner

Conference:

ECER 2012

Network:

02. Vocational Education and Training (VETNET)

Format:

Paper

Session Information

02 SES 10 D, Competence Assessment: Methodological Issues And Approaches

Parallel Paper Session

Time:

2012-09-20

15:30-17:00

Room:

FCEE - Aula 2.8

Chair:

Lars Heinemann

Contribution

In the last four years, the authors have conducted various national and international projects on large-scale competence assessment (COMET).

COMET measures occupational competence by using complex open test tasks, derived from the world of work. Here, no simple 'yes'- or 'no'- solutions are possible. Rather, different legitimate solutions vary according to criteria like cost-effectiveness, creativity, etc. The project identified 8 of such criteria that are inherent in skilled work: functionality, clearness/representation, efficiency/effectiveness, use value/sustainability, orientation on work and business processes, environmental responsibvility, the solution's creativity, and social responsibility (Rauner et al. 2009a). The eight competence criteria are assigned to levels of competence, which are of fundamental importance in assessing the proper performance of occupational tasks. Developing open test tasks taht are complex enough to adress these criteria, we created a possibility to measure occupational competence in terms of the way meaningful tasks are carried out.

But then, occupational competence is not restricted to the way work tasks are carried out. As vocational learning requires an integration into a community of practice, and consists in a developmental process from novice to expert, the process of identity formation is an intrinsical part of competence development as well. Additionally, in vocational education motivation is not an intervening variable, but a very aim of VET. Consequently, the COMET approach in competence development adresses these issues as well. For methodological reasons, these aspects are covered separately (see below).

The COMET aproach allows to analyse occupational competence of different groups of learners that undergo VET in different settings. Consequently, it has been used for international comparisons. This paper presents findings of projects with more than 2000 learners from Germany, China, and South Africa.

Method

The research instruments consist at a) realistic, open test tasks clearly related to problems of the world of work in a given occupational profile. For large-scale assessment, the tasks solutions have to be rated consistently to be comparable. b) This rating is carried out using a rating sheet, where the criteria of skilled work are broken down into 40 items. Consistence of ratings is secured by c) a rater training. Here, professionals and experts in a usually 2-day training rate sample solutions and discuss and compare their results until a degree in accordance is reached that mets psychometric criteria (Finn-coefficient etc.). Each solution than is rated by two raters. d) To secure the data is in a form to be represented by scales, items and dimensions have been analysed psychometrically (Rasch-scalability, latent-class analysis, etc.). e) Occupational commitment and the development of occupational identity is measured by questionnaires using standardised scales. f) Context data - socio-demographic data (prior learning etc.), data on VET schools (learning climate, collaboration with companies, etc.), and data on in-company training (integration into communities of practice, feedback by trainers, etc.), is surveyed by questionnaires as well.

Expected Outcomes

From the multitude of findings, we will present the ones most interesting for international comparisons: a) occupational validity: texperts in the different countries have found the tasks to cover occupation-specific problems b) validity of scales on occupational commitment/vocational identity: if VET does not aim at an occupational core (or 'Beruf'), items on occupational commitment and vocational identity tend to be less consistent c) test scores and VET system: in relation with the way VET is organised, one finds typical developments from 2nd to 3rd year of apprenticeship according to test scores. Astonishing is the degree of heterogeneity within test groups. d) test scores and context data: the way in-company learning is organised has an effect on how learners are able to solve the open test tasks. e) commitment, identity, and context data: the way in-company learning is organised, has an effect on the development of commitment and identity. All these findings are analysed and described comparing Germany, PR China, and South Africa.

References

L. HEINEMANN, AND F. RAUNER, Engagement und Ausbildungsorganisation. I:BB, Bremen, 2009 F. RAUNER et al., Messen beruflicher Kompetenzen. Grundlagen und Konzeption des KOMET-Projekts, Vol. I, second edition, Berlin, 2009(a) F. RAUNER et al., Messen beruflicher Kompetenzen. Ergebnisse KOMET 2008, Vol. II, second edition, Bildung und Arbeitswelt series, Berlin, 2009(b) F. RAUNER et al., Messen beruflicher Kompetenzen. Ergebnisse KOMET 2009, Vol. III Berlin, 2011

Author Information

Lars Heinemann (presenting / submitting)

Bremen University

IBB

Bremen

Ursel Hauschildt (presenting)

Bremen University, Germany

Felix Rauner