Thinking about Measurement

Author(s):

Viola Klotz(presenting / submitting)Julia Sangmeister(presenting)Esther Winther

Conference:

ECER 2013

Network:

02. Vocational Education and Training (VETNET)

Format:

Paper

Session Information

02 SES 09 B, Measurement and Assessment

Paper Session

Time:

2013-09-12

11:00-12:30

Room:

A-103

Chair:

Andreas Sebe-Opfermann

Contribution

Measuring in professional settings is (a) focused on the professions, (b) competence-based and (c) authentic, i.e. real world-oriented. In this context the paper deals in particular with (1) an identification of valid competence measures (test question) and (2) an identification of predictive variables (interpretation question).

To answer the test question an assessment model, incl. a computerized test environment for the domain of economic competencies is presented (Winther & Achtenhagen, 2009; Winther, 2010). Such assessment approaches are not only more economic – in terms of data procession and testing time for the testees (when using an adaptive testing approach) – but also do they inhere a natural authenticity. This is in particular the case for the economic domain, where the computer serves as the actual workplace, but also in other vocational areas e. g. for the vocation of mechanics, where computer-based simulations have shown to convey over a high predictive power for real work observations (Nickolaus, Gschwendter, & Abele, 2009; Gschwendter, Abele, & Nickolaus, 2009).

However, a modern modelling of competency does not exclusively concern cognitive aspects but integrates socio-economic factors as well as motivational and volitional components. Therefore in order to respond to the interpretation question we refer to explanatory models for competence development including workplace and learning variables, social parameters and identity conceptions. In this context the integration of vocational identity plays a predominant role as it is considered to strongly influence the development of vocational competencies (Heinz, 1995; Heinemann & Rauner, 2008; Rauner, Heinemann, & Haasler, 2009). First empirical results regarding the explanatory power of vocational identity for vocational competency development will be presented.

Method

To answer the test question we had to describe real workplace situations and tasks (by observation and interviews), the tools used (mainly computers and ERP-software), and personal abilities (by official documents with regard to necessary training and observation). This procedure led to the construction of “authentic” workplace situations and challenges by developing a web-based enterprise (for the criterion “authenticity” and the necessary steps of modeling authenticity (cf. Achtenhagen, 2001) in which the assessment could be “embedded” (cf. Black et al., 2003). To answer the interpretation question we outline an alternative measurement concept for vocational identity. The measurement items are designed with the intention to capture the extent to which an individual’s occupation has become its personal vocation, referring to the theoretical framework of Billett (2011). Based on the instrument derived, a coherent model, comprehending and testing major influencing factors and impacts of vocational identity, is introduced. The model as well as the relevant measurement instrument were tested on an empirical basis of N = 504 German students. In order to assess the influence of vocational identity on competency development, a simultaneous testing of students with a competency test as well as with the identity instrument is planned for April 2013.

Expected Outcomes

The computerized test environment was tested with 265 apprentices in their 3rd year of apprenticeship in seven different commercial schools in three different German States (a follow up is running). The results demonstrate that it is possible by using methods of Item Response Theory to measure validly and reliably professional competencies as well as behaviour settings. The item and person parameters can be drawn on one common logit-scale and also ordered according to a priori defined increasing levels of competencies. The empirical results regarding vocational identity indicate that among all tested variables, practical training proves to be the most powerful influencing factor for the development of vocational identities (r = 0.52**). For the vocational learner this emphasizes the indispensability of practical learning at the workplace. The results further underline the importance of voluntariness when choosing a vocational career for the later process of vocational identity shaping (r = 0.49**). Vocational identity is further confirmed as an important predictor for organisational identity (r = 0.51**) as well as for an individual’s work commitment (r = 0.65**). Concerning the relation between vocational identity and learning only small effects are identified within this study (r = 0.14**), yielding no definite results on this matter. First results regarding the relationship between vocational identity and competency development are expected for June 2013.

References

Achtenhagen, F. (2001). Criteria for the development of complex teaching‐learning environments. Instructional Science, 29, 361-380. Black, P., Harrison, C., Lee, C., Marshall, B., & Wiliam, D. (2003). Assessment for learning: Putting it into practice. Berkshire, England: Open University Press. Billett, S. (2011). Vocational Education: Purposes, traditions and prospects. Dordrecht: Springer. Gschwendtner, T., Abele, S., & Nickolaus, R. (2009). Computersimulierte Arbeitsproben: Eine Validierungsstudie am Beispiel der Fehlerdiagnoseleistungen von Kfz-Mechatronikern. Zeitschrift für Berufs- und Wirtschaftspädagogik, 4, 557-578. Heinemann, L. & Rauner, F. (2008). Identität und Engagement: Konstruktion eines Instruments zur Beschreibung der Entwicklung beruflichen Engagements und beruflicher Identität. A+B Forschungsberichte, 1, 1-24. Heinz, W. R. (1995). Arbeit, Beruf und Lebenslauf. Eine Einführung in die berufliche Sozialisation. Juventa Verlag: Weinheim und München. Nickolaus, R., Gschwendtner, T., & Abele, S. (2009). Die Validität von Simulationsaufgaben am Beispiel der Diagnosekompetenz von Kfz-Mechatronikern. Vorstudie zur Validität von Simulationsaufgaben im Rahmen eines VET-LSA. Rauner, F., Heinemann, L. & Haasler, B. (2009). Messen beruflicher Kompetenz und beruflichen Engagements. A+B Forschungsberichte, 2. Bremen, Heidelberg, Karlsruhe: A+B Forschungsnetzwerk. Winther, E. & Achtenhagen, F. (2009). Simulationsaufgaben als innovatives Testverfahren für Industriekaufleute im Rahmen eines VET-LSA. Wirtschaft und Erziehung, 10, 317-324. Winther, E. (2010). Kompetenzmessung in der beruflichen Bildung. Bielefeld: wbv.

Author Information

Viola Klotz (presenting / submitting)

University of Paderborn

Paderborn

Julia Sangmeister (presenting)

University of Paderborn

Business Administration and Economics

Paderborn

Esther Winther

University of Paderborn, Germany

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.