The Contribution Of The Learning outcomes Approach To Improved Quality Assessment Procedures In Vocational And Educational Training

Author(s):

Janine Wulz(presenting / submitting)Monika Auzinger(presenting)Mariya Dzhengozova

Conference:

ECER 2016

Network:

02. Vocational Education and Training (VETNET)

Format:

Paper

Session Information

02 SES 14 B, Competency Based Learning and Assessment in VET

Paper Session

Time:

2016-08-26

15:30-17:00

Room:

Vet-Theatre 115

Chair:

Ludger Deitmer

Contribution

Countries are increasingly using learning outcomes-based approaches with a view to generating trust and to supporting the relevance of qualifications for the labour market and individuals. It is a key element of this approach that learners are assessed against a set of clear reference points, expressed in terms of learning outcomes. Learning outcomes in this context are understood as statements of what a learner knows, understands and is able to do on completion of a learning process.

It is expected that the shift to learning outcomes-based approaches will have an impact on assessment procedures in the medium to long run, supporting the quality and relevance of qualifications. However, current evidence also suggests that learning outcomes so far have limited impact on the ways in which learning is assessed.

This makes it particularly interesting to study how education and training institutions carry out their assessment procedures in practice, and the settings and arrangements in place to ensure that the assessment carried out complies with the set requirements.

The quality assurance of assessment very frequently refers to principles or characteristics such as validity and reliability that have to be achieved. Focusing on ‘practice-oriented assessment’, Stenström et al (2006) have identified the following set of characteristics: validity, reliability, objectivity, transparency, equity, and fairness. Validity in the context of assessment ensures that assessment methods, materials and instruments measure as precisely as possible the intended learning outcomes and that evidence fully supports the assessment. Reliability is about whether the same assessment results can be obtained in different cases (e.g. in relation to context, time, assessors or assessment tasks) (Cedefop, 2015).

The above mentioned increased focus on outcome-oriented approaches stems to a considerable extent from EU-level initiatives, such as EQF and EQAVET. EQAVET is the European reference framework for quality assurance in VET. EQAVET is not a quality assurance system, but rather a meta-framework for quality assurance. It invites countries to promote and monitor continuous improvement in their VET systems, through the use of a quality assurance and improvement cycle based on four phases (Planning, Implementation, Evaluation and Review), which are linked to quality criteria and indicative descriptors. The learning outcomes approach is central to the European Qualifications Framework (EQF), which is a common European reference framework acting as a translation device to make qualifications acquired within the different education and training systems in Europe more readable and understandable.

This paper focuses on the application of the learning outcomes approach in assessment of learning in vocational education and training. The main research question is: "How does the learning outcomes approach contribute to achieving valid, objective and reliable assessment of learning in vocational education and training?"

The objective of the research undertaken is to find out how VET institutions assure the quality of their assessment procedures by applying the learning outcomes approach in practice. The principal aims include: to explore in more detail the different tools used as well as the challenges faced by institutions in order to make their assessment procedures more valid, objective and reliable.

Method

The research undertaken is based on a comprehensive, qualitative methodological approach. Based on the theoretical framework, research was undertaken in two countries (Austria and Spain) to identify the use of the learning outcomes approach in certification procedures in VET. Countries selected differ in their VET systems and in their assessment procedures. While Austria has a long tradition in both, school-based VET and dual apprenticeship training, VET in Spain has a focus on school-based training. The assessment leading to the award of a qualification is based on a final exam in Austria, compared to continuous assessment in Spain. However, both countries are considered "recent developers" of the learning outcomes approach: they have been introducing relevant IVET reforms since 2005, and therefore developed their learning outcomes approach in assessment procedures within a similar time period. Research was undertaken at national level including desk research, document analysis and qualitative interviews based on an open questionnaire with national authorities and experts involved in VET. As a second step researchers carried out case studies focussing on VET institutions to achieve a deep understanding of the use of the learning outcomes approach in assessment procedures. The case studies were both selected in the field of school-based IVET and at the same ISCED level to ensure better comparability. Both studies focus on examination of practical subjects: the final "Reifeprüfung" and diploma examination (AT) analysed at a tourism VET college (area of gastronomy)and the assessment of students´ competences in the “in-company training module” in the health and care sector in Spain. The case studies included on-site visits, participant observation of assessment procedures, in-depth interviews with trainers and teachers, focus groups with learners as well as a document analysis. The main focus was the use of learning outcomes, in particular the application of tools to ensure the reliability, objectivity and validity of the assessment procedure. It included the quality assurance of assessment methods and procedures as well as grading methods. Moreover, selection requirements and training of assessors, information policy about requirements and regulations as well as the provision of clear reference points for assessment, appeal procedures and monitoring processes were observed. As a third step a comparative analysis of the case studies was undertaken to learn about the similarities and differences to apply in practice and the learning outcomes approach used in order to assure better quality of their assessment procedures.

Expected Outcomes

By comparing the case studies, outcomes are expected to show how the learning outcomes approach is used at specific institutions in diverse environments to enhance quality of procedures, assessing the competences learners gained in practical training. For example, assessors in Austria use grids based on learning outcomes to document candidates’ performance in their "final exam", which are developed by experts. In Spain, validity and transparency are also ensured in Spain through the use of assessment sheets, used in continuous assessment of in-company training. Moreover, this allows the comparability of learners´ competences. However, assessment tools alone cannot ensure validity, reliability or transparency of assessment procedures even if they apply the learning outcomes approach. There is a need of a so-called communication component, either by regular meetings between tutor-instructor-student where learning outcomes and ‘assessment criteria’ are defined (ES) or a final conference meeting of the examination committee (AT) to relate learners competences to learning outcomes. A gap between the assessment guidelines developed at regional/national level and their implementation at the institution was identified in both countries. Another challenge refers to the understanding and application of assessment criteria related to learning outcomes. For instance, assessment criteria are not defined in detail (AT), and do not define what a candidate has to fulfil in order to achieve a specific grade. In Spain, assessors of the ‘in-company training’ need to be provided with additional training concerning the understanding of ‘assessment criteria’ their application in practice.

References

Bifie (2012). Standardisierte kompetenzorientierte Reifeprüfung / Reife- und Diplomprüfung. Grundsatzpapier BMUKK (2012). Information abschließende Prüfungen an Höheren Lehranstalten, Fachschulen (inkl. Aufbaulehrgängen), Kollegs für Tourismus, Wirtschaftliche Berufe, Mode, Künstlerische Gestaltung, Sozialberufe, Höhere Land- u. Forstwirtschaftl. Schulen. BMUKK (2012). Kompetenzorientiertes Unterrichten an berufsbildenden Schulen. Grundlagenpapier. Stand Juli 2012. 5. Auflage. Cedefop (2010). Learning outcomes approaches in VET curricula. A comparative analysis of nine European countries. Luxembourg: Office for Official Publications of the European Communities. Cedefop (2012). Curriculum reform in Europe: the impact of learning outcomes. Luxembourg: Office for Official Publications of the European Communities. Confebask (2004). Manual del Instructor de Empresa – Formación en Centros de Trabajo. Bilbao Dufaux, Stefanie (2012). Assessment for Qualification and Certification in Upper Secondary Education. A review of country practices and research evidence. OECD Education Working Papers, No. 83, OECD Publishing. Cedefop (2015). Ensuring the quality of certification in vocational education and training. Luxembourg: Publications Office. Cedefop research paper; No 51. http://dx.doi.org/10.2801/25991 European Commission (2011). Using Learning Outcomes. European Qualifications Framework Series: Note 4 Höhere Lehranstalten für Tourismus und Hotelfachschulen/ Stadtschulrat Wien (n/a). Handreichung zur Organisation und Durchführung der fachpraktischen Klausuren.(not published) MODUL tourism VET college (n/a) Prüfungsprotokoll. Vorprüfung zur Reifeprüfung Höhere Lehranstalt für Tourismus. (not published) MODUL tourism VET college (2014a). Vorprüfung zur Reife- und Diplomprüfung. Höhere Lehranstalt für Tourismus. 4HLA. Praktische Klausurarbeit im Fachgebiet Service- Haupttermin 2013/14. (not published) MODUL tourism VET college (2014b). Vorprüfung zur Reife- und Diplomprüfung. Höhere Lehranstalt für Tourismus. 4HLA. Praktische Klausurarbeit im Prüfungsgebiet Küche - Haupttermin 2013/14. (not published) MODUL tourism VET college (n/a.) Beurteilungsblatt Küche. Prüfungsprotokoll - Vorprüfung zur Reifeprüfung - Diplomprüfung. (not published) MODUL tourism VET college (n/a). Kriterien der Leistungsbeurteilung im praktischen Gegenstand Küchenorganisation & Kochen (KOKO) an den Tourismusschulen MODUL der Wirtschaftskammer Wien. Höhere Lehranstalt 1. Jahrgang. National Europass Center (n/a): Certificate Supplement. Reife- und Diplomprüfungszeugnis der Höheren Lehranstalt für Tourismus. QIBB (n/a): Bundesweite Qualitätsschwerpunkte an den humanberuflichen Schulen. Stenström, M.-L.; Laine, K. (eds.) (2006a). Quality and Practice in Assessment: New Approaches in Work-Based learning. Jyväskylä: Jyväskylä University Press. Stenström, M.-L.; Laine, K. (eds) (2006b). Towards Good Practices for Practice-Oriented Assessment in European Vocational Education. Jyväskylä: Jyväskylä University Press.

Author Information

Janine Wulz (presenting / submitting)

research

Vienna

Monika Auzinger (presenting)

3s, Austria

Mariya Dzhengozova