Participant satisfaction, learning success and learning transfer. An empirical investigation of correlation assumptions in Kirkpatrick’s four-level evaluation model.

Author(s):

Michael Gessler

Conference:

ECER 2009

Network:

2. Vocational Education and Training (VETNET)

Format:

Paper

Session Information

02 SES 01 A, Challenges and Issues for VET

Paper Session

Time:

2009-09-28

09:15-10:45

Room:

HG, HS 23

Chair:

Pekka Ilmari Kamarainen

Contribution

Kirkpatrick published in 1959 his four-level model for evaluation. It becomes one of the most important evaluation models for advanced trainings. Kirkpatrick advices evaluation on four levels. These levels are reaction (participant satisfaction), learning (learning success), behavior (learning transfer) and results (business success) (Kirkpatrick 1998, Kirkpatrick & Kirkpatrick 2005). In the later interpretation of the model correlations between the four levels were assumed and since the 1980s these assumptions had been proved with limited success (Noe & Schmitt 1986, Allinger and Janak 1989, Engestroem 1994, Guelpen 1996, Phillips and Schirmer 2005). On the one hand, the measurement of participant satisfaction at the end of an advanced training offering has a long tradition; on the other hand, it takes a key position, even though unsubstantiated assumptions are the basis of this practice. Assumption 1: Input data, process data, output data, and outcome data are linked via chains of effects. Assumption 2: Participant satisfaction at the end of a training/seminar/course prospectively indicates the quality of an educational activity. Assumption 3: Participant satisfaction at the end of a training/seminar/course retrospectively indicates the quality of an educational activity. Assumption 4: Due to the key position of participant satisfaction, it is sufficient and also economical to only measure participant satisfaction. Our study is focused on the proofing of the first and the second assumption listed above, whereas the chain of effects (Phillips & Schirmer 2005:29) customer satisfaction (reaction) -> learning success (learning) -> learning transfer (behavior) is examined more closely. The following content-related hypotheses were examined: Hypothesis 1: There is a correlation between participant satisfaction at the end of an advanced training activity and learning success. Hypothesis 2: There is a correlation between learning success and transfer performance of a participant.

Method

In the presented empirical investigation 43 courses with altogether 335 participants were evaluated on three levels (participant satisfaction, learning success and learning transfer). Statistical tools used: Reliability Analysis, T-test with Dependent Samples, Kolmogorov-Smirnov Test, Correlation Analysis, Regression Analysis.

Expected Outcomes

The results shows that there is no correlation between participant satisfaction and the learning success and there is also no correlation between participant satisfaction and the transfer performance. In view of this result, the practice of evaluating advanced training requires further development. The sole assessment of satisfaction values is not sufficient to evaluate the quality of rendered services.

References

Beywl, W., Speer, S. & Kehr, J. (2004). Wirkungsorientierte Evaluation im Rahmen der Armuts- und Reichtumsberichterstattung. Perspektivstudie im Auftrag des Bundesministeriums für Gesundheit und Soziale Sicherung (BMGS). Online im WWW: http://www.univation.org/index.php?class=seite&id=9031 (22.12.05) Stufflebeam, D.L. (2002). The CIPP Model for Evaluation. In D.L. Stufflebeam, G.F. Madaus & T. Kellaghan. Evaluation Models. Viewpoints on Educational and Human Services Evaluation. Second Edition. eBook. New York u.a., 290 - 317. Carnevale, A.P. & Schulz, E.R. (1990). Return on Investment: Accounting for Training. Training and Development Journal, 7, S1-S32. Gülpen, B. (1996). Evaluation betrieblicher Verhaltenstrainings unter besonderer Berück-sichtigung des Nutzens. Mering: Hampp. Engeström, Y. (1994). Training for change. New approach to instruction and learning in working life. Geneva : International Labour Office. EduQua (2004). Schweizerische Qualitätszertifikat für Weiterbildungsein¬rich¬tungen. Handbuch über das Verfahren, Anleitung zur Zertifizierung. http://www.eduqua.ch/pdf/eduqua_handbuch.pdf Kirkpatrick, D. L. (1998). Evaluating Training Programs. The Four Levels. San Francisco. Kirkpatrick, J. D. & Kirkpatrick, D. L. (2005). Transferring Learning to Behavior. Using the Four Levels to Improve Performance. San Francisco. Phillips J. (1994). In Action. Measuring Return on Investment - Volume 1. Alexandria. Phillips, J. & Schirmer F. (2005). Return on Investment in der Personalentwicklung. Der 5-Stufen-Evaluationsprozess. Berlin, Heidelberg. AZWV 2004. Verordnung über das Verfahren zur Anerkennung von fach¬kundigen Stellen sowie zur Zulassung von Trägern und Maßnahmen der beruflichen Weiter¬bildung nach dem Dritten Buch Sozialgesetzbuch (Anerkennungs- und Zulassungs-verordnung – Weiterbildung – AZWV)“ vom 16. Juni 2004, Bundesgesetzblatt Jahrgang 2004 Teil I Nr. 28, ausgegeben zu Bonn am 22. Juni 2004. Gonon, P. (2008). Wirkung oft ungewiss. In: Weiterbildung, Jg. 18, H. 1, S. 12–15. Noe, R.A. & Schmidt, N. (1986). The influence of trainee attitudes on training effectiveness: test of a model. Personnel Psychology, 39, 497-523. Alliger, G.M. & Janak, E.A. (1989). Kirkpatrick´s levels of training criteria: thirty years later. Personnel Psychology, 42, 331-342.

Author Information

Michael Gessler

Universität Bremen

Institut Technik und Bildung

Bremen

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER.

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.