Assessment of VET Students’ Competencies: Student Assessments Compared with Assessments by Their Coaches
Author(s):
Harm J.A. Biemans (presenting / submitting)
Conference:
ECER 2012
Format:
Paper

Session Information

02 SES 10 D, Competence Assessment: Methodological Issues And Approaches

Parallel Paper Session

Time:
2012-09-20
15:30-17:00
Room:
FCEE - Aula 2.8
Chair:
Lars Heinemann

Contribution

Within the framework of the vocational qualification structure in Dutch secondary vocational education, the development of student competencies takes a central position. This focus on students’ competence development can be seen in the Vocational Education and Training (VET) systems in many other countries inside and outside Europe (Mulder et al., 2007). Competencies can be described as integrated knowledge, skill and attitudinal aspects needed to perform adequately in specific professional situations (Biemans et al., 2009). This definition stressing the integrative nature of competencies characterises a comprehensive approach towards competence-based vocational education (Delamare Le Deist & Winterton, 2005). In this respect, the comprehensive competence-based education (CCBE) approach is fundamentally different from instrumental approaches (e.g., McClelland 1976), in which competencies are specified in detailed lists of fragmented and assessable behavioural elements related to job performance (Biemans et al., 2004).

 

In CCBE, students should be able to optimally develop their own competencies, not only for handling professional core problems (vocational expertise or ‘knowing in practice’ – cf. Billett, 2001), but also for their personal development, which is an important prerequisite for further education, employability, citizenship, and lifelong learning (Wesselink et al., 2010). In this regard, in Dutch VET, the 25 so-called SHL competencies are leading: these concern competencies like “Following instructions and procedures”, “Planning and organising” and “Co-operating and deliberating” (Khaled et al., 2011).

 

A valid and reliable assessment of these competencies is crucial for education (Braun et al., 2012). According to Wolf (1995, 1), ‘competence-based assessment is a form of assessment that is derived from the specification of a set of outcomes; that so clearly states both the outcomes – general and specific – that assessors, students and interested third parties can all make reasonably objective judgements with respect to student achievement or non-achievement of these outcomes; and that certifies student progress on the basis of demonstrated achievement of these outcomes’. Since traditional assessment methods are ill-suited to a competence-based curriculum, new instruments have to be developed that meet criteria for competence-based assessments (Biemans et al., 2004; 2009).

 

In VET, student competencies are frequently assessed by the students themselves: questionnaires on self-rated competencies have increasingly been employed. However, self-ratings are often criticised for their lack of validity (Braun et al., 2012). Therefore, in addition to the self-assessment by students of their own competencies, it is common practice in VET that students’ competencies are assessed by their coaches as well (and, in some cases, by their fellow-students) as part of a 360̊ feedback assessment procedure. This raises the issue to what extent the assessments of the students’ competencies by the students themselves and the assessments by their coaches are related. In other words, do the students themselves rate their own competencies at the same level as their coaches do? Thus, this study aimed to answer the following concrete research questions: 1) To what extent do the assessments of competencies by students and by their coaches correlate?; and 2) Do the assessments by the students themselves match with the assessments by their coaches?

 

 

Method

The assessments of their own competencies by 142 students (98 boys; 44 girls) and the assessments of these competencies by their own coach were taken into account. Students were in their second, third and fourth study year of 4 educational programmes in Dutch agricultural VET (level 4; fulltime), offered by 5 different schools. To assess students’ competencies, the Competence Development Test (Khaled et al., 2011) was used. Eight SHL competencies that were addressed explicitly in the educational programmes, were assessed by the student him/herself and by his/her coach through a written inventory. For each of the competencies, 4 to 9 corresponding criteria for mastery were given. Assessment was done by assigning a grade from 1 to 10 (1=does not apply at all; 10=fully applies) to each criterion. The reliability coefficients for the competencies varied for the students from α=0.69 to α=0.88 and for the coaches from α=0.90 to α=0.96. The level of correlation between the student assessments and the assessments by the coaches was determined through Pearson correlation coefficients. For each of the competencies, the mean assessment score given by the students was compared through a paired samples t test with the mean assessment score by the coaches.

Expected Outcomes

Only for 4 competencies (“Co-operating and deliberating”, “Formulating and reporting”, “Planning and organising” and “Delivering quality”), a significant correlation between the student assessments and the assessments by the coaches was found. For the other 4 competencies (“Deciding and initiating activities”, “Applying professional expertise”, “Following instructions and procedures” and “Operational acting”), this was not the case. For 7 competencies, a significant difference between student assessments and assessments by the coaches was revealed. In all cases, the mean assessment score given by the students was higher than the mean assessment score given by the coaches. Only for “Planning and organising”, no significant difference was found. To conclude, this study showed that significant differences existed between the assessments of student competencies by the students themselves and by their coaches. For the students, a positive self-evaluation bias could play a role (Bouffard & Narciss, 2011). They do not seem to be able (yet) to assess their own competencies in a realistic way. The assessments by the coaches could be influenced by certain prepossessions as well (Kwam et al., 2008). Thus, in order to assess student competencies in a balanced way, it is important to collect additional, more objective proofs of competence.

References

Biemans, H., Nieuwenhuis, L., Poell, R., Mulder, M. & Wesselink, R. (2004). Competence-based VET in the Netherlands: background and pitfalls. Journal of Vocational Education and Training, 56 (4), 523-538. Biemans, H., Wesselink, R., Gulikers, J., Schaafsma, S., Verstegen, J. & Mulder, M. (2009). Towards competence-based VET: dealing with the pitfalls. Journal of Vocational Education and Training, 61 (3), 267-286. Billett, S. (2001). Knowing in practice: re-conceptualising vocational expertise. Learning and Instruction, 11, 431-452. Bouffard, T. & Narciss, S. (2011). Benefits and risks of positive biases in self-evaluation of academic competence: Introduction. International Journal of Educational Research, 50 (4), 205-208. Braun, E., Woodley, A., Richardson, J.T.E. & Leidner, B. (2012). Self-rated competences questionnaires from a design perspective. Educational Research Review, 7 (1), 1-18. Delamare Le Deist, F. & Winterton, J. (2005). What is competence? Human Resource Development International, 8 (1), 27-46. Khaled, A., Gulikers, J., Oonk, C., Biemans, H. & Lans, T. (2011). Competentie Ontwikkeling Meter [Competence Development Test]. Wageningen: Wageningen University. Kwam, V.S.Y., John, O.P., Robins, R.W. & Kuang, L.L. (2008). Conceptualizing and assessing self-enhancement bias: A componential approach. Journal of Personality and Social Psychology, 94, 1062-1077. McClelland, D. (1976). A Guide to Job Competency Assessment. Boston, MA: McBer. Mulder, M., Weigel, T. & Collins, K. (2007). The concept of competence in the development of vocational education and training in selected EU member states. A critical analysis. Journal of Vocational Education and Training, 59 (1), 67-88. Wesselink, R., Dekker-Groen, A. M., Biemans, H. J. A. & Mulder, M. (2010). Using an instrument to analyse competence-based study programs; experiences of teachers in Dutch vocational education and training. Journal of Curriculum Studies, 42 (6), 813-829. Wolf, A. (1995). Competence-based assessment (assessing assessment). Buckingham, UK: Open University Press.

Author Information

Harm J.A. Biemans (presenting / submitting)
Wageningen University, Netherlands, The

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.