One of the current trends in education is the shift towards more competence-based education (Baartman, Bastiaens, Kirschner & Van der Vleuten, 2007). In the Netherlands, for example, the ministry of education decided that all vocational education institutes must formulate their curriculum according to principles of competence-based education which has led to concomitant changes in learning outcomes. Whereas students used to be taught knowledge and skills separately, they are now acquiring competences in which knowledge, skills, and attitudes are integrated. One of the implications of this change in educational emphasis is an increased use of competence assessments such as performance assessments, situational judgement tests, and portfolio assessments (Baartman, Bastiaens, Kirschner & Van der Vleuten, 2006). These competence assessments are often combined into a competence assessment program (CAP) that is aligned with the competence-based curriculum. Within a context of competence-based education CAPs are used for high stake decisions on students, therefore, it is important that the quality of these CAPs is ensured.

An important aspect of the quality of assessments in general is reliability, and also for CAPs it is important to quantify the precision of scores. Like test validity, reliability must be interpreted relative to particular testing purposes and context (Haertel, 2006). Within the context of a CAP there are some elements that need to be considered regarding the estimation of reliability. First of all it should be considered that a CAP results in one decision about a student, therefore, the accuracy of exact scores on the elements of the CAP is less important than the accuracy of the decision made about the student (Nitko & Brookhart, 2007). And furthermore, when continuous scores are interpreted with respect to one or more cut scores, conventional indices of reliability may not be appropriate. Since the score interpretation of a CAP is usually standards-based and makes use of cut scores to a classify students in a series of performance levels, it may be more useful to derive probabilities of misclassification from standard errors (Haertel, 2006).

In order to evaluate the reliability of CAPs we propose to estimate the percentage of misclassification of the decision that is made. Verstralen (2009a; 2009b) showed in his reports how this can be done for a combination of tests. Within this method the classification accuracy is estimated with regard to cut scores and decision rules that are used to aggregate the results of different tests (Van Rijn, Béguin & Verstralen, 2009). Within this study we will apply this method to the results of an actual CAP to investigate whether the estimation of the percentage of misclassification can be used to evaluate the reliability of a competence-based exam program.

Baartman, L. K., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. (2006). The wheel of competency assessment: Presenting quality criteria for competency assessment programs. Studies in Educational Evaluation, 32, 153-170. Baartman, L. K., Bastiaens, T. J., Kirschner, P. A., & van der Vleuten, C. P. (2007). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational Research Review, 114 – 129. Haertel, E. H. (2006). Reliability. In R. L. Brennan (ed.), Educational Measurment 4th edition. (pp. 65-110). Westport: American Council on Education and Praeger Publishers. Nitko, A.J. and Brookhart, S.M. (2007). Educational Assessment of Students (5th Edition). Upper Saddle River ,NJ: Pearson Education. Van Rijn, P., Béguin, A., & Verstralen, H. (2009). Failing or Passing? Measurement Precision of Examinations in Secondary Education. Pedagogische Studiën, 86, 185-195. Verstralen, H. (2009). Accuracy of Exams: CTT and IRT Compared. Internal report, Cito. Verstralen, H. (2009). Quality of Certification Decisions. Internal report, Cito.