Decision Rules in Competence Based Assessment for Vocational Education and Training

Author(s):

Marianne HubregtseHuub Verstralen Theo Eggen

Conference:

ECER 2010

Network:

9. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 02 B, Assessment: Methods and Applications I

Paper Session

Time:

2010-08-25

11:15-12:45

Room:

P673, Porthania

Chair:

Aleksander Baucal

Contribution

Vocational education and training (VET) is an important form of secondary education. Most final exams in VET nowadays are competence-based. In these exams students are asked to perform certain core-tasks of their future jobs in authentic, practice situations. It is essential that the decisions made are of high quality, since obtaining a VET-diploma strongly influences one’s chances of getting a suitable job. This holds for VET in any country.

Though there has been some research into the decision quality for traditional secondary education (for instance Van Rijn, Béguin & Verstralen, 2009), it remains unstudied how the quality of decision for a high stakes competence-based exam is influenced by the decision rules used. To decide whether a student receives a diploma, there are two moments in which a decision rule is invoked.

In competence-based VET examination is not done per course, but per core-task. Each core-task consists of one or two practical exams, or observations. Per observation a score is determined, which is derived from the observed behavior of the student during the exam. This is where the first decision rule prescribes with which scores (passes and fails on the observations) a student passes the core-task.

Subsequently, a separate, second decision rule specifies the combination of passes and fails on all core-tasks that yields a diploma. There are about four core-tasks per vocational direction.

In general, there are three types of decision rules (Chester, 2003; Douglas, 2007):

- Conjunctive; all passes

- Complementary; passes on certain parts

- Compensatory; on average a pass on the whole

The scope of this study is what the best form for the first decision rule is, with as goal to guarantee high-quality decisions. The first decision rule will be varied to be conjunctive, complementary or compensatory with respect to passes and fails on the competences.

In this study, the second decision rule is set to be conjunctive; all core-tasks should be passed to receive a diploma. This is done to be able to investigate the exact influence of the first decision rule. This seems reasonable, since the core-tasks are initially chosen to be highly critical descriptions of tasks in practice. It makes sense to demand that they are all passed.

Method

The data is simulated, based on real data. First, virtual students, representing real-life students, are randomly drawn from a specified population. They should either pass or fail their diploma (true-group). Second, it is pretended that the students do practical exams, by adding random error to the individual scores on simulated test-moments of the students, representing observation in practice. The results thus obtained are generalized using the chosen first decision rule and the set conjunctive second decision rule. Thus is decided whether the student obtained her diploma. This is compared to the true group of the student, yielding the following classification: - due passes; student passed due to true-group - undue passes; student passed due to measurement error - due fails; student failed due to true-group - undue fails; student failed due to measurement error The decision rule that yields the least undue passes and undue fails is seen as ensuring the highest quality decision.

Expected Outcomes

In the real world there is always measurement error. The effects of measurement error on the quality of the decision may be mitigated by a form of compensation. Especially students whose true ability lies close to the cut-off point between pass and fail, are likely to end up unduly failing or unduly passing. This also seems intuitively reasonable, since high-performing students are not likely to be observed as failing an observation. Neither are low-performing students likely to be seen as passing the observation. Thus it is reasonable to assume that students further away from the cut-off point end up in their correct group. (See also Van Rijn, Béguin & Verstralen, 2009.) Therefore, it is expected that the more compensatory the decision rule, the better the quality of the decision. Since conjunctive rules are the least compensatory, it is expected that this rule will yield the lowest quality decision. A complementary rule is expected to yield higher quality decisions than a conjunctive rule, since there are more possibilities for mitigating the effects of measurement error. Finally, it is expected that a compensatory rule yields the highest quality decision. From the findings, recommendations for decision rules are given.

References

Chester, M.D. (2003). Multiple Measures and High-Stakes Decisions: A Framework for Combining Measures. Educational Measurement: Issues and Practice, 22, 32-41. Douglas, K.M. (2007). General Method for Estimating the Classification Reliability of Complex Decisions Based on Configural Combinations of Multiple Assessment Scores. Dissertation. University of Maryland, College Park, MD, United States of America. Van Rijn, P., Béguin, A. & Verstralen, H. (2009). Failing or Passing? Measurement Precision of Examinations in Secondary Education. Pedagogische Studiën, 86, 185-195. Verstralen, H. (2009). Accuracy of Exams: CTT and IRT Compared. Internal report, CITO. Verstralen, H. (2009). Quality of Certification Decisions. Internal report, CITO.

Author Information

Marianne Hubregtse

Kenniscentrum Handel, RCEC

De Bilt

Huub Verstralen

Cito, The Netherlands

Theo Eggen

University of Twente, The Netherlands

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.