Computer Adaptive Testing in the Vocational Training Domain ‘Accountancy’

Author(s):

Christoph Helm(presenting / submitting)

Conference:

ECER 2015

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 14 B, Developing and Scrutinizing Tests in the Domains of Finance, Accountancy and Economics

Paper Session

Time:

2015-09-11

15:30-17:00

Room:

328. [Main]

Chair:

Luc Le

Contribution

Objectives

Austria is one of the OECD countries with the largest share of students in vocationally-oriented upper secondary education (OECD, 2012). These full-time, vocational schools aim to help students to attain A-levels in commercial areas, especially in Accountancy. However, the few existing approaches to developing psychometric tests that allow assessing students’ competence in the domain Accountancy are designed for use in the German dual training system (Winther, 2010), and thus not appropriate to diagnose Austrian students’ competence at upper secondary school. Given the lack of appropriate tests the so-called WBB (Helm, 2015) was constructed – based on Evidence-Centred Assessment Design framework (ECD; Mislevy & Riconscente, 2006) and Item Response Theory (e.g. Embretson & Reise, 2000) framework – for assessing students’ academic achievement at grade 9 through 11 (using vertical scaling methods; Kolen & Brennan, 2004). However, like all paper-pencil tests the WBB has several disadvantages, such as the need for manual scoring and estimation of students’ theta, common test length for all students, etc. Thus a computer-based adaptive test (CAT) was developed in cooperation with researchers from the Institute for Informatics at JKU Linz. A first (German language) beta-version of the CAT can be investigated on http://adaptivetesting.ce.jku.at/. The main objective of the presented paper is to show how this CAT was developed by referring to essential fundamentals from content-related didactics (domain and instructional analysis), psychometrics (IRT and CAT background) and informatics (java-implementation). Furthermore we present findings from a validation study in order to proove satisfying reliability and validity of the CAT version of the WBB.

Conceptual framework

The main theoretical frameworks for constructing the computer-adaptive test in Accountancy are two-folded:

First, a test was constructed based on substantial content-related grounds. Starting with layer 1 of the ECD framework, an analysis of the domain “Accountancy” was conducted with respect to its appearance in students’ school life (e.g. analysis of text books and teacher interviews). Moreover, in accordance with the Austrian educational standards for vocational training (www.berufsbildungsstandards.at) the latent construct of interest was defined: It represents the ability to apply the system of double-entry bookkeeping and to make use of central lows of bookkeeping (Grohmann-Steiger et al., 2008). Subsequently, in line with layer 2 of the ECD approach (Domain Modeling), an assessment argument was formulated that backs our claims about students’ abilities in Accountancy. These arguments (warrants) link potential students’ responses (evidence) to our inferences about their competence. An example: Tobias has answered a number of bookkeeping problems that call for a variety of operations involving building booking records, calculating taxes and estimating a business transaction’s impact on the profit and loss of a company, and so on. We posit that if a student is able to carry out these operations he/she has mastered the basic concept of bookkeeping. This is the warrant, and the backing comes from both classroom experience and research such as that from Helm (2014). Layer 3 “Conceptual Assessment Framework” refers to the operational implementation of the assessment and asks for how to collect and evaluate data that inform the target inference about students’ ability as well as which underlying statistical model is used.

Second, against the background of IRT the underlying statistical model is the unidimensional dichotomous Rasch model. All test items were dichotomously coded and students’ response pattern is subsequently predicted by the 1pl model. However, in the framework of CAT (see below) the 2pl model is used additionally in order to improve measurement efficiency. Thus, the item bank needed for the subsequent CAT version of WBB contains item parameters for both the 1pl model (item difficulty only) and the 2pl model (item difficulty and discrimination).

Method

The CAT approach represents the third framework of the present paper: A java-based online platform was programmed in order to display items to examinees and to collect their response in line with the process of adaptive testing according to Wainer, et al. (2000). Following the basic idea of adaptive tests a random item is selected initially out of an item bank and presented to the examinee. The observed response to this item (e.g. a bookkeeping task) is automatically compared with the correct solution and coded either as correct or wrong. Based on the response(s) observed and the 1pl model (or 2pl model respectively), a student’s ability and the standard error of measurement is estimated using maximum likelihood weighted information criterion (MLWI). If the stopping rule (S.E. < .3) is satisfied, the end of the test is reached; otherwise the next WBB item is selected based on MLWI (and an item exposure control function). All estimations are done using catR (Magis & Raiche, 2011). In order to create the aforementioned item bank, a calibration study based on 993 students from full-time vocational schools was conducted in grades 9 to 11. Based on that data the WBB item parameters were estimated against the background of the 1pl and 2pl model respectively. Those parameters then were transformed to a common ability scale using vertical scaling and common item design (Kolen & Brennan, 2004). These transformations were conducted using the R package “plink” (Weeks, 2010). Results were subsequently transferred to the java environment. However, in order to show whether the CAT version of the WBB reveals satisfying reliability and validity, an additional study has to be conducted. This study will take place in Spring 2015. At the moment it is only possible to report results from a “simulation study” that uses examinees’ responses from the paper-pencil test (see calibration study) as entries for the CAT version. Two analyses were conducted: (1) Using the “randomCAT” command implemented in catR, the total number needed for satisfying the stopping rule (S.E. < .3) was explored for the 1pl and the 2pl model; (2) Students’ ability scores from “simulations” with the CAT version of the WBB are correlated with students’ real WBB scores from the calibration study and their school grades (both from grade 9 throughout grade 11) as well as their Math ability at the beginning of grade 9.

Expected Outcomes

Early results show that when using the 2pl model in order to estimate students’ abilities far less items (on average 16) are needed to get a precise enough estimate – compared to the 1pl model (on average 46). Furthermore the validity checks showed that the scores obtained from the CAT version of the WBB are substantially related to external validity criterion such as students’ scores on the paper-pencil version of the WBB (rgrade9 = .42, rgrade10 = .33, rgrade11 = .51), their Math proficiency (r = .57) and their grades (r grade9 = -.36). However, the correlation with students’ grades at grade 10 vanishes indicating that there might be a problem of validity!? Further analyses are necessary here. To summarize, the presented investigations show that (1) CAT in Accountancy can lead to measurement efficiency, shorter tests and test time, automated feedback, etc., (2) the 2pl model is preferred to the 1pl model (16 vs. 46 items), (3) satisfying reliability (α .91) and validity could be reached, and (4) that more calibrated items are needed in order to avoid item exposure. Apart from that, the major goal of this test development is to provide students and teachers with instant diagnostic information on their skills in Accountancy. Whereas so far students’ could login on the website www.edumetrics.at to get feedback on their performance on the paper-pencil version, the CAT automatically informs students about which tasks they solved correctly and which they did not. This kind of diagnostic information is discussed against the background of Cognitive Diagnosis Models in vocational education (Helm, Trost, George, & Pocrnja, 2015).

References

Grohmann-Steiger, C., Schneider, W., & Eberhartinger, E. (2008). Einführung in die Buchhaltung im Selbststudium. Band I. Wien: Facultas. Helm, C. (2014). Lernen in Offenen und Traditionellen UnterrichtsSettings (LOTUS). Empirische Analysen zur Kompetenzentwicklung im Fach Rechnungswesen sowie zu förderlichen Elementen kooperativen, offenen Lernens an berufsbildenden mittleren und höheren Schulen in Österreich. Unv. Dissertation. Institut für Pädagogik und Psychologie. Johannes Kepler Universität, Linz. Helm, C. (2015). Berufsbildungsstandards und Kompetenzmodellierung im Fach Rechnungswesen. In: Bundesinstitut für Berufsbildung (Eds.), Bildungsstandards und Kompetenzorientierung. Herausforderungen und Perspektiven der Bildungs- und Berufsbildungsforschung (pp. xx-XX). xx: XX. Helm, C., Trost, S., George, A. C. & Pocrnja, M. (2015). Potentiale kognitiver Diagnose-modelle für den berufsbildenden Unterricht. In: Stock, M., Schlögl, P., Schmid, K., & Moser, D. (Hrsg.), Kompetent – wofür? Life-Skills – Beruflichkeit – Persönlichkeitsbildung (pp. 206-224). Innsbruck: StudienVerlag. Kolen, M. J., & Brennan, R. L. (2004). Test equating, scaling, and linking: Methods and practices. New York: Springer-Verlag. Magis, D., & Raiche, G. (2011). catR: an R package to generate IRT adaptive tests. R package version 2.1. Mislevy, R. J., & Riconscente, M. M. (2006). Evidence-centered assessment design. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 61-90). Mahwah, NJ: Erlbaum. OECD (2012). Education at a Glance 2012: OECD Indicators. OECD Publishing. http://dx.doi.org/10.1787/eag-2012-en R Core Team. (2014). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org. Wainer, H. (2000) Computerized Adaptive Testing (A Primer). New Jersey: Lawrence Erlbaum Associates. Weeks, J. P. (2010). plink: An R Package for Linking Mixed-Format Tests Using IRT-Based Methods. Journal of Statistical Software, 35(12), 1-33. URL http://www.jstatsoft.org/v35/i12/. Winther, E. (2010). Kompetenzmessung in der beruflichen Bildung. Bielefeld: Bertelsmann.

Author Information

Christoph Helm (presenting / submitting)

Johannes Kepler University of Linz, Austria

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.