This paper presents the results of a feasibility study using real paper test items and actual non-trained common LLMs. While in theory (Breuer et al, 2023) closed-type items, such as multiple-choice and similar items, are highly objective and scoring tends to be straightforward, there is still a small proportion of errors. Those come from students‘ ambiguous responses (multiple options selected, crossed out, errors corrected), from unfocused rater or other sources (for example, design flaws of an item). There are several ways to improve the quality and veracity of the rating process and ensure accurate results like better item design, additional checking, etc. (Betts et al, 2021). Since the advance of LLMs and other AI models, there is also an option to harness digital technology to control and reduce the error rate further. This is fundamentally different from the field of automated scoring of open-ended items, where AI models are trained to make estimates about content (Lit et al, 2014). In our case, it is more about using AI for routine and repetitive tasks of transcription or simple scoring. We will examine four scenarios that combine two different item types (MC item and connecting item) with two rating procedures (rater just transcribing the answer vs. rater awarding points directly). All four scenarios are actual test results from the Slovenian General matura (Grade 13 students) and national assessment (Grade 9 students) for the spring term of 2025. Connection items are from General matura English HE (1869 students) and National Assessment English Gr9 (5165 students), while multiple choice items are from General matura Physics (1429 students) and National Assessment Biology Gr9 (5132 students). For each scenario, the students‘ responses will be given to AI model with scoring instructions. Whenever the final result (transcribed letter, awarded point) will differ, the response will be examined by a human expert, and his decisions will be final (implicitly correct). This will miss the errors where both AI and humans would score incorrectly, but since the error rates as estimated from appeals and requests for remarking are low to begin with, checking for those would present a disproportionate burden in available resources and wouldn’t be feasible. The error rates of AI compared to the error rates of human raters will give us the answer if the inclusion of such an additional procedure in a regular testing situation can substantially improve the quality of test results.

Betts, J., Muntean, W., Kim, D., & Kao, S. (2021). Evaluating Different Scoring Methods for Multiple Response Items Providing Partial Credit. Educational and Psychological Measurement, 82, 151 - 176. https://doi.org/10.1177/0013164421994636. Breuer, S., Scherndl, T., & Ortner, T. (2023). Effects of response format on achievement and aptitude assessment results: multi-level random effects meta-analyses. Royal Society Open Science, 10. https://doi.org/10.1098/rsos.220456. Liu, O., Brew, C., Blackmore, J., Gerard, L., Madhok, J., & Linn, M. (2014). Automated Scoring of Constructed-Response Science Items: Prospects and Obstacles. Educational Measurement: Issues and Practice, 33, 19-28. https://doi.org/10.1111/emip.12028.