Computerised adaptive testing (CAT) is increasingly used in large-scale educational assessment studies. Compared to paper-based fixed tests, CAT has several advantages, including increased accuracy of the measurement and a better match of the difficulty level of the test with the test taker’s abilities (van der Linden & Glas, 2000). The idea of adapting the test’s difficulty to the test taker originates from individually administered traditional intelligence tests (e.g., Wechsler, 1944), but computer-based testing has made it possible to administer the tests also in group settings. The main idea of CAT is to have a validated item bank, in which each item has difficulty and discrimination parameters that have been defined in advance based on calibration data using IRT (item response theory) modeling (Birnbaum, 1968). After a fixed set of anchor items, the subsequent items are selected from the item bank based on the test taker’s performance.

Even though CAT solves many measurement issues compared to a more traditional approach with fixed tests, accurate assessment of the abilities of test takers who demonstrate significantly lower or higher performance than average can still be challenging. In educational assessment studies utilising item banks calibrated with data collected in school contexts, these problems may emerge when trying to assess the performance level of particularly highly skilled students or lower-performing students. Despite the large calibration data and the number of items, the performance estimates may not be trustworthy if a student does not fail enough items, or if they fail all of them within the limited time allowed for completing the test. In these occasions, adaptive testing does not solve the problems often faced in group-level assessments utilising fixed tests, including the so-called ceiling effect (Wang et al., 2009) which can distort analyses where the ability distributions of different sub-groups are compared on the population level. Thus, the IRT-based approach alone does not provide sufficient information on student performance if the testing time is not extended or if the difficulty level of the starting point of the test is not individually manipulated. This is usually not possible in large-scale group-level assessments. Therefore, in the context of a large-scale longitudinal assessment study with a particular focus on students with special educational needs, we aim at solving this measurement problem by – besides considering students’ performance – taking into account multiple background-related factors associated with performance in earlier assessment studies. We first conducted a simulation study to compare different performance estimation methods in a longitudinal context (IRT, plausible values) and then applied the most appropriate solution on the empirical data.

In the simulation study, we explored whether plausible values can better capture the true differences between subgroups in the population when there are severe ceiling effects in the raw test scores or IRT estimates. Under two conditions (ceiling effects vs. no ceiling effects), we created six different scenarios that differed from each other in how subgroup differences evolved between three time points. The empirical analysis of the study will be based on the data (N ~ 3100) from the longitudinal study conducted in Finnish primary schools (ISCED level 1). In this single-cohort study, the same students were followed yearly from the 4th grade to the 6th grade in the spring semesters of 2022–2024. The analysis focuses on students’ arithmetical reasoning skills that were assessed with tasks that adjusted to each student’s performance level. The data come from a computerised adaptive test that was calibrated with over 10000 primary and secondary school students in Finland. After four anchor items that were the same for all students, the task set presented the student with either slightly more difficult or easier items depending on the student’s responses. The task set was programmed to end automatically once the student had completed 15 tasks or had spent 15 minutes on the test. In the empirical study, we will create plausible values for the students based on their raw performance scores and their background (gender, parents’ occupational status, immigration background, special educational needs). We will study whether the use of plausible values will influence the conclusions made on student performance and its development compared to the IRT-based scoring (ML type scores). We focus particularly on student subgroups whose performance deviates considerably from average students for whom the adaptive test is primarily calibrated.

The results of the preliminary simulations indicate that the use of plausible values can to some extent correct the ceiling effects observed when the estimation of performance is only based on IRT. In the next stage of the analysis, we will create plausible values for students in the empirical data and conduct corresponding comparisons. Based on the results of the preliminary simulations, we expect to increase the accuracy of the measurement of performance particularly for highly skilled students who reached the end of the testing time before finding the upper limit of their abilities. From the perspective of the research project providing the empirical data for the study, it is also important to see whether we manage to improve the accuracy of estimation for students with special educational needs for whom the group-level adaptive test starts with items that are too difficult. The results of the study can be used in the design of assessment projects focusing on educational equality, as such projects often analyse the results of student subgroups whose performance deviates from that of the average student at a certain age or grade level.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley. van der Linden, W. J. & Glas, C. A. W. (2000). Computerized adaptive testing : theory and practice. Kluwer Academic Publishers. https://doi.org/10.1007/0-306-47531-6 Wang, L., Zhang, Z., McArdle, J. J., & Salthouse, T. A. (2009). Investigating ceiling effects in longitudinal data analysis. Multivariate Behavioral Research, 43(3), 476–496. https://doi.org/10.1080/00273170802285941 Wechsler, D. (1944). The measurement of adult intelligence (3rd ed.). The Williams & Wilkins Company.