This session combines studies that examine different forms of missing data in international comparative large-scale studies. The overall aim is to investigate current challenges that have emerged in recent years, including issues around the sample representativeness and the validity of performance measures. Five contributions from international scholars use data from the international studies TIMSS (Trends in International Mathematics and Science Study), PIRLS (Progress in International Reading Literacy Study), and PISA (Programme for International Student Assessment) to investigate missing data related issues including the bias, scaling or reliability of these data. The contributions in this session evaluate issues in achievement tests as well as background surveys.

Missing values are a practical problem in virtually all empirical surveys. In particular, non- intentional missing values that are missing by design are a problem in empirical research because they can compromise the integrity of the data. Typical problems relate to the representativeness of data or the accurate measurement of constructs. In this session, we examine missing values in international comparative assessments, looking at missing values in both achievement tests and background surveys. Furthermore, the individual contributions examine both unit and item non- response. The different contributions study both the reasons for missing data and consequences for the integrity of the data. The papers address non-response and exclusions in background surveys and performance tests in the large-scale assessments.

The first two papers look at missing values in performance data. In the first paper, the authors look at students who were excluded from PISA in Sweden and draw on the results of national tests. The other paper, which uses performance data, compares what happens when missing values on individual test items are judged to be incorrect or not administered. This study is based on PIRLS data as well as a simulation study. The other three papers look at missing values in survey data. One of them investigates how exclusion rates have changed over a 20-year period using data from TIMSS, PIRLS, ICCS and ICILS, and the authors note an increase in some countries. The remaining paper the impact of the administration mode on the survey participation  comparing paper- vs online parental surveys in TIMSS.

The session consolidates research on a theme that often receives too little attention, and that is non-response and exclusion in large-scale tests and surveys. The session investigates different methodological issues related to missing data in different international assessments. The session is divided into six parts, four presentations, a discussion by a renowned expert, and an open discussion. 

Anders, J., Has, S., Jerrim, J., Shure, N., & Zieger, L. (2020). Is Canada really an education superpower? The impact of non-participation on results from PISA 2015. Educational Assessment, Evaluation and Accountability, 33, 1, 229-249. https://doi.org/10.1007/s11092-020- 09329-5 Debeer, D., Janssen, R., & De Boeck, P. (2017). Modeling Skipped and Not-Reached Items Using IRTrees. Journal of Educational Measurement, 54(3). 333-363. https://doi.org/10.1111/jedm.12147 De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-Based Item Response Models of the GLMM Family. Journal of Statistical Software, 48, 1-28. https://doi.org/10.18637/jss.v048.c01 Gafni, N., & Melamed, E. (1994). Differential tendencies to guess as a function of gender and lingual-cultural reference group. Studies in Educational Evaluation, 20(3), 309–319. https://doi.org/10.1016/0191-491X(94)90018-3 DeLeeuw, E. D. (2018). Mixed-Mode: Past, Present, and Future. Survey Research Methods, 12(2), 75-89. https://doi.org/10.18148/srm/2018.v12i2.7402 Jerrim, J. (2021). PISA 2018 in England, Northern Ireland, Scotland and Wales: Is the data really representative of all four corners of the UK? Review of Education, 9(3). doi:10.1002/rev3.3270 Micklewright, J., Schnepf, S. V., & Skinner, C. J. (2012). Non-response biases in surveys of school children: the case of the English PISA samples. Journal of the Royal Statistical Society. Series A (General), 175, 915–938.