Large-scale assessments increasingly rely on computer-generated process data to complement traditional test responses. Log files from computer-based assessments are often treated as a particularly reliable data source, based on the implicit assumption that machine-generated data is inherently precise, complete, and free from error. This assumption, however, obscures a range of data quality issues that can substantially affect secondary analyses if left unexamined. In the context of International Large-Scale Assessments (ILSAs), where comparability and reproducibility are central concerns, such issues warrant closer scrutiny. This presentation challenges the notion of “clean” log data by drawing on empirical experiences from processing log files in ILSA studies. It demonstrates that log data frequently contain errors that originate not from respondents, but from the technical and administrative infrastructure of computer-based testing. These errors may arise from programming flaws in assessment software, inconsistencies across software versions, hardware or operating system failures, unstable network connections, or deviations from prescribed testing procedures at the field level. While often rare in relative terms, such errors can have disproportionate consequences for process indicators, time-based measures, or sequence analyses. Using selected real-world examples, the presentation illustrates typical error patterns found in ILSA log files. These include missing or duplicated events, implausible time stamps, corrupted event sequences, and inconsistencies between log data and background or response data. The presentation discusses how such issues can be detected through systematic plausibility checks, rule-based validation, and cross-referencing with technical documentation and field reports. Particular attention is paid to distinguishing genuine respondent behavior from artefacts introduced by the testing system itself. Beyond identification, the presentation addresses strategies for handling erroneous log data. It compares different approaches, ranging from data exclusion and flagging to targeted correction and imputation, and discusses their respective implications for transparency, replicability, and substantive interpretation. The presentation argues that decisions made at the log-processing stage are analytically consequential and should therefore be explicitly documented rather than treated as purely technical preprocessing steps. In conclusion, the presentation advocates for a more critical and reflexive approach to log-file processing in ILSA research. Recognizing and addressing the illusion of clean data is essential not only for improving data quality, but also for strengthening the methodological foundations of process data research in large-scale assessments.

Chen G, Liu Y, Mao Y (2024) Understanding the log file data from educational and psychological computer-based testing: A scoping review protocol. PLoS ONE 19(5): e0304109. https://doi.org/10.1371/journal.pone.0304109 Goldhammer, F., Hahnel, C., Kroehne, U., & Zehner, F. (2021). From byproduct to design factor: On validating the interpretation of process indicators based on log data. Large-Scale Assessments in Education, 9(1), 1–25. https://doi.org/10.1186/s40536-021-00113-5 Goldhammer, F., & Zehner, F. (2017). What to make of and how to interpret process data. Measurement: Interdisciplinary Research and Perspectives, 15(3–4), 128–132. https://doi.org/10.1080/15366367.2017.1411651 He, S. and Cui,Y. (2025). A systematic review of the use of log-based process data in computer-based assessments. Comput. Educ. 228, C (Apr 2025). https://doi.org/10.1016/j.compedu.2025.105245