A high number of European students face challenges in reading (Betthäuser et al., 2023; Mullis et al., 2023) and require reading support as the new Progress in International Reading Literacy Study (PIRLS) displayed last year. Consequently, it is crucial to employ formative diagnostics to record the learning levels of students, enabling the design of targeted interventions at an early stage. As a result, the German Kultusminsterkonferenz [Commission of the Conference of Ministers of Education] advocates for the early implementation of nationwide diagnostics (Köller et al., 2022). Furthermore, the PIRLS, is not only conducted through a computer-based assessment since 2016, but also has a focus point in digital forms of reading (Mullis & Martin, 2019). Formative assessment serves not only to monitor the students’ learning but also to provide ongoing feedback, refine teaching approaches, and address the unique needs of individual students (OECD, 2005; OECD, 2008). Formative assessment is essential for both students and teachers. Additionally, in the realm of monitoring and aggregating student data, is learning analytics, which plays a vital role for researchers and developers of educational applications. Researchers seek to understand students' learning behaviors and their utilization of platforms, while developers strive to leverage this information for enhancing learning platforms (SoLAR, 2021).

In the project DaF-L an adaptive, digital, and competence-oriented reading screening with aligned reading packages, consisting of literary texts and reading exercises, was developed, tested, standardized, and subsequently made available as an Open Education Resource (OER) on the online learning platform Levumi (Gebhardt et al., 2016) for primary schools in Germany.

The digital reading packages were developed for three reading ability levels, in which the students were sorted into through the screening. The packages’ reading ability levels consist of the same story line for the literary texts and the same exercise formats with some variation depending on the ability level.

One key importance of the reading packages is the digitalization. The reading packages support different ability levels and an individualized learning where students can work on the exercises at their own speed with integrated tools such as immediate individualized feedback, second try-options, and solutions. Additionally, and essential is that the reading packages can be used as a diagnostic tool, which enables teachers to support students in the best way possible. Teachers are required and encouraged to conduct and to employ diagnostics with the focus to support their students. However, it is very time consuming and difficult for teachers to execute as they often do not know what ability test/s they should convey, how to determine if the application was helpful, if students used their full potential to answer questions, where they might need help, and what tools were helpful or unnecessary in an application. Therefore, the DaF-L project provides diagnostic tools for teachers with everything they need in order to support their students.

Furthermore, through the intervention study, researchers who developed the digital reading packages received essential data. The gathered data offered insights into the reading packages, allowing for an assessment of the reading packages' strengths and weaknesses. This information was utilized to make essential adjustments, ensuring the development of the most optimal application.

Moreover, the research conducted a four-week intervention in a regular classroom setting continuously gathering data to gain a deeper understanding of students' learning behaviors and their utilization of the learning platform.

The presentation will discuss the digitalization of the reading packages with focusing on learning analytics. With the objective of exploring how learning analytics, such as examining the time dedicated to reading literary texts, time spent on answering questions, and assessing the use of the skip-button, can improve learning platforms worldwide.

The collaborative project follows a multi-method design. An ABA-design was selected for the intervention study (Graham et al., 2012). The study was expedited from April 2023 until July 2023 and collected quantitative data of individuals, groups, and classes. It consisted of a survey group (N = 59) and a control group (N = 53). A) Start of the initial testing started, which consisted of the self-developed digital and competence-oriented reading screening and the ELFE 2, which is an established diagnostic test. B) Approximately two weeks later, in the first lesson the students took a self-developed digital a-version test tailored to their reading packages, marking the initiation of a four-week intervention phase. The intervention (reading support) occurred three times a week for 30 minutes within a classroom setting. Students were provided with a reading package based on their proficiency levels and worked on them individually. Throughout all intervention sessions students’ responses as well as any additional information regarding their learning and their platform usage were digitally recorded. At the end of the intervention, students participated in the b-version of the aligned test as well as a second administration of the competence-oriented reading screening and the ELFE 2. A) A follow-up was conducted with the ELFE 2, the screening, and the c-version of the aligned test. Throughout the study, educators were interviewed, and observation protocols were employed. The learning platform Levumi and digital reading packages underwent adjustments based on insights gleaned from these interviews and observation protocols. However, the most intriguing insights of the students’ learning behavior emerged from the data collected during the intervention (learning analytics). This data encompassed various aspects, including the duration students spent completing the entire reading package, the time allocated to each exercise, the time students devoted to initially reading the literary text, the time spent on text rereading, whether students attempted exercises a second time, and the frequency of skip-button usage. Moreover, the scrutinized learning behavior could also be compared with the test results, examining whether factors such as the duration spent on reading materials align with test scores.

The insights gained from interviews with educators and the observation protocols played a crucial role in advancing and enhancing the learning platform Levumi and the digital reading packages. While the observation protocols partially revealed intriguing findings, the data, especially concerning the misuse of the skip-button, was a noteworthy revelation. The collected information also shed light on important aspects of students' reading and response behavior, including the time spent reading the text before tackling exercises, the duration devoted to each exercise, utilization of the text-going-back function for rereading, use of the showing solution feature, engagement with the 2nd attempt, and the frequency of skip-button usage. Furthermore, based on the collected data, adjustments were made to the reading packages. Others such as the removal of the skip-button, in response to the observed misuse, are planned. Moreover, the learning behavior could also be examined in correspondence with the results of the tests, such as if an excessive usage of the skip-button has a negative effect on the test results or if a short reading time correlates with the test results. Additionally, the insights gained on the project can be applied to other learning platforms worldwide in order to enhance those.

Betthäuser, B. A., Bach-Mortensen, A. M. & Engzell, P. (2023). A systematic review and meta-analysis of the evidence on learning during the COVID-19 pandemic. Nature human behaviour. Vorab-Onlinepublikation. Gebhardt, M., Diehl, K. & Mühling, A. (2016). Online Lernverlaufsmessung für alle SchülerInnen in inklusiven Klassen. www.LEVUMI.de. Zeitschrift für Heilpädagogik, 67(10), 444-454. Graham, J. E., Karmarkar, A. M., Ottenbacher, K. J. (2012). Small Sample Research Designs for Evidence-Based Rehabilitation: Issues and Methods. Archives of Physical Medicine and Rehabilitation, 93(8, Supplement), 111-S116. https://doi.org/10.1016/j.apmr.2011.12.017 Köller, O., Thiel, F., van Ackeren, I., Anders, Y., Becker-Mrotzek, M., Cress, U., Diehl, C., Kleickmann, T., Lütje-Klose, B., Prediger, S., Seeber, S., Ziegler, B., Kuper, H., Stanat, P., Maaz, K. & Lewalter, D. (2022). Basale Kompetenzen vermitteln – Bildungschancen sichern. Perspektiven für die Grundschule. Gutachten der Ständigen Wissenschaftlichen Kommission der Kultusministerkonferenz (SWK). SWK: Bonn. Mullis, I.V.S., von Davier, M., Foy, P., Fishbein, B., Reynolds, K.A., & Wry, E. (2023). PIRLS 2021 International Results in Reading. Boston College, TIMSS & PIRLS International Study Center. https://doi.org/10.6017/lse.tpisc.tr2103.kb5342 Mullis, I. V. S., & Martin, M. O. (Eds.). (2019). PIRLS 2021 Assessment Frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/pirls2021/frameworks/ Organisation for Economic Co-Operation and Development (OECD). (2008). Assessment for Learning - Formative Assessment . AE Assessment for Learning . https://www.oecd.org/site/educeri21st/40600533.pdf Organisation for Economic Co-Operation and Development (OECD). (2005). Formative Assessment: Improving Learning in Secondary Classrooms. Policy Brief. OECD. https://www.oecd.org/education/ceri/35661078.pdf PIRLS International Study Center website: https://timssandpirls.bc.edu/pirls2021/frameworks/ Society for Learning Analytics Research (SoLAR). (2021). What is Learning Analytics?. Society for Learning Analytics Research (SoLAR). https://www.solaresearch.org/about/what-is-learning-analytics/ Stanat, P., Schipolowski, S., Schneider, R., Sachse, K. A., Weirich, S. & Henschel, S. (2022). IQB-Bildungstrend 2021: Kompetenzen in den Fächern Deutsch und Mathematik am Ende der 4. Jahrgangsstufe im dritten Ländervergleich. Waxmann Verlag.