Session Information
09 SES 07 C, Scrutinizing Tests and Assessments in Reading
Paper Session
Contribution
In Hungary we have a well established, extensive student assessment system, with international studies (PIRLS, TIMSS, PISA) constituting one of its main pillars. PIRLS measures students’ reading abilities in Grade 4 on a nationally representative student sample. Along with the international student assessments the National Assessment of Basic Competencies (National ABC) was introduced in Hungary at the beginning of the millennium. Growing from a sample-based study in 2001 to a full cohort census, the National ABC measures reading and mathematical abilities of every student form Grade 6, 8 and 10 annually since 2008. The main aim of these assessments is to provide annual reports about every school and school maintainer. The reports show students’ results in the two measured areas, and compare these results to students’ socio-economic background and from 2010 their previous results as well. Besides, the National ABC gives a report card for every student, which is accessible only by the student, their parents and teachers. Teachers have access to the results on student and item level through an online software designed to analyze results in detail.
From 2008, every student has a measurement ID, which is used during the administration of both the national and the international student assessments. Therefore, databases of the different student assessments can be linked with each other on the student level from that year on. This provides the possibility of analyzing the results and other characteristics of students participating in PIRLS 2011 at Grade 4 together with their data collected in the National ABC in 2013 at Grade 6 and in 2015 at Grade 8. Linking these two studies on student level can help us answer some validity research questions about both studies.
Although there are some minor differences between the test materials of the two studies, reading abilities of students are measured in both studies based on similar frameworks. In our presentation we give a short comparison of the two reading frameworks, emphasizing similarities and differences between the two definitions and the classifications of the test items. Based on the frameworks, we can anticipate a strong relationship between the reading results of students in PIRLS and the National ABC. We compare PIRLS and National ABC results on the student level, and a strong correlation between them can be seen as a proof of validity for the tests.
Beside test results, we compare background variables from the two studies as well. Students’ socio-economic status (SES) is highly correlated with their abilities, and the National ABC uses SES as one of the main characteristics in school reports. Hence, its validity is crucial for the study. PIRLS measures the same or similar socio-economic variables, using them to provide international comparisons of the effect of SES on reading abilities. Therefore, cross-checking the stability of these variables can support the validity and relevance of reports based on SES for both studies.
Given that the National ABC is a census, we can also examine the sampling processes of PIRLS, analyzing how students in the sample of PIRLS represent the whole population based on the results and other characteristics measured in the National ABC. Similarly, we examine how missing data of non-participating students can distort the results of the studies. On the one hand, we can analyze how the achievement of students in the PIRLS sample but with missing data compares to the achievement of PIRLS participants and how their participation would have affected the national PIRLS result of Hungary. And, vice versa, we can analyze on the PIRLS sample, how students with missing data in the National ABC performed in PIRLS.
Method
Expected Outcomes
References
Balázsi, I., Balkányi, P., Bánfi, I., Szalay, B., Szepesi, I. (2012a). PIRLS és TIMSS 2011 Összefoglaló jelentés a 4. évfolyamos tanulók eredményeiről. Oktatási Hivatal, Budapest. Balázsi, I., Balkányi, P., Bánfi, I., Szalay, B., Szepesi, I. (2012b). A PIRLS és TIMSS 2011 tartalmi és technikai jellemzői. Oktatási Hivatal, Budapest. Balázsi, I., Balkányi, P., Ostorics, L., Palincsár, I., Rábai-Szabó, A., Szepesi, I., Szipőcs-Krolopp, J., Vadász, Cs. (2014). Az Országos kompetenciamérés tartalmi keretei – Szövegértés, matematika, háttérkérdőívek. Oktatási Hivatal, Budapest. Balázsi, I., Lak, Á. R., Ostorics, L., Szabó, L. D., Vadász, Cs. (2016). Országos kompetenciamérés 2015 – Országos jelentés. Oktatási Hivatal, Budapest. Balázsi, I., Lak, Á. R., Szabó, V., Szabó, L. D., Vadász, Cs. (2014). Országos kompetenciamérés 2013 – Országos jelentés. Oktatási Hivatal, Budapest. Balkányi, P. (2012). A comparison of international and Hungarian reading literacy studies’ frameworks. European Educational Reasearch Association Conference, Cádiz, 2012. 18-21, September. Mullis, I.V.S., Martin, M.O., Foy, P., & Drucker, K.T. (2012). PIRLS 2011 International Results in Reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Mullis, I.V.S., Martin, M.O., Kennedy, A.M., Trong, K.L., Sainsbury, M. (2011). PIRLS 2011 Assessment Framework. Chestnut Hill, MA: Boston College.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.