09 SES 05 A, Issues in Measurement and Sampling in Large Scale Assessments
International comparative studies in different school subjects showed that countries differ greatly in both level and equity of educational achievement (e.g., Mullis, Martin, Foy, Drucker, 2012; OECD, 2014). One possible explanation of these differences is that there is variation in opportunity to learn (OTL, e.g., Schmidt, Zoido & Cogan, 2014) due to differences in curricula, organization of the educational system, quality of instruction, contextual factors, and home conditions.
Results from meta-analyses suggested that OTL has a strong average-effect size. Reviewed 51 research studies that were carried out during the last twenty years, and a secondary analysis of TIMSS and PISA results, Scheerens (2012) concluded that OTL has a larger effect on achievement than other effectiveness enhancing instruction conditions, such as learning time and instructional leadership.
Even though the importance of OTL is highly recognized, the effect sizes vary across PISA and TIMSS. Luyten (2017) examined the effect of OTL on math and science achievements of 22 countries participating in both TIMSS 2011 and PISA 2012. The average effect across the countries was 0.37 in PISA but basically zero in TIMSS. The explanation may have to do with the measurement property of OTL instrument used in these studies. For example, Luyten attributed the finding to the fact that “TIMSS OTL measures were based on teacher responses and the PISA OTL measures on student responses” (p. 111, Luyten, 2017). Further, the both studies not only use different sourced but also different items to measure OTL. It can thus be argued that the measures of OTL may be multidimensional and this needs to be attended to when study the effect of OTL on achievement.
The purpose of this proposed study is to scrutinize the construct validity of OTL, through gauging the measurement property of OTL measures. Specifically, we question the construct validity of the OTL measure used in PISA. For this purpose, we examine if the OTL measures in PISA 2012 involve construct irrelevant components such as self-concept. By separating OTL and other dimensions, we aim to show overlaps and differences in the OTL measures used in PISA and TIMSS. In further multi-level analyses with the adjusted OTL measure, we estimate the relationship between OTL and student achievement.
Scheerens, J. (2017). Opportunity to Learn, Curriculum Alignment and Test Preparation: A Research Review. Springer International Publishing. Luyten, H. (2017). Predictive Power of OTL Measures in TIMSS and PISA. In Opportunity to Learn, Curriculum Alignment and Test Preparation (pp. 103-119). Springer International Publishing. Schmidt, W. H., Burroughs, N. A., Zoido, P., & Houang, R. T. (2015). The Role of Schooling in Perpetuating Educational Inequality an International Perspective. Educational Researcher, 44(7), 371-386. doi: 10.3102/0013189X15603982 Schmidt, W., P. Zoido and L. Cogan (2014), "Schooling Matters: Opportunity to Learn in PISA 2012", OECD Education Working Papers, No. 95, OECD Publishing, Paris. Paper retrieved from DOI: http://dx.doi.org/10.1787/5k3v0hldmchl-en OECD (2014), PISA 2012 Technical Report, PISA, OECD Publishing, Paris. Report retrieved from https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf Mullis, I.V.S., Martin, M.O., Foy, P., & Drucker, K.T. (2012). The PIRLS 2011 International Results in Reading. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. OECD (2014). PISA 2012 Results in Focus: What 15-year-olds know and what they can do with what they know: Key results from PISA 2012. OECD: Paris.
00. Central Events (Keynotes, EERA-Panel, EERJ Round Table, Invited Sessions)
Network 1. Continuing Professional Development: Learning for Individuals, Leaders, and Organisations
Network 2. Vocational Education and Training (VETNET)
Network 3. Curriculum Innovation
Network 4. Inclusive Education
Network 5. Children and Youth at Risk and Urban Education
Network 6. Open Learning: Media, Environments and Cultures
Network 7. Social Justice and Intercultural Education
Network 8. Research on Health Education
Network 9. Assessment, Evaluation, Testing and Measurement
Network 10. Teacher Education Research
Network 11. Educational Effectiveness and Quality Assurance
Network 12. LISnet - Library and Information Science Network
Network 13. Philosophy of Education
Network 14. Communities, Families and Schooling in Educational Research
Network 15. Research Partnerships in Education
Network 16. ICT in Education and Training
Network 17. Histories of Education
Network 18. Research in Sport Pedagogy
Network 19. Ethnography
Network 20. Research in Innovative Intercultural Learning Environments
Network 22. Research in Higher Education
Network 23. Policy Studies and Politics of Education
Network 24. Mathematics Education Research
Network 25. Research on Children's Rights in Education
Network 26. Educational Leadership
Network 27. Didactics – Learning and Teaching
The programme is updated regularly (each day in the morning)
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.