In the highly interconnected world we are living today, students’ development including their academic performance is influenced by a variety of variables from different levels (Darling, 2007). Researchers, parents, educators, and policy makers are eager to figure out what factors really matter for students’ academic success to support children based on research and empirical evidence. International large-scale studies such as the Trends in International Mathematics and Science Study (TIMSS) collects rich data not only on student’ academic performance, but also on their home and school environment. Often these data are used to describe associations between achievement scores and context variables (e.g., Mullis et. al., 2020). For example, children who were in kindergarten score higher on TIMSS tests than children who were not in kindergarten. Children who report bullying perform worse than children who do not report bullying. Although ‘correlation does not imply causation’ is a truism, this principle is often ignored when interpreting such correlative findings.

Researchers, parents, educators, and policy makers are well advised to carefully interpret correlations, since there may be other observed or unobserved factors that introduce a confounding bias. Children that attended kindergarten may score higher on an achievement test than children who did not becausethey differ not only in terms of kindergarten attendance but also in terms of other determinants of student achievement.

The problem of confounding bias becomes more complicated in a multilevel system such as the school system where students are nested in schools and schools are nested in countries. Confounding within educational context are subject to different levels of bias: e.g., On student level, families with more resources (e.g., more educated parents) may be also more likely to send their children to kindergarten (e.g., Sibley et al., 2015). On school level, the allocation to schools is typically not at random but school choice and existing residential segregation often result in an unequal distribution of resources and teaching quality across schools (e.g., Akiba et al., 2007). Parental choices about kindergarten and school may be correlated, leading to confounding bias from school level variables. Another level of bias comes from the system level. The share of children enrolled into preschools has a high variation across countries, with economically affluent countries tending to perform better on achievement tests and have better-developed preschool systems (e.g., Strietholt et al., 2020).

The purpose of this paper is to illustrate that the associations between student achievement and context variables is biased by confounding factors located on various levels of educational systems. Specifically, we stepwise address the bias on country-, school-, and student-level to illustrate that confounding variables on these level bias associations between student achievement and student context data.

Sample We used TIMSS 2019 grade 4 data consisting of a total 303,954 students, nested in 11,285 schools in 57 countries. The analyses were done using senate weights, meaning that samples are representative of their respective population, and that each country has the same ponderation on the results even when sample sizes differ across countries. Variables The outcome variable was the fourth graders’ mathematic achievement as measured by five plausible values. We used four IRT-based indices that are part of the TIMSS Public Use Files as explanatory variable: early literacy and numeracy activities, participation in preschool education, students' sense of belonging to school, and bullying experiences. As a control variable on student-level we used the IRT-based Home Educational Resources index as a measure of student SES. Mullis et. al. (2020) provides a detailed description of all variables used in this study. Analysis In the present study we used OLS models to study change in the association between student achievement and various contextual variables. As a baseline we regressed achievement on the explanatory variables (model 1). Thereafter, we added stepwise fixed effects (FEs) for countries and schools to account for any unobserved heterogeneity at country and school levels. Based on the baseline model we add fixed effects for countries to remove any unobserved heterogeneity on country-level (model 2). We then replace fixed effects for countries with fixed effects for schools, to remove unobserved heterogeneity on school level (model 3). Note that fixed effects for schools implicitly control for unobserved heterogeneity at the country level, as schools are nested within countries. Finally, in addition to the school fixed effects, we include a key individual-level variable, student social status, in the model to capture at least part of the student-level bias (model 4). For the sake of simplicity, we study the four explanatory variables in separate models, i.e., we replicated the aforementioned models for each explanatory variable. To account for the complex sampling design, we used the Jackknife procedure for the estimation of the standard errors. All analyses below were repeated for each plausible value achievement score, and the results were combined using the Rubin formula (Rubin, 1987).

The analyses provide robust evidence that student-, school-, and country-level confounding variables bias the relationship between student performance and the explanatory variables. In the baseline model we observed a strong correlation between attending preprimary education and the students’ achievement. We find a difference of 75.88 points (SE = 1.80; p < 0.01) between children that had 3 or more years of pre-school versus children that did not attend to pre-primary education at all (model 1). However, after adding country FEs to account for unobserved heterogeneity on country-level this coefficient is halved to 33.61 points (SE = 1.72; p < 0.01; model 2), suggesting bias from country level confounding factors. In the model with school FEs the estimate further reduced to 17.60 points (SE = 0.90; p < 0.01; model 3), suggesting additional bias from school-level confounding factor. Finally, in the model with school FEs and student SES as control variables, the association between student achievement and preschool attendance decreases to 11.93 points (SE = 0.90; p < 0.01; model 4). These findings offer compelling evidence for confounding biases at multiple levels of the education system. We also observe qualitatively the same patterns of results with respect to other three explanatory variables (early literacy and numeracy activities, students' sense of belonging to school, and bullying experiences): strong association reduce considerably when controlling for observed and unobserved heterogeneity at different levels. The above presented analyses are based on cross-sectional data and we do not claim to reveal the true causal effect of different explanatory variables on achievement. However, our analyses illustrate that correlations are typically biased if researchers do not follow proper strategies to account for confounding bias on different levels of educational systems.

Akiba, M., LeTendre, G. K., & Scribner, J. P. (2007). Teacher quality, opportunity gap, and national achievement in 46 countries. Educational Researcher, 36(7), 369–387. https://doi.org/10.3102/0013189X07308739 Darling, N. (2007). Ecological systems theory: The person in the center of the circles. Research in human development, 4(3-4), 203-217. IEA Education. (2021a). Launch of TIMSS 2019 international results & international report [Video]. YouTube. https://www.youtube.com/watch?v=WZeWCiMDpNo&t=1833s IEA Education. (2021b). TIMSS 2019 international results: Home, school, and classroom contexts for teaching and learning [Video]. YouTube. https://www.youtube.com/watch?v=M6JOHTdFOCk Moehring, K. (2021, February 22). The fixed effects approach as an alternative to multilevel analysis for cross-national analyses. https://doi.org/10.31235/osf.io/3xw7v Mullis, I. V. S., Martin, M. O., Foy, P., Kelly, D. L., & Fishbein, B. (2020). TIMSS 2019 International Results in Mathematics and Science. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/timss2019/international-results/ Mullis, I.V.S., Martin, M.O., Foy, P., & Arora, A. (2012). TIMSS 2011 international results in mathematics. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College Rubin, D.B. (1987). Multiple Imputation for Nonresponse in Surveys. John Wiley + Sons, New York, NY. Sibley, E. & Dearing, E. & Toppelberg, C. & Mykletun, A. & Zachrisson, H. (2015). Do increased availability and reduced cost of early childhood care and education narrow social inequality gaps in utilization? Evidence from Norway. International Journal of Child Care and Education Policy, 9. 10.1007/s40723-014-0004-5. https://doi.org/10.1007/s40723-014-0004-5 Strietholt, R., Hogrebe, N., Zachrisson, H.D. (2020). Do increases in national-level preschool enrollment increase student achievement? Evidence from international assessments, International Journal of Educational Development, 79. https://doi.org/10.1016/j.ijedudev.2020.102287 Waldfogel, J., & Zhai, F. (2008). Effects of public preschool expenditures on the test scores of fourth graders: evidence from TIMSS. Educational Research and Evaluation, 14(1), 9-28.