Session Information
09 SES 01 C, Meta-Analysis, Research Synthesis and Evidence Quality
Paper Session
Contribution
Adolescents working in paid jobs may struggle with time management due to the very different expectations of work and school. When they add work obligations to their educational responsibilities, they face potential dangers that could negatively affect their academic performance (Greenhaus & Beutell, 1985). Recently, with economic instability and increasing social inequality becoming more apparent, a growing number of adolescents have been forced to combine their school education with paid work out of necessity rather than choice (Staff et al., 2010). Indeed, we cannot assert that a negative relationship always exists between student employment and academic achievement. Holding multiple social positions can indirectly positively influence success by contributing to the development of various social skills and self-efficacy (Sieber, 1974). The direction and magnitude of these effects depend on contextual factors such as work intensity, the nature of employment, and socioeconomic conditions (Marsh & Kleitman, 2005). Therefore, local structural conditions deeply influence the outcomes of student employment. This study examines how working in a paid job during adolescence shapes academic success in different national contexts, focusing on gender dimensions. Since cultural norms regarding adolescent employment and gender expectations vary significantly across societies, understanding gender-related patterns necessitates a cross-national comparative approach. The study addresses three interconnected research questions:
- What is the general relationship between paid work and students' achievements in mathematics, reading, and science in countries participating in PISA 2022?
- Does gender moderate the relationship between paid work and success?
- How does this effect vary among different academic disciplines? To what extent do the effects of paid work on success vary across national contexts?
This study critically examines how large-scale international assessments that shape education policies can shed light on the complex interaction between the economy and education. By disaggregating the effects across 79 countries, the heterogeneity of the findings was examined, yielding results resistant to the homogenizing tendencies inherent in global assessment frameworks (Rutkowski & Rutkowski, 2016). The marked cross-country differences observed in the results indicate the importance of context-sensitive approaches rather than uniform policy recommendations. Furthermore, this research has addressed some of the crises facing contemporary education systems. Recent global crises have increased financial pressure on families, leading more adolescents to enter paid employment out of necessity rather than choice (Lee & Staff, 2007). If the outcomes of combining work and school are truly context-dependent, understanding which structural conditions exacerbate or mitigate negative effects becomes an urgent policy issue. The findings have important implications for labor regulations designed to protect adolescents' educational opportunities in various national contexts, as well as for countries' education policies and social support systems (Post & Pong, 2000).
Method
This study employed a two-stage analytical design that combined multilevel modeling with meta-analysis, using data from the 2022 cycle of the Programme for International Student Assessment (PISA) (OECD, 2024). The sample comprised 79 countries and economies participating in PISA 2022, representing diverse geographic, economic, and cultural contexts. Student achievement in mathematics, reading, and science was measured using five plausible values per domain, as provided by PISA. Students’ engagement in paid work was assessed using the WORKPAY index, which captures how frequently students work for pay before or after school during a typical school week. Gender was included both as a covariate and as a moderator. All continuous predictors were standardized to facilitate cross-country comparability. The analytical procedure consisted of two stages. In the first stage, two-level hierarchical linear models were estimated separately for each country, with students nested within schools (Raudenbush & Bryk, 2002). These models included main effects of paid work and gender, as well as interaction terms to test gender moderation. Random intercepts at the school level were specified to account for clustering. Each model was estimated separately for ten plausible values, and results were combined using Rubin’s (1987) multiple imputation rules. Student- and school-level sampling weights were applied to account for PISA’s complex sampling design (Rutkowski et al., 2010). In the second stage, country-specific estimates obtained from the multilevel models were synthesized using random-effects meta-analysis (Borenstein et al., 2009). Separate meta-analyses were conducted for each academic domain and predictor, yielding a total of nine meta-analytic models. Cross-national heterogeneity was evaluated using Cochran’s Q statistic, the I² index (Higgins & Thompson, 2002), and the between-study variance component τ². These indicators allowed for an assessment of whether observed variation in effect sizes reflected substantive differences between national contexts rather than sampling variability. This two-stage approach preserves the multilevel structure of the data while enabling the examination of cross-national variability in effects (Snijders & Bosker, 2012). All analyses were conducted in R (R Core Team, 2024). Multilevel models were estimated using the mixPV function (Huang, 2024), which builds on the WeMix package (Bailey et al., 2023) and supports weighted estimation with plausible values. Meta-analyses were performed using the metafor package (Viechtbauer, 2010).
Expected Outcomes
Meta-analytic findings show a consistent negative relationship between paid work and academic achievement across reading, science, and mathematics. Students who work more frequently perform worse in reading (β = −0.152, p < .001), science (β = −0.143, p < .001), and mathematics (β = −0.121, p < .001). These findings suggest that paid work interferes with students’ academic responsibilities, supporting role conflict theory (Greenhaus & Beutell, 1985). Despite this overall pattern, substantial heterogeneity is observed across all domains, with I² values exceeding 89%, indicating considerable cross-national variation in the effects of paid work on achievement (Higgins & Thompson, 2002). Geographic patterns reveal that economically developed countries—such as Scandinavian nations, Canada, the United States, and Australia—exhibit stronger negative associations. Although adolescent employment in these contexts is often not economically necessary, education systems typically assume that students devote most of their time and energy to schooling. Consequently, paid work may conflict more strongly with academic expectations, leading to greater performance declines (Post & Pong, 2000). Gender differences follow established patterns: male students perform better in mathematics and science, whereas female students outperform males in reading (OECD, 2023). Interaction effects examining whether the impact of paid work varies by gender are statistically significant but small (β = −0.007 to −0.018), suggesting that the challenge of balancing work and school affects male and female students in a largely similar manner (Voydanoff, 2005). Overall, these findings indicate that policies targeting working students should be context-sensitive rather than uniform, as the academic risks associated with paid work differ markedly across countries, particularly in developed economies.
References
Bailey, P., Kelley, C., Nguyen, T., & Huo, H. (2023). WeMix: Weighted mixed-effects models using multilevel pseudo maximum likelihood estimation (R package version 4.0.1). https://CRAN.R-project.org/package=WeMix Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley. Greenhaus, J. H., & Beutell, N. J. (1985). Sources of conflict between work and family roles. The Academy of Management Review, 10(1), 76–88. https://doi.org/10.2307/258214 Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558. https://doi.org/10.1002/sim.1186 Huang, F. L. (2024). Using plausible values when fitting multilevel models with large-scale assessment data using R. Large-Scale Assessments in Education, 12(1). https://doi.org/10.1186/s40536-024-00192-0 Lee, J. C., & Staff, J. (2007). When work matters: The varying impact of work intensity on high school dropout. Sociology of Education, 80(2), 158-178. https://doi.org/10.1177/003804070708000204 Marsh, H. W., & Kleitman, S. (2005). Consequences of employment during high school: Character building, subversion of academic coals, or a threshold? American Educational Research Journal, 42(2), 331-369. https://doi.org/10.3102/00028312042002331 OECD (2023), PISA 2022 Results (Volume I): The State of Learning and Equity in Education, PISA, OECD Publishing, Paris, https://doi.org/10.1787/53f23881-en. OECD (2024), PISA 2022 Technical Report, PISA, OECD Publishing, Paris, https://doi.org/10.1787/01820d6d-en. Post, D., & Pong, S. (2000). Employment during middle school: The effects on academic achievement in the U.S. and abroad. Educational Evaluation and Policy Analysis, 22(3), 273-298. https://doi.org/10.3102/01623737022003273 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage. Rubin, D. B. (1987). Multiple imputation for nonresponse in surveys. Wiley. https://doi.org/10.1002/9780470316696 Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142-151. https://doi.org/10.3102/0013189X10363170 Rutkowski, L., & Rutkowski, D. (2016). A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher, 45(4), 252-257. https://doi.org/10.3102/0013189X16649961 Sieber, S. D. (1974). Toward a theory of role accumulation. American Sociological Review, 39(4), 567–578. https://doi.org/10.2307/2094422 Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Sage. Staff, J., Schulenberg, J. E., & Bachman, J. G. (2010). Adolescent work intensity, school performance, and Academic Engagement. Sociology of Education, 83(3), 183-200. https://doi.org/10.1177/0038040710374585 Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1–48. https://doi.org/10.18637/jss.v036.i03
Update Modus of this Database
The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.