Session Information
99 ERC SES 05 D, Educational Improvement and Quality Assurance
Paper Session
Contribution
There is substantial evidence supporting the importance of education and its significant positive effect on our lives. Specifically, it leads to higher income (Card, 1999), reduced crime rates (Machin et al., 2011), and better health (Conti & Hechman, 2010), and for women, in particular, it leads to lower birth rates and higher woman autonomy (Cygan-Rehm & Maeder, 2022; The World Bank, 2022). One of the common ways to assess the quality of education in a country, albeit subject to debate, is through standardized examinations. In the international arena, Kazakhstani students score lower than the Organisation for Economic Co-operation and Development’s (OECD) average on international tests like the Programme for International Student Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS), with an overall declining national average over time. To improve the quality of education at primary, basic, and general secondary education levels, the Ministry of Education of Kazakhstan has implemented numerous educational programs in the past decade. However, there is little to no empirical evidence to support these programs and justify the allocated budget apart from success in the low-scale pilot studies. To address this challenge, I perform a policy evaluation of a shift to per capita funding in primary and secondary education levels, piloted in 2014 in partnership with UNESCO and launched in public schools in the capital Astana in 2018 and two other largest cities in the country, Almaty and Shymkent in the following year. Using the Difference-in-Differences, Synthetic Control and the newly introduced Synthetic Difference-in-Difference estimator by Arkhangelsky and colleagues (2021), I find no statistically significant evidence of the shift to per capita funding on the share of students who score below the threshold on a United National Testing (UNT) taken by high school graduates. While there certainly are limitations due to the data’s availability, the study’s short period, and the policy’s possible lagged effects, this paper is a significant step in using empirical research to inform policymaking and evidence-based social intervention in Kazakhstan.
Method
Question What is the impact of the shift to per capita funding in K-12 in Kazakhstan on student UNT scores in Astana? Objective To evaluate the impact of the shift to per capita funding in Astana using a quasi-experimental design and publicly available data, controlling for confounding variables across regions. Program In 2018, 76% of schools in Astana city had to undergo a mandatory transition from ”smeta” funding to per capita funding, followed by two other major cities, Almaty and Shymkent, in 2019 (the cities were, thus, excluded from the analysis). Design, Setting and Units of Analysis Using difference-in-differences (DID), synthetic control (SCM), and synthetic difference-in-difference (SDID) estimators, this study examines the effect of the shift to per capita funding in the city of Astana using UNT examination scores from 2014 until 2022 and compares it to the student outcomes of the remaining 13 regions (excluding an outlier). Main Outcomes and Measures The study uses UNT examination scores (specifically, the number of students who score below the national threshold to enrol in higher education institutions) as an outcome variable. Future studies will also use PISA and TIMSS scores to measure the policy’s success as soon as the results become available in 2023 and 2024. Limitations Due to inconsistency in data reporting, numerous other educational programs initiated by the government simultaneously, and possible lagged effects of the policy, the current study only lays the foundation for further analyses for impact evaluation. Future studies will significantly benefit from using scores from international examinations after sufficient time has passed.
Expected Outcomes
Results The results from all three methods (DID, SC, SDID) show no impact of the program on student UNT scores (coefficients 0.04, 0.03, 0.03, respectively). Findings Using high school examination scores as a measure of success and controlling for demographic characteristics across regions, the shift to per capita funding does not affect student outcomes in public schools. Meaning Despite the government’s report of the program’s positive effect on student outcomes (8.6% increase), the results of this study show no effect of this program.
References
Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of california’s tobacco control program. Journal of the American Statistical Association, 105(490), 493–505. doi: 10.1198/jasa.2009.ap08746 Abadie, A., Diamond, A., & Hainmueller, J. (2014). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510. doi: 10.1111/ajps.12116 Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the basque country. American Economic Review , 93 (1), 113–132. doi: 10.1257/000282803321455188 Arkhangelsky, D., Athey, S., Hirshberg, D. A., Imbens, G. W., & Wager, S. (2021). Synthetic difference-in-differences. American Economic Review , 111 (12), 4088–4118. doi: 10.1257/aer.20190159 Asian Development Bank. (2018). Kazakhstan country gender assessment. Country Planning Documents, 1–109. doi: 10.22617/tcs179181 International Association for the Evaluation of Educational Achievement. (2023b). Trends in international mathematics and science study (timss). IEA. Re- trieved from JSC Financial Center. (2022). Jsc financial center. Retrieved from https:// www.invest.fincenter.kz/
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.