Who makes it into PISA? Understanding the impact of PISA sample eligibility using Turkey as a case study (PISA 2003 – PISA 2012)

Author(s):

Nicholas Spaull(presenting / submitting)

Conference:

ECER 2017

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 05 A, Issues in Measurement and Sampling in Large Scale Assessments

Paper Session

Time:

2017-08-23

13:30-15:00

Room:

W3.11

Chair:

Agnes Stancel-Piatak

Contribution

Of the OECD countries that participate in the Programme for International Student Assessment (PISA), Turkey has one of the lowest levels of performance and the highest rates of improvement in PISA scores between 2003 and 2012. New evidence presented in this paper suggests that existing accounts have underestimated both progress and inequity in Turkey because they did not take into account the large proportion of 15-16 year olds that are ineligible for the PISA sampling frame, either because they are no longer in school or because they are severely delayed. Using Turkey’s Demographic and Health Survey (DHS) data for 2003, 2008 and 2013 we show that the proportion of 15-16 year olds that were actually eligible for the PISA sample in Turkey has nearly doubled from 45% in PISA 2003, to 80% in PISA 2012. By combining household survey data on access/attainment and PISA survey data on learning outcomes we show that: (1) the improvement in the percentage of 15-16 year olds reaching Level 2 in PISA (functional literacy and functional numeracy) is up to twice as large as that reflected in official PISA reports, (2) the gap in functional literacy rates between rich and poor youth in 2012 is 2.3 times as large as was previously thought, and (3) contrary to earlier research the gap between rich and poor has not declined between 2003 and 2012. The paper emphasises the importance of accounting for sample eligibility and representivity when making inter-country and inter-temporal comparisons using PISA (or any other international assessment) data, particularly for developing countries with expanding education systems.

Method

In PISA participating countries with low coverage rates, in order to gain an accurate picture of school system success and how this has changed over time, it becomes necessary to utilise (1) household survey data on enrolment and progression in conjunction with (2) PISA data on achievement for those in the sampling frame. Doing so allows one to disaggregate the total number of 15-16 year old youth into four mutually-exclusive categories: 1. Those who never enrol in school, 2. Those who enrol in school initially but dropout or leave school with a primary school diploma before they are 15 years old, 3. Those who enrol and remain in school but do not reach Grade 7 by the time they are 15-16 years old, 4. Those who enrol, remain in school and reach at least Grade 7 by the time they are 15-16 years old (i.e. those who make it into the PISA sampling frame). In order to do this one needs two data-sources: (1) a measure of educational enrolment and attainment by age, and (2) a measure of learning outcomes at a particular age (15-16 years of age in the case of PISA). The reason for this is that cross-national studies of educational achievement such as PISA typically use schools as the sampling unit and thus cannot provide information on the out-of-school population. Information on topics such as non-enrolment, progression and dropout by gender, and particularly socioeconomic status, are also not usually available in government administrative data (see the online appendix of Spaull & Taylor, 2015 for a full discussion). In a similar vein, household surveys such as the DHS have only rudimentary assessments of learning outcomes, or none at all. Combining data from household surveys and international assessments allows one to combine the benefits of each survey into a more meaningful metric of school-system success. The present analysis builds on earlier work of mine where I have applied the same method to 11 African countries using Southern and Eastern African Consortium for Monitoring Education Quality (SACMEQ) data (Spaull & Taylor, 2015; Taylor & Spaull, 2015), as well as the conceptual insights of prominent scholars in the field (Filmer, Hasan, & Pritchett, 2006; Pritchett, 2013).

Expected Outcomes

Close scrutiny of the estimates provided in the tables and figures in this paper point to 6 main findings: 1. Significant increases in PISA sample eligibility over time: Across all groups there have been large and statistically significant increases in PISA sample eligibility in Turkey over the 2003-2012 period. While only 45% of 15-16 year olds were eligible for PISA in 2003, by 2013 this had nearly doubled to 80%. This agrees with the existing literature that highlights the pro-poor policies focussed on access in Turkey (Aydagül, 2008b; Dülger, 2004). 2. The improvement in the proportion of 15-16 year olds reaching Level 2 in PISA is between 2 times larger (for numeracy) and 5 times larger (for literacy) than previously thought. 3. There are considerably more functionally illiterate and functionally innumerate Turkish 15-16 year olds than PISA reports indicate because the PISA sampling frame only considers those 15-16 year olds that are still enrolled in school and in Grade 7 or higher. 4. Severe delays in progression have been virtually eliminated in Turkey. 5. Dropout / early school-leaving among 15-16 year olds declined by 42% over the period: This again would seem to point to the success of government policies focussing on inclusion and gender (Köseleci, 2015; Tansel, 2002). 6. The rich are much more likely to be sampled in PISA: In all three waves of PISA the poorest 40% of 15-16 year olds are statistically significantly less likely to be in the PISA sample than the richest 20%. In 2003, 63% of the richest group were eligible for PISA compared to 30% of the poorest group. Thus the rich were twice as likely to be sampled as the poor in 2003.

References

Bloem, S. (2013). PISA for low- and middle-income countries. OECD Education Working Papers, (93), 40. Dülger, I. (2004). Case Study on Turkey: Rapid Coverage for Compulsory Education Program. In Scaling Up Poverty Reduction (pp. 1–46). Shanghai, China. Filmer, D., Hasan, A., & Pritchett, L. (2006). A Millennium Learning Goal: Measuring Real Progress in Education. SSRN Electronic Journal. http://doi.org/10.2139/ssrn.982968 Gur, B., Celik, Z., & Ozoglu, M. (2012). Policy options for Turkey: a critique of the interpretation and utilization of PISA results in Turkey. Journal of Education Policy, 27(1), 1–21. Hanushek, E. A., & Woessmann, L. (2011a). Sample Selectivity and the Validity of International Student Achievement Tests in Economic Research. Economics Letters. Retrieved from http://www.sciencedirect.com/science/article/pii/S0165176510003605 Hanushek, E., & Woessmann, L. (2008). The role of cognitive skills in economic development. Journal of Economic Literature, 46(3), 607–668. Retrieved from http://www.jstor.org/stable/27647039 Lewin, K. M., & Little, A. W. (2011). Access to education revisited: Equity, drop out and transitions to secondary school in South Asia and Sub-Saharan Africa. International Journal of Educational Development, 31(4), 333–337. OECD. (2004). PISA Report 2003: Learning for Tomorrow’s World. Paris: Organization for Economic Cooperation and Development. OECD. (2012b). PISA 2012 Results: Excellence through Equity: Giving Every Student the Chance to Succeed (Vol.2). Paris: Organization for Economic Cooperation and Development. Pritchett, L. (2013). The Rebirth of Education: Schooling Ain’t Learning. 2013. Washington, DC.: Center for Global Development. Rotberg, I. (1995). Myths About Test Score Comparisons. Science, 270(5241), 1446–1448. http://doi.org/10.1126/science.270.5241.1446 Spaull, N., & Taylor, S. (2015). Access to What? Creating a Composite Measure of Educational Quantity and Educational Quality for 11 African Countries. Comparative Education Review, 59(1), 133–165. Taylor, S., & Spaull, N. (2015). Measuring access to learning over a period of increased access to schooling: The case of Southern and Eastern Africa since 2000. International Journal of Educational Development, 41, 47–59. UNESCO Institute for Statistics. (2010). Measuring educational participation: Analysis of data quality and methodology based on ten studies (Technical Paper 4).

Author Information

Nicholas Spaull (presenting / submitting)

Stellenbosch University

Economics

Stellenbosch

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.