As the education agenda of global agencies changed after 2015 to emphasise minimum standards of quality for all countries to be delivered by 2030, the OECD has sought to expand its most successful comparative instrument, the Programme for International Student Assessment (PISA), to include low- and middle-income countries (LMICs). In 2014, it introduced PISA for Development (PISA-D) as the means to establish PISA as a universal measure of learning, and in 2020, it declared PISA-D a success. The most innovative feature of PISA-D was that the assessment included out-of-school youth (OOSY); that task was sub-contracted to Educational Testing Service (ETS). The OOSY population is a geographically dispersed group which present considerable challenges to any researchers seeking to access them. Given this, we ask: who did the OECD assess? More specifically, how did the OECD define the target population of PISA-D out-of-school assessment, what was the sampling frame, and were they accurately represented in the PISA-D OOSY sample?

Much of the existing literature on the OECD influence is based on different theoretical positions. These differences in perspective have real consequences, however, often determining which legitimation dynamics researchers see and which they overlook. In this paper, we seek to adopt a holistic approach by drawing on Suchman’s (1995) framework for analysing the multiple sources of organisational legitimacy and the means by which it is promoted and repaired. 

In applying Suchman’s framework, we argue that PISA-D was a macro level exercise designed to legitimate the OECD’s extension of PISA into LMICs and to establish its role in a new arena. The incorporation of OOSY in the assessment was a key micro level endeavour which would allow the OECD to achieve that end and, if not done properly, would challenge aspects of its legitimacy. For example, not assessing sufficient OOSY would debase the quality of the OECD’s products and services; this would also damage the OECD’s moral claims with regard to monitoring the attainment of the SDGs and promoting an inclusive approach. In parallel, at the cognitive level, this would challenge the whole logic of the novelty and value of PISA-D. Overall, the successful identification and assessment of OOSY was vital to ensuring its legitimacy. This would require the OECD to either address the considerable difficulties of accessing OOSY or find a tactical solution which obscured the many challenges to its legitimacy.

Suchman (1995) also analyses how organisations respond to challenges to their legitimacy and identifies three broad approaches: (a) offer normalising accounts; (b) restructure, and (c) don’t panic. He suggested that although legitimacy crises may coalesce around performance issues, most challenges ultimately rest on failures of meaning, where ‘audiences begin to suspect that putatively desirable outputs are hazards, that putatively efficacious procedures are tricks, or that putatively genuine structures are facades’ (1995, 597). Consequently, the initial task in mending a breach of legitimacy usually will be ‘to formulate a normalising account’ that separates the threatening revelation from larger assessments of the organisation as a whole. He identified ‘justifications’ and ‘explanations’ as the two main types of normalising accounts. Suchman also noted that organisations may also re-establish legitimacy through micro-level strategic restructuring, in the sense that ‘narrowly tailored changes that mesh with equally focused normalising accounts can serve as effective damage-containment techniques’ (ibid., 598).

To understand what challenges the OECD encountered and how it managed to address them, we firstly draw on two categories of documents: the first are the UNICEF and UNESCO Institute for Statistics (UIS) publications on the Out-of-school Children Initiative (OOSCI) and Lewin’s (2011) work as part of the Consortium for Research on Educational Access, Transitions and Equity (CREATE) initiative which provide the standard approaches to identifying OOSY and describing their characteristics which PISA-D draws upon. The second are the OECD publications which explain the PISA-D out-of-school sample design and selection plans , and which present the final results . We also draw on three interviews with: a key member of the PISA-D team at the OECD, a technical expert who had undertaken OOSY surveys, and a lead analyst from one of the piloting nations.

We argue that, as an organisation with no experience with assessing OOSY and working in poorer nations, the OECD was faced by a ‘disruptive event’ (Suchman 1995) as it was unable to effectively sample youth based on their initial definition. This event, if not addressed immediately, would interrupt its ongoing PISA-D legitimation activities and may severely deplete its long-term legitimacy. In line with Suchman’s (1995) analysis of how organisations respond to challenges to their legitimacy, we demonstrate that the OECD pursued a normalising strategy by tailoring and justifying how OOSY were defined and by minimising coverage of its tactical changes. Consequently, it avoided addressing the many problems which face researchers on OOSY by quietly imposing a sampling frame which matched its available sources of data and established methodologies. The analysis builds on our earlier work which identified the broader strategies that the OECD employed to create the legitimacy to monitor SDG 4 (Li and Morris 2022) and extends it by focusing on legitimacy maintenance and repair work. It also contributes to the important work of others who have critiqued the validity and impact of various assessments undertaken by global agencies.

Addey, Camilla. 2017. ‘Golden Relics & Historical Standards: How the OECD Is Expanding Global Education Governance through PISA for Development’. Critical Studies in Education 0 (0): 1–15. https://doi.org/10.1080/17508487.2017.1352006. Auld, Euan, and Paul Morris. 2021. ‘A NeverEnding Story: Tracing the OECD’s Evolving Narratives within a Global Development Complex’. Globalisation, Societies and Education 19 (2): 183–97. https://doi.org/10.1080/14767724.2021.1882959. Berten, John, and Matthias Kranke. 2019. ‘Studying Anticipatory Practices of International Organisations: A Framework for Analysis’. Framing Paper for Workshop on Anticipatory Governance at 6th European Workshops in International Studies. Kraków. Carr-Hill, Roy. 2013. ‘Missing Millions and Measuring Development Progress’. World Development 46 (June): 30–44. https://doi.org/10.1016/j.worlddev.2012.12.017. Grek, Sotiria. 2009. ‘Governing by Numbers: The PISA “Effect” in Europe’. Journal of Education Policy 24 (1): 23–37. https://doi.org/10.1080/02680930802412669. Grey, Sue, and Paul Morris. 2018. ‘PISA: Multiple “Truths” and Mediatised Global Governance’. Comparative Education 54 (2): 109–31. https://doi.org/10.1080/03050068.2018.1425243. Lewin, Keith. 2011. ‘Making Rights Realities: Researching Educational Access, Transitions and Equity’. Project Report. Brighton: University of Sussex. http://www.create-rpc.org/pdf_documents/Making-Rights-Realities-Keith-Lewin-September-2011.pdf. Li, Xiaomin, and Euan Auld. 2020. ‘A Historical Perspective on the OECD’s “Humanitarian Turn”: PISA for Development and the Learning Framework 2030’. Comparative Education 56 (4): 503–21. https://doi.org/10.1080/03050068.2020.1781397. Hamilton, Mary. 2017. ‘How International Large-Scale Skills Assessments Engage with National Actors: Mobilising Networks through Policy, Media and Public Knowledge’. Critical Studies in Education 58 (3): 280–94. https://doi.org/10.1080/17508487.2017.1330761. Martens, Kerstin. 2007. ‘How to Become an Influential Actor - The “comparative Turn” in OECD Education Policy’. In New Arenas of Education Governance: The Impact of International Organisations and Markets on Educational Policy Making, edited by Kerstin Martens, Alessandra Rusconi, and Kathrin Leuze. Basingstoke: Macmillan. Li, Xiaomin, and Paul Morris. 2022. ‘Generating and Managing Legitimacy: How the OECD Established Its Role in Monitoring Sustainable Development Goal 4’. Compare: A Journal of Comparative and International Education 0 (0): 1–18. https://doi.org/10.1080/03057925.2022.2142038. Robertson, Susan L. 2020. ‘Guardians of the Future: International Organisations, Anticipatory Governance and Education’. Presented at the International Webinar on UNESCO’s and OECD’s Ambition to Govern the Future of Education, Copenhagen, April 23. Suchman, Mark C. 1995. ‘Managing Legitimacy: Strategic and Institutional Approaches’. The Academy of Management Review 20 (3): 571–610. https://doi.org/10.2307/258788. Zapp, Mike. 2020. ‘The Authority of Science and the Legitimacy of International Organisations: OECD, UNESCO and World Bank in Global Education Governance’. Compare: A Journal of Comparative and International Education, 1–20. https://doi.org/10.1080/03057925.2019.1702503.