The UK Government has produced a ‘Resilience Framework’, which aims to ensure the country’s prosperity by having a national infrastructure that is better equipped to tackle adverse events such as, but not limited to, pandemics, weather events and cybersecurity attacks. The qualification system is now being seen as an essential part of government infrastructure. Delivering qualifications is a complex, high-volume, distributed activity involving multiple actors across organisations with a range of relationships. Over 26 million examination scripts and coursework tasks are dealt with in the system in England annually, for qualifications taken by 16- and 18-year-olds. There is a quasi-market of four examination boards who offer academic qualifications and there are hundreds of organisations offering vocational qualifications. In this paper, we report on a project that sought to investigate not only how resilient the qualification system in England is, but what such resilience might mean. Using publicly available documents, input from an expert advisory group (10 people) and elite interviews with 21 assessment insiders, we analysed the resilience of England’s qualification system. The recent exams crisis created by the pandemic was one focus, but we explored resilience more broadly. To define resilience, we drew upon definitions published in the literature for other complex, distributed systems (food, healthcare and utilities). Systematic reviews of the term in other fields pointed out that the term ‘resilience’ is fluid in its meaning. For the purposes of this research project, we defined resilience as, ‘The capacity of the qualification system and its units at multiple levels to actively engage with, manage and learn from periods of change and unforeseen disturbances to deliver timely and sufficiently accurate, trusted, and valid grades to fulfil their purpose(s) now and in the future.’ Interviewees included regulators (3), civil servants (2), academics (2), teacher leaders or union representatives (5), individuals with think tanks, communications or PR perspectives (4), and exam board or awarding organisation insiders (5). We explored how qualification system resilience might be defined, its characteristics, resilience of the qualification system during the pandemic, threats to resilience and what countermeasures might be taken to them. Our interview data showed no consensus on the definition of resilience amongst the industry insider participants. Nor was there agreement on whether the system is currently resilient. Various proposed countermeasures for perceived lack of resilience have been publicly debated (teacher assessment, modular examinations, digitalisation). Our analysis outlines the risks, as well as potential benefits of each of these proposals. We conclude that the term resilience must be defined in relation to specified aims. Many threats to resilience were identified, including political pressure - a key feature of the 2020 exam policies. Fundamentally redesigning the system for resilience to unlikely catastrophic events would be a mistake. The cause of the 2020 crisis is best described as poor policy rather than as system fragility. Prospects for managing policy mistakes through government agencies (‘quangos’) are not encouraging due to the relationship with government. This case demonstrates fundamental weaknesses for the UK in delivering resilience, in the qualifications system and beyond. The role of politics in educational assessment policy differs across nations. This case serves to illustrate how the management of political agendas and policy mistakes is integral to managing education systems. This nebulous concept is useful in political terms, as policymakers can point to a lack of system resilience, rather than identifying issues as policy failures. Pointing to resilience is a useful vehicle for shifting policy evaluation criteria and responsibility for those. Qualification systems may be at particular risk of political pressures because examination grades are symbolic and intangible; their value is socially constructed.    

Documentary analysis included journal publications, grey literature, parliamentary Select Committee transcripts and reports and statistical publications. This led to a working project definition of resilience; depiction of the qualifications system; understanding of previously documented qualification crises; and consideration of various potential countermeasures to the problems encountered during the pandemic. An advisory group comprising 10 experts was formed to advise on methodology, conceptual development of the project and interpretation of research findings. Members were selected for their knowledge of regulation, awarding body research, government policy, understanding of the school and college sector or for their academic expertise. An innovative use of this expert group was in collecting data through the advisory meetings. Twenty-one interview participants were recruited. Participants were selected to give a range of political and ideological perspectives, including individuals openly supportive of the system, as well as those calling for reform. Interviewees included regulators (3), civil servants (2), academics (2), teacher leaders or union representatives (5), individuals with think tank, communications or PR perspectives (4), and exam board or awarding organisation insiders (5). Interviews were conducted online and were transcribed. Whilst an interview schedule was used to guide the interviews, this was used flexibly. Perspectives of elites – particularly bureaucratic elites – were considered when reflecting on the positionality of the data and of us as insider-outsider researchers. At the end of each interview the main themes of the interview were summarised, giving the participant an opportunity to correct, clarify or extend ideas. Transcripts were coded deductively by three researchers using a codebook. In a training phase, the researchers independently coded the same three transcripts, randomly selected from the sample of 21. These were then compared for inter-rater agreement, and a coding meeting was held to reflect on the process and the clarity and comprehensiveness of the codebook. One code was revised for clarity. The remaining 18 transcripts were divided randomly between the researchers who again coded independently before a final analysis meeting to discuss the results. There was near-perfect agreement between the coders at each stage. Interrater agreement was calculated as between 95-100% for all but one code - “threats to resilience”. The interrater reliability for this code ranged from 80-88% and disagreements were straightforward to resolve in coding meetings. Data was synthesised across the datasets by code and research question.

Our definition of qualification system resilience was broadly supported by the interviewees. They commented on threats related to unforeseen circumstances and periods of planned change, including qualification reform. These experts discussed the need to manage the timely delivery of sufficiently valid and reliable grades. Some experts also reflected on the need for public trust in – or at least societal acceptance of – grades. None of the interviewees questioned the very notion of resilience, but some recognised the political capital to be gained from claims about the weaknesses of the system. Having a clear definition of resilience is one step towards being able to debate what are realistic expectations of the system. Afterall, as some interviewees pointed out, there is a limit to the extent to which the delivery of qualifications can be resilient to all potential threats, and a system that functions well under extreme circumstances is unlikely to be suitable in normal times. Nonetheless, interviewees suggested changes to the system that would, in their view, improve resilience: teacher assessment, modular examinations and digitalisation. Political interference in the system was identified as a significant threat to resilience. Although government agencies have been established to manage activities where direct political control is undesirable, they are nonetheless still under political control. The concept of resilience meant different things to different stakeholders, which was a lever for creating change agendas aligned with interviewees’ values and ideologies. None of the suggested countermeasures for improving resilience come without their own risks to resilience. Selecting between these policies is therefore a matter of values and politics, rather than a neutral, technocratic procedure. As insider researchers ourselves, we conclude that who defines the term resilience is key to interpretation of the resilience of the system.

Baird, J.-A., & Coxell, A. (2009). Policy, Latent Error and Systemic Examination Failures. CADMO, XVII(2), 105–122. Baird, J.-A., Isaacs, T., Opposs, D., & Gray, L. (2018). Examination Standards: How Measures and Meanings Differ Around the World. UCL, IOE Press. Baird, J.-A., & Lee-Kelley, L. (2009). The dearth of managerialism in implementation of national examinations policy. Journal of Education Policy, 24(1), 55–81. Bergström, J., van Winsen, R., & Henriqson, E. (2015). On the rationale of resilience in the domain of safety: A literature review. Reliability Engineering & System Safety, 141, 131–141. https://doi.org/10.1016/j.ress.2015.03.008 Biddle, L., Wahedi, K., & Bozorgmehr, K. (2020). Health system resilience: A literature review of empirical research. Health Policy and Planning, 35(8), 1084–1109. https://doi.org/10.1093/heapol/czaa032 Birkmann, J., Dech, S., Hirzinger, G., Klein, R., Klüpfel, H., Lehmann, F., Mott, C., Nagel, K., Schlurmann, T., Setiadi, N. J., Siegert, F., & Strunz, G. (2006). Measuring vulnerability to promote disaster resilient societies: Conceptual frameworks and definitions. http://collections.unu.edu/view/UNU:2793 Bowe, R., Ball, S., J., & Gold, A. (1992). Reforming education and changing schools. Routledge. Cabinet Office. (2022). The UK Government Resilience Framework. UK Government. https://www.gov.uk/government/publications/the-uk-government-resilience-framework/the-uk-government-resilience-framework-html Hammerstein, S., König, C., Dreisöner, T., & Frey, A. (2021). Effects of COVID-19-related school closures on student achievement—A systematic review. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.746289 Hayward, L., Baird, J.-A., Allan, S., Godfrey-Faussett, T., Hutchinson, C., MacIntosh, E., Randhawa, A., Spencer, E., & Wiseman-Orr, M. L. (2023). National qualifications in Scotland: A lightning rod for public concern about equity during the pandemic. European Journal of Education, 58(1), 83–97. https://doi.org/10.1111/ejed.12543 Humbert, C., & Joseph, J. (2019). Introduction: The politics of resilience: problematising current approaches. Resilience, 7(3), 215–223. https://doi.org/10.1080/21693293.2019.1613738 Kelly, A. (2014). Monopolising the examining board system in England: A theoretical perspective in support of reform. Journal of Education Policy, 29(1), 44–57. https://doi.org/10.1080/02680939.2013.790078 McCaig, C. (2003). School Exams: Leavers in Panic. Parliamentary Affairs, 56(3), 471–489. https://doi.org/10.1093/parlij/gsg101 Opposs, D., Baird, J.-A., Chankseliani, M., Stobart, G., Kaushik, A., McManus, H., & Johnson, D. (2020). Governance structure and standard setting in educational assessment. Assessment in Education: Principles, Policy & Practice, 27(2), 192–214. https://doi.org/10.1080/0969594X.2020.1730766 Ozga, J., Baird, J.-A., Saville, L., Arnott, M., & Hell, N. (2023). Knowledge, expertise and policy in the examinations crisis in England. Oxford Review of Education, 1–19. https://doi.org/10.1080/03054985.2022.2158071 Wisner, B., Blaikie, P., Cannon, T., & Davis, I. (2003). At Risk: Natural Hazards, People’s Vulnerability and Disasters. Routledge.