How did International Large-Scale Assessment (ILSA) contracting emerge and develop? And what is the legacy of this history? 

In studying the history of education data, research has focused on assessment practices (i.e. Hutt and Schneider 2018) and governance uses (Ozga 2009; Merry 2011; Moss 2014). Until recently, scholarship was less concerned with the making of data despite Science and Technology Studies underlining that science (this includes data) is politics by other means (Latour and Woolgar 1979; Latour 1987, 1999), thus begging for scholoarly investigation. This paper contributes to literature on the history of data, by rendering visible the history and legacy of a group of actors who have so far remained invisible. Although ILSAs are administered by the International Association for the Evaluation of Educational Achievement (IEA) and the Organization for Economic Cooperation and Development (OECD), they are mostly developed, implemented and analysed by ILSA contractors. Until today, contractors’ involvement has not been studied. This paper studies the emergence of ILSA contracting by looking into the first ILSAs at the IEA, which came under pressure to keep up with assessment developments in the USA in the 1980s. The paper then focuses on the development of new assessment approaches in the USA that were picked up in adult literacy assessments at the OECD. Finally, the paper analyses the development of the Programme for International Student Assessment (PISA) contracting. To show how the legacy of this history still shapes current ILSAs, the paper analyses how PISA contracting evolved. This history shows how individuals and organizations, which had previously been collaborating in academic projects at the IEA, won the first ILSA contracts by virtue of accumulated capitals and interpersonal trust that put them at an advantage over potential ILSA contractors. What comes to the surface are government interests, emotional bonds, and personal struggles. 

To analyse how ILSA contracting emerged and evolved (between 1960 and 2020), I draw on Bourdieu’s (1993) concept of field - described as a dynamic social space of struggles with its own laws of functioning and unequally distributed power - to understand how contactors relate to one another; and actors’ habitus. The paper also draws on business network approach, which recognizes the importance of networks and the position of actors and changes of position within networks (Ford and Håkansson 2013). Finally, the paper draws on Huang and Wilkinson (2013) to analyse the dynamics and evolution of trust in ILSA contracting. Huang and Wilkinson describe both cognitive and affective trust as key to understanding business relationships and behaviour. Cognitive trust can be described as the ‘evaluation of the competence, responsibility and dependability’ (2013: 456) of actors, while affective trust is described as emotional bonds and ‘the belief that an exchange partner cares about your welfare, will act positively towards it and take care to avoid harming it’ (2013: 456). Trust can be interpersonal and interorganizational (Zaheer et al. 1998). 

The paper is based on a qualitative research design, using Ball’s (2016) network ethnography. The approach suggests mapping, following, questioning, and visiting people and nodal actors, their lives, stories, conflicts, money, and things. Junemann et al. describe it as focusing on ‘the content, nature, and meaning of the exchanges and transactions between network participants, the roles, actions, motivations, discourses, and resources of the different actors involved’ (2016: 539). For the purposes of this paper, I focused in particular on PISA, TIMSS, and PIRLS contractors. These three ILSAs were chosen because each has had multiple implementations: seven cycles of PISA plus two in progress, seven cycles of TIMSS, and four cycles of PIRLS. By juxtaposing the data from each implementation chronologically, I was able to follow the actors and identify key actors, patterns in relationships over time, changes or anomalies that might hint at significant struggles. To map the ILSA contractors, I drew on IEA and OECD ILSA technical reports between 1990 to 2020. I then used this data to visualise the ILSA contractors as a topology of nodes connected with lines (representing the actors and relationships). The choice of interviewees was developed as an iterative process, with interviewees helping to interpret information in the documents, and documents pointing to potential interviewees and struggles. Approximately 35 interviews were carried out with high-level staff at IEA, OECD and contracting organizations. Almost all interviews were carried out over online platforms (i.e. Teams) in 2020. Interviewees are identified through a combination of randomly-assigned letters, institutional affiliation, or no letter where anonymity could be compromised.

TThe paper shows how the current ILSA contractors emerged from a loosely structured academic project at IEA (approximately from the 1960s to early 1990s) which had a lasting influence over ILSA contracting in the subsequent decades. In the early years, when a passionate network of individuals with big research ideas came together, emotional bonds and feuds were developed, theoretical and methodological choices were made, and insider knowledge and experience were accumulated. In Bourdieu terms, different forms of capitals were developed in these early years, determining how the first ILSA contracts were distributed when substantial funding became available. The individuals in this network developed cultural knowledge, competences and dispositions that were key to obtaining and carrying out contracts. The reliance on the good will of experts who were willing to donate their time suggests economic capital did not shape practices in this space, until substantial US funding was secured. When ILSAs were funded, new practices and struggles emerged. In particular, the paper highlights the importance of interpersonal, affective trust and shared history. The IEA has continued to work as it did in TIMSS 1995, openly relying on trust and shared history, whereas the OECD does this under the guise of global competition. With PISA’s formal bidding process, former interrelationships between contractors ended and contractors regrouped, formalizing former struggles between individuals and organizations. Affective trust but also the lack of it where relationships have ended, continue to shape ILSA contracting as the structure of contractors remains mostly stable, despite ILSA developments. Finally, the paper argues that government interests, interpersonal and interorganizational struggles, and emotional bonds are embedded in the data as they determined and continue to determine who develops ILSAs.

Ball, S. (2016). Following policy: networks, network ethnography and education policy mobilities. Journal of Education Policy 31(5): 549-566. Bengtsson, M. & Kock, S. (1999). Cooperation and competition in relationships between competitors in business networks. The Journal of Business & Industrial Marketing; Santa Barbara, 14(3), 178-194. Bourdieu, Pierre. 1993. The field of cultural production. New York: Columbia University Press. Ford, D., & Håkansson, H. (2013). Competition in business networks. Industrial Marketing Management, 42, 1017–1024. Moss, Gemma. 2014 “Putting literacy attainment data in context: examining the past in search of the present.”Comparative Education,50:3,357 - 373,DOI:10.1080/03050068.2014.921369 Huang, Y. & Wilkinson, I. F. (2013). The dynamics and evolution of trust in business relationships. Industrial Marketing Management, 42, 455-46 Hutt, Ethan and Jack Schneider. 2018.“A thin line between love and hate: educational assessment in the United States in Assessment Cultures – Historical Perspectives, 235 - 258. Berlin: Peter Lang. Junemann, C., Ball, S. J., & Santori, D. (2016).Joined-up Policy: network connectivity and global education governance. In K. Mundi, A. Green, B. Lingard, & A. Verger (Eds.),Handbook of Global Education Policy(pp. 535-553). Wiley-Blackwell. Latour, B. (1987). Science in action: How to follow scientists and engineers through society. London: Harvard University Press. Latour, B. (1999). Pandora’s hope: Essays on the reality of science studies. Cambridge, MA and Cambridge: Harvard University Press. Latour, B., & Woolgar, S. (1979). Laboratory life – The construction of scientific facts. Princeton, NJ: Princeton University Press. Merry, Sally Engle. 2011. “Measuring the World: Indicators, Human Rights, and Global Governance.” Current Anthropology, Vol. 52, No. S3, pp. S83-S95 Ozga, Jenny. 2009. “Governing education through data in England: from regulation to self‐evaluation.” Journal of Education Policy,24:2,149 162,DOI:10.1080/02680930902733121