Session Information
23 SES 09 C, Politics of International Assessment and Tests
Paper Session
Time:
2009-09-30
10:30-12:00
Room:
HG, HS 16
Chair:
Anja Sinikka Heikkinen
Contribution
In an era of increasing uncertainty, education policy makers in Europe and elsewhere seem to be turning to large-scale psychometric assessments such as the PISA survey for assurance and certainty. Statistical evidence is often positioned as a neutral, rigorous and objective. It is seen as a basis for calculating return on investment and identifying areas for reform in a straightforward and rational manner, without becoming mired in politics and rhetoric and ideological arguments. Numbers, especially those produced by large, international, expert bodies, are seen as reliable and politics-free.
Yet there is nothing self-evident about the neutrality and objectivity of statistical evidence. The idea that numerical accounts are somehow different to narrative accounts is itself to be challenged (Munro 2001).
How do statistical accounts perform themselves as clear and unproblematic? What are the processes by which surveys such as PISA are able to bring together such a range of data and weave it all together into a single spatio-temporal frame? How do these studies produce coherence and become seemingly self-evident representations of reality? How do they ‘pack the world into words’ (Latour 1999)?
This paper provides a behind-the-scenes look at the making of certainty in educational statistics. Using the OECD’s Programme for International Student Assessment (PISA) as a case study and the material semiotic lens of actor-network theory, it traces the associations, negotiations and translations that go into the production of a coherent account of the relative efficiencies of educational systems around the world.
Method
Using the anti-foundationalist approach of actor-network theory, this analysis seeks explanations in the empirical. Based on PISA documents and in-depth interviews with two senior PISA officials who are also psychometricians, the study gets behind the certain, tidy world of PISA statistics to explore the processes which lead it from a state of unknowing and uncertainty, to one of assurance and certainty. Using a material-semiotic sensibility, and based in particular on Latour’s theory of ‘circulating reference’(Latour 1999), it traces, step-by-step, the twin processes of reduction and expansion by which actors are detached from their contexts, made mobile, translated into commensurate entities and attached to other entities, resulting in the PISA league tables, graphs and reports. The paper is based on an ongoing doctoral study of the use of PISA in Australian education policy.
Expected Outcomes
This analysis finds that PISA is as much a performance of reality as a representation of it. PISA performs realities based on the valuing of certain skills, and restricted by the frameworks it uses and the pragmatics of what is conveniently measurable on a large scale. Other frameworks may highlight issues the PISA framework cannot recognise. The debate is not about the validity and reliability of PISA; this is not a debunking exercise. But when one type of reality starts to congeal as the only accurate representation, other accounts, which may highlight other issues, get stifled. Given that no account can be entirely ‘accurate’, a multiplicity of accounts would make for a more informed basis for policy making. For this reason ‘certainties’ performed by statistical accounts need to be challenged. This paper hopes to provide such a challenge.
References
Latour, B. (1999). Pandora's Hope: Essays on the Reality of Science Studies. Cambridge, London, Harvard University Press. Munro, R. (2001). "Calling for accounts: numbers, monsters and membership." The Sociological Review 49(4): 473-493.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.