Session Information
23 SES 02 B, Policies and Practices of Performativity and Assessment (Part 2)
Paper Session continued from 23 SES 01 B
Contribution
A number of scholars have been providing excellent critique of the influence of the rankings and global metrics produced by the OECD’s Programme of International Student Assessment (PISA) and other global comparative assessments (for example, Sellar & Lingard, 2013). There has been less critique and discussion of the secondary analysis that is done using the databases produced by these large-scale international assessments (Author, xxxx). Neither the practices of secondary analysis, which involves the use of existing data to ask new questions (Glass, 1976), nor their influence on policy, have drawn much attention from policy critics. This speculative paper fills this gap in part by taking up an examination of secondary analysis of large-scale numeric comparisons as a knowledge practice.
With the establishment of an extensive infrastructure for collecting data on an on-going basis, many national and international agencies and think tanks have been generating large databases over the last couple of decades. The investment required to generate such databases then leads to encouraging the use of these databases for secondary analysis. In education, the OECD’s Programme for International Student Assessment provides its data for free to anyone who would like to use it. It has instituted the Thomas J. Alexander Fellowship which facilitates and encourages scholars to use the PISA database for secondary analysis. The OECD itself produces, through PISA and Education at a Glance and other sources, thematic issues using secondary analyses.
Using the theoretical resources of Science and Technology Studies (STS), particularly the concepts advanced by Tarde and Latour on quantification; the study of two published secondary analyses; and extensive interview data in which measurement experts explain the complexities and intricacies of the PISA database; this paper argues that secondary analysis of such databases as PISA may be mathematically defensible, but it could be ontologically absurd.
This presentation advances three main arguments. First, it claims that the synoptic, comparative view taken by PISA’s global comparisons may be useful to examine societies rather than individuals. Such a distant gaze denies access to the relations between individuals and societies which are key to understanding groups and societies. Second, scientific rigour is maintained as long as the chain of translation allows translations to occur in both directions (Latour, 1999). In the case of the PISA database, the objects are so radically flattened by the time they get to the database, that they lose connection with the luxurious ontologies they mean to represent. Finally, the highly abstracted objects in the PISA database are able to freely combine with other, similarly flattened objects. They develop promiscuous and possibly illegitimate relations with other similar objects to produce knowledge that might be mathematically defensible, but is ontologically absurd.
Although there are spirited debates about the relative merits of particular methodologies or analytical choices among measurement experts, the challenges posed in these discussions come from within the field, where established methodologies and associated orthodoxies are passed down from mentor to student, often as tacit knowledge, unexamined and often unquestioned. In this regard, the observations of a sociologist of science could offer the field some new questions to ponder. Whether or not the critique offered here is accurate in every detail, the contention is that there is value in moving the internal debates about secondary analysis to a larger space in which this practice can be scrutinised both for its practicality and its epistemological and ontological soundness.
Given the extensive use of the secondary analysis of PISA in Europe, questions concerning the validity and ontological soundness of these data are of great relevance to European scholars in education, assessment and measurement.
Method
Expected Outcomes
References
(Author, xxxx). Glass, G. V. (1976). Primary, Secondary and Meta-Analysis of Research. Laboratory of Educational Research, University of Colorado. Latour, B. (1986). Visualisation and Cognition: Drawing Things Together. In H. Kuklick (Ed.), Knowledge and Society - Studies in the Sociology of Culture Past and Present (Vol. 6, pp. 1-40): Jai Press. Latour, B. (1999). Padora's Hope - Essays on the Reality of Science Studies. Cambridge, MA; London, England: Harvard University Press. Latour, B. (2009). Tarde's Idea of Quantification. In M. Candea (Ed.), The Social After Gabriel Tarde: Debates and Assessments (pp. 145-162). London: Routledge. Porter, T. M. (2012). Thin Description: Surface and Depth in Science and Science Studies. Osiris, 27, 209-226.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.