Contemporary education research is characterized by the generation of large datasets, such as those produced by the Programme for International Student Assessment (PISA). The availability of such datasets has increased the appetite for secondary analysis – i.e., using existing databases to ask new questions or re-examine questions using new techniques (Glass, 1976). Secondary analysis allows researchers with even modest funds to produce research of significance because large databases are often available at a nominal cost (Smith, 2006). Recognizing its importance, researchers are being trained in secondary analysis as a matter of national priority in the US and the UK. Given its increasing importance and use, it is crucial to analyze the possibilities, promises, and limits of secondary analysis and its ability to inform policy. This paper reports on the tensions, dilemmas and methodological conundrums that confront researchers engaged in secondary analysis of PISA. It uses the theoretical resources of Science and Technology Studies (STS) to engage with the “politics of method” in processes of knowledge making. Based on interviews with secondary analysts in the US, Australia, and New Zealand, it elaborates the complexities of this epistemic practice. Secondary analysis presents constraints on the types of questions that can be asked. Assumptions made in the primary data collection may be invisible to secondary users. The variables significant to the original purpose may not coincide with the variables in the secondary research. Differences in definitions, nomenclatures, and the indicators used in the measurement may significantly impact the validity of the secondary research. Methodological decisions to resolve these conundrums are often only visible within the measurement community, whereas their results circulate far more widely as certainties and precise calculations. This paper argues that it is important to keep the conundrums and uncertainties of research practices visible and open for discussion.
