While international large-scale assessments in education (ILSA) have existed for more than 50 years and are valuable source of information for researchers and policy-makers, still they are often used just for producing league-tables. Such comparisons of student achievement across the participating countries are recognized as legitimate method to estimate the development and productivity, but often turn into just “horse-ranking” (Wiseman, 2010). Such limited use is often supported by the media, although in many cases the differences are insignificant. These practices often turn the use of results to their abuse (Bos & Schwippert, 2003). The purpose of this presentation is to discuss the general practice of using league-tables for reporting the results versus the use of ILSA data for in-depth analysis. The general questions this presentation has are: 1. What are the limitations of the single numbers, as found in league-tables, to draw valid conclusions? 2. How the secondary analysis can lead to in making relevant decisions and drawing adequate course of reforms? There are several issues with reporting single numbers in league-tables. First, educational systems are only ranked relative to other countries (Rutkowski & Prusinski, 2011), but participating countries change from one cycle to the other. Second, the differences between educational systems are not always significant (Bos & Schwippert, 2003), so the differences are often in countries’ position in the table, not in their students’ achievement. Third, single numbers do not represent neither the whole spectrum of the achievement distribution, nor the factors associated with them, although being convenient and attractive for the media and politicians (Takayama, 2008). However, the utility of the data is in providing comprehensive evidence 1) of existing problems and needs that require public intervention, fundamental for improving the policy responses; and 2) on the process of objective setting to better understand the problems to be addressed. These two aspects of evidence are usually neglected (Sanderson, 2002). The description above lead to the main point of evidence based policy-making in education using ILSA data: in-depth, analysis of the rich contextual ILSA data can provide understanding of the underlying determinants and consequences of cognitive skills. Such analyses can reveal if certain relationships are more country-specific or universal, or spurious (Hanushek & Woessmann, 2013). Findings from these analyses can inform policies about the relative effect of factors related with achievement, the trends, needed interventions and course of action, but more importantly, the setting of relevant objectives.

Bos, W., & Schwippert, K. (2003). The Use and Abuse of International Comparative Research on Student Achievement. European Educational Research Journal, 2(4), 559–573. Hanushek, E. A., & Woessmann, L. (2013). The Role of International Assessments of Cognitive Skills in the Analysis of Growth and Development. In M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto (Eds.), The Role of International Large-Scale Assessments: Perspectives from Technology, Economy, and Educational Research (pp. 47–66). Dordrecht: Springer Science + Business Media, LLC. Rutkowski, D., & Prusinski, E. L. (2011). The Limits and Possibilities of International Large-Scale Assessments. Center for Evaluation & Education Policy, 9(2), 1–4. Sanderson, I. (2002). Evaluation, Policy Learning and Evidence-Based Policy Making. Public Administration, 80(1), 1–22. doi:10.1111/1467-9299.00292 Takayama, K. (2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education, 44(4), 387–407. doi:10.1080/03050060802481413 Wiseman, A. W. (2010). Introduction: The Advantages and Disadvantages of National Education Policymaking Informed by International Achievement Studies. In A. W. Wiseman (Ed.), The Impact of International Achievement Studies on National Education Policymaking (pp. xi–xxii). Bradford: Emerald Group Publishing Limited.