Over the past decade, two large scale international surveys have been carried out both assessing students’ achievement in mathematics and science, IEA TIMSS and OECD PISA, providing results that have not always been consistent. There has certainly been some interest in comparing the two surveys but, as Hutchinson and Schagen (2006) remark, there still is little formal research investigating the reasons for differences in the relative ranking of countries between the two surveys.

Certainly the differences can be linked to different survey designs (TIMSS is grade-based and PISA is age-based), different assessment frameworks (the former stating the survey as aiming at improving teaching and learning and the latter at assessing how well 15-year-olds are prepared for their future occupational and academic life) (Mullis et al. 2007; OECD, 2009) and different weights of the item characteristics both in content and in format (Ruddock et al., 2006; Neidorf et al., 2006).

But there undoubtedly are substantial similarities. Just to mention three of them: as for the item classification variables, PISA “content” and “process” correspond largely to TIMSS content and cognitive dimensions; both surveys use a booklet rotation design that, thanks to IRT models, allows to exploit a large pool of items without  burdening individual students; both studies, in order to take into consideration the basic uncertainty involved in the score estimates, generate five plausible values for each student and each scale (OECD, 2011; Olson et al., 2008). Therefore one would expect to obtain similar insights and/or to easily explain apparent discrepancies.

Drawing from previous evidence (Hutchinson and Schagen, 2007; Wu, 2009), mostly based on data from TIMSS 2003 and PISA 2003, this paper will try to compare results in PISA 2009 and TIMSS 2007 of countries participating in both studies, considering that they basically involved the same cohort of students and therefore expecting to get much more evidence of similarity than of discrepancies. A tentative exploration of which countries do better in one study with respect to the other, thus implying an improvement or a deterioration of the students’ performance, will also be made. Then, since national and international surveys generally show that in Italy performance levels are strongly differentiated in different geographical areas within the country, we will go into Italian results in order to explore possible differences in the two surveys related to geographical areas.

OECD, (2009), PISA 2009 Assessment Framework - Key Competencies in Reading, Mathematics and Science, Paris: OECD. OECD (2011), PISA 2009 Technical Report (Preliminary Version), Paris: OECD. Olson, J.F., Martin, M.O., & Mullis, I.V.S. (Eds.). (2008). TIMSS 2007 Technical Report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Hutchinson D. and Schagen I., (2007), Comparisons Between PISA and TIMSS – Are We the Man with Two Watches?, in Loveless, T. (Ed.), “Lessons Learned: What International Assessments Tell Us About Math Achievement”, Washington, DC: Brookings Institution Press. Wu M., (2009), “A comparison of PISA and TIMSS 2003 achievement results in mathematics”, Prospects, 39, pp. 33-46.