A Comparison of Mathematics and Science Performance in TIMSS 2007 and PISA 2009 of Countries Participating in Both Surveys

Author(s):

Elisa Caponera(submitting)Valeria Damiani(presenting)Laura Palmerio(presenting)Cristina Lasorsa

Conference:

ECER 2014

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 03 A, Comparing Large-Scale Assessments across Countries and Domains: Issues in Interpretation and Adaptation Procedures

Paper Session

Time:

2014-09-02

17:15-18:45

Room:

B010 Anfiteatro

Chair:

Jana Strakova

Contribution

Over the past decade, two large scale international surveys have been carried out both assessing students’ achievement in mathematics and science, IEA TIMSS and OECD PISA, providing results that have not always been consistent. There has certainly been some interest in comparing the two surveys but, as Hutchinson and Schagen (2006) remark, there still is little formal research investigating the reasons for differences in the relative ranking of countries between the two surveys.

Certainly the differences can be linked to different survey designs (TIMSS is grade-based and PISA is age-based), different assessment frameworks (the former stating the survey as aiming at improving teaching and learning and the latter at assessing how well 15-year-olds are prepared for their future occupational and academic life) (Mullis et al. 2007; OECD, 2009) and different weights of the item characteristics both in content and in format (Ruddock et al., 2006; Neidorf et al., 2006).

But there undoubtedly are substantial similarities. Just to mention three of them: as for the item classification variables, PISA “content” and “process” correspond largely to TIMSS content and cognitive dimensions; both surveys use a booklet rotation design that, thanks to IRT models, allows to exploit a large pool of items without burdening individual students; both studies, in order to take into consideration the basic uncertainty involved in the score estimates, generate five plausible values for each student and each scale (OECD, 2011; Olson et al., 2008). Therefore one would expect to obtain similar insights and/or to easily explain apparent discrepancies.

Drawing from previous evidence (Hutchinson and Schagen, 2007; Wu, 2009), mostly based on data from TIMSS 2003 and PISA 2003, this paper will try to compare results in PISA 2009 and TIMSS 2007 of countries participating in both studies, considering that they basically involved the same cohort of students and therefore expecting to get much more evidence of similarity than of discrepancies. A tentative exploration of which countries do better in one study with respect to the other, thus implying an improvement or a deterioration of the students’ performance, will also be made. Then, since national and international surveys generally show that in Italy performance levels are strongly differentiated in different geographical areas within the country, we will go into Italian results in order to explore possible differences in the two surveys related to geographical areas.

Method

Data from TIMSS 2007 and PISA 2009 are used in the analyses. Results of Italian students are compared with data from 24 countries that participated in both international studies. We selected only countries with students at eight grade in TIMSS 2007 and with the same age. In order to compare the two different datasets, we previously standardized the studentscores in both math and science test, based on five plausible values derived form IRT analyses, with a mean of 500 and standard deviation of 100. In order to verify whether some countries performed better in one study with respect to the other, a rank order correlation were computed between students' performance in TIMSS and PISA. In Italy, different geographic areas obtain different results in national and international surveys. In order to evaluate differences related to geographic area in Italy, we also calculated a rank order correlation for five geographic areas (North West, North East, Center, South and South and Island), with representative sample in both studies. A multiple regression model was conducted to evidence whether economic indicators and other characteristics at a national level (e.g. GDP, education expenditure) are differentially related to the achievement of students between the two surveys across countries. Analyses were performed with the IBM-SPSS 19.0 software (e.g. Wu, 2009).

Expected Outcomes

Rank order correlation between mathematics achievement in PISA and in TIMSS was calculated across countries. The results shown a high level of correlation. We also obtained the same results for science achievement, even though the coefficient is smaller. We expect to find a high correlation coefficient within geographic areas in Italy. Regarding differences in PISA and TIMSS performance across countries, Singapore, Korea, Chinese Taipei and Japan performed better in TIMSS than in PISA. On the other hand countries such as Slovenia, Norway and Sweden performed better in PISA. The results of a multiple regression model conducted by using the scores on PISA as dependent variable and the scores on TIMSS as independent variable shown a high impact of TIMSS results on PISA survey for both mathematics and science achievement. The results of multiple regression model with economic indicators and other characteristics at a national level as independent variables evidenced different patterns of predictors across countries for the two surveys.

References

OECD, (2009), PISA 2009 Assessment Framework - Key Competencies in Reading, Mathematics and Science, Paris: OECD. OECD (2011), PISA 2009 Technical Report (Preliminary Version), Paris: OECD. Olson, J.F., Martin, M.O., & Mullis, I.V.S. (Eds.). (2008). TIMSS 2007 Technical Report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Hutchinson D. and Schagen I., (2007), Comparisons Between PISA and TIMSS – Are We the Man with Two Watches?, in Loveless, T. (Ed.), “Lessons Learned: What International Assessments Tell Us About Math Achievement”, Washington, DC: Brookings Institution Press. Wu M., (2009), “A comparison of PISA and TIMSS 2003 achievement results in mathematics”, Prospects, 39, pp. 33-46.

Author Information

Elisa Caponera (submitting)

Invalsi

Frascati (Rome)

Valeria Damiani (presenting)

University of Rome Tre, Italy

Laura Palmerio (presenting)

Invalsi, Italy

Cristina Lasorsa