International large-scale assessments (ILSAs) are playing an increasingly important role in decision-making and reforms, both nationally and internationally (e.g. Grek, 2009; Lindblad et al., 2018). One of the most influential ILSAs is the Programme for International Student Assessment (PISA). Given the impact PISA has on educational debate and policy, it is crucial that results are trustworthy. Yet, parallel to an increase in the number of ILSAs, there has been growing validity concerns regarding for example the content being tested, the influence on national educational systems and potential bias due to lack of sample representativeness (Grek, 2009; Jerrim, 2021; Meyer & Benavot, 2013). Relatively few studies, however, have focused on whether students are motivated to do their best in ILSAs as compared with high-stakes tests.

A motive for our research is international evidence suggesting that tests of low stakes impact student motivation and effort (Finn, 2015; Wise & DeMars, 2005). Whereas the relation between high- and low-stakes testing has been studied previously, findings are inconsistent, and we know little about this relationship in Sweden. Therefore, the following study examines whether there is evidence for the hypothesis that the lack of personal consequences may bias PISA test scores downwards. Indeed, self-reports from Swedish students indicate that they do not do their best in PISA (Eklöf & Hopfenbeck, 2019). However, PISA test scores have not yet been compared to external criteria such as national test scores. The theoretical framework used to interpret the results of the present study is the expectancy-value theory (Eccles & Wigfield, 2002; Wigfield & Eccles, 2000), postulating that test motivation depends on the student's expectations of succeeding at a particular task, the value the student places on the task, and the interaction between the two (Eccles & Wigfield, 2002). The expectancy-value theory has successfully been used in previous studies to explain the test-taking motivation construct (e.g., Eklöf & Knekta, 2017).

Previous research on the relationship between national tests and PISA/TIMSS revealed moderate to high but imperfect correlations (Skolverket, 2022; Wiberg, 2019; Wiberg & Rolfsman, 2019). One possible explanation is that ILSAs have low stakes for students while national tests have high stakes. In order to test the assumption that motivation influences student achievement, we will examine whether test motivation moderates the relationship between PISA scores and the national test scores. Skolverket (2022) found a correlation of .61 between the two measures but our hypothesis is that the relationship is different for different levels of motivation to take the PISA test. With reference to the expectancy-value theory we assume that the average level of test motivation is higher for national tests since this is a test with higher stakes. For students that were particularly unmotivated to do the PISA test, the correlation with their national test scores could therefore be lower. Consequently, the study examines the following research questions: (1) What is the correlation between test motivation in PISA and PISA achievement? and (2) Is the relationship between low-stakes PISA test scores and high-stakes national test scores moderated by students’ test motivation in PISA?

In the 2018 Swedish PISA test, students’ personal identification numbers were collected, making it possible to link PISA tests scores with register data on students’ national test grades and student background characteristics, collected from Statistics Sweden (SCB). The analyses are based on this combined dataset, including a sample of 5,504 students. The main method used was latent moderated structural equations modelling. The outcome variable is students’ PISA achievement, measured through the ten plausible values by including the type = imputation option in Mplus, the software used. Since reading was the major domain in PISA 2018, the analyses focus on reading. However, robustness checks were conducted using PISA achievement in mathematics and science. The predictors used are students’ motivation to take the PISA test, formulated as a latent variable and used as moderator in the interaction analysis, and students’ national test grade. The latent variable PISA_Motivation is measured by six statements about students’ motivation in PISA, answered on a four-point Likert scale ranging from “strongly agree” to “strongly disagree” (reverse-coded in the analyses). The scale is provided as a national option in the PISA student questionnaire and contains items intended to measure effort, e.g., “I felt motivated to do my best in the PISA test” and importance, e.g., “Doing well in the PISA test was important to me”. Cronbach's alpha for the PISA motivation scale was .90 for the six items, indicating a high internal consistency. As an indicator of a high-stakes assessment, the students’ national test grade in reading, ranging from A–F and coded numerically, was used as an observed independent variable. Student background characteristics will be used as control variables in further analyses. In a first step, a measurement model of PISA_Motivation was estimated using confirmatory factor analysis (CFA), and model fit was ensured. Subsequently, structural models were estimated in consecutive steps (Muthén, 2012), starting with models without latent interaction, and then including both main effects and the latent interaction in the final model. The independent observed variable (national test grade) was centered prior to analysis. Model fit was evaluated using commonly used fit indices for structural equation modelling (Marsh et al, 2005). Models were estimated using MLR, and the complex option in Mplus was employed to account for the nested data structure. Analyses were weighted using the final student weight. Missing data was treated under the default method in Mplus (Full Information Maximum Likelihood).

Results revealed a significant positive correlation between PISA_Motivation and PISA achievement (r = .15), indicating that test motivation predicts achievement. In line with Skolverket (2022), the correlation between PISA achievement in reading and the national test grade in reading was found to be around .6. When controlling for students’ reading ability, in the form of the grade on the high-stakes national test, PISA_Motivation still significantly and positively influenced PISA achievement. In the final model, a significant positive interaction was shown between PISA_Motivation and the national test grade (β = .05, p < .001), indicating that students’ motivation in PISA affects the strength of the relationship between the high-stakes national test grade and the low-stakes PISA achievement. Graphical analyses of the interaction effects for students with different motivational levels showed that the simple slope differed particularly for students who indicated a low level of motivation in PISA and who received high grades on the high-stakes national test. The students with low motivation in PISA thus had a lower correlation between their PISA test score and their national test grade than the students who reported high motivation. This could be explained, in accordance with the expectancy-value theory, by the fact that these students put in less effort in PISA than on the national test because they do not see PISA as important to them personally. In sum, the study provides some evidence that the low-stakes nature of PISA may bias test scores for certain groups of students, in particular high achievers on the national test with low reported motivation in PISA. In the discussion, other reasons for the discrepancy between PISA test scores and national test grades will be addressed, such as differences in content, format and aims. Additionally, problematic aspects of measuring test effort with self-reported measures are considered.

Baumert, J., & Demmrich, A. (2001). Test motivation in the assessment of student skills: The effects of incentives on motivation and performance. European Journal of Psychology of Education, 16(3), 441–62. https://doi.org/10.1007/BF03173192 Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53(1), 109-132, https://doi.org/10.1146/annurev.psych.53.100901.135153 Eklöf, H. & Knekta, E. (2017). Using large-scale educational data to test motivation theories: A synthesis of findings from Swedish studies on test-taking motivation. International Journal of Quantitative Research in Education, 4(5), 52-71. Eklöf, H. & Hopfenbeck, T. (2019). Self-reported effort and motivation in the PISA test. In B. Maddox (Red.), International large-scale assessments in education: insider research perspectives (s. 121–136). Bloomsbury Academic. Finn, B. (2015). Measuring motivation in low-stakes assessments (Research Report No. RR-15-19). Princeton, NJ: Educational Testing Service. doi:10.1002/ets2.12067 Grek, S. (2009). Governing by numbers: the PISA ‘effect’ in Europe. Journal of Education Policy, 24(1), 23-37. https://doi.org/10.1080/02680930802412669 Jerrim, J. (2021). PISA 2018 in England, Northern Ireland, Scotland and Wales: Is the data really representative of all four corners of the UK?. Review of Education, 9(3). https://doi.org/10.1002/rev3.3270 Lindblad, S., Pettersson D., & Popkewitz, T.S. (2018). Numbers, Education and the Making of Society: International Assessments and Its Expertise. Routledge Marsh, H. W., Hau, K., & Grayson, D. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares and J. McArdle (Eds.), Contemporary Psychometrics (pp. 275–340). Erlbaum. Meyer, H. D., & Benavot, A. O. (Eds.). (2013). PISA, power, policy. The emergence of global educational governance. Oxford Studies in Comparative Education. Muthén B. (2012). Latent variable interactions. http://www.statmodel.com/download/LV%20Inter action.pdf Skolverket. (2022). PISA 2018 och betygen. Analys av sambanden mellan svenska betyg och resultat i PISA 2018 [PISA 2018 and school grades. Analyses of the relationship between Swedish school grades and results in PISA 2018]. Skolverket. Wiberg, M. (2019). The relationship between TIMSS mathematics achievements, grades, and national test scores. Education Inquiry, 10(4), 328-343. https://doi.org/10.1080/20004508.2019.1579626 Wiberg, M., & Rolfsman, E. (2019). The association between science achievement measures in schools and TIMSS science achievements in Sweden. International Journal of Science Education, 41(16), 2218-2232. doi:10.1080/09500693.2019.1666217 Wigfield, A., & Eccles, J. S. (2000). Expectancy-value theory of achievement motivation. Contemporary Educational Psychology, 25, 68–81. https://10.1016/ceps.1999.1015 Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://10.1207/s15326977ea1001_1