International large-scale assessments (ILSAs) have gradually become tools influencing educational policies in various domestic contexts (Johansson, 2016). Some regions in Mainland China have participated in the Programme for International Student Assessment (PISA), one of the most influential ILSAs. Though PISA has been experienced critics on fueling the phenomenon of policy borrowing (e.g. Auld and Morris, 2016) and debates on its measurement limitations (e.g. Rutkowski and Rutkowski, 2016; Goldstein, 2018), various educational reforms have been motivated by PISA in a number of countries (e.g. Neumann et al., 2010). PISA is even considered a political tool governing European education systems through international comparison (Grek, 2009). Literature widely discusses PISA’s impact on policy reforms at national level across jurisdictions (e.g. Ertl, 2006; Breakspear, 2012). However, empirical research on the consequences of PISA in local contexts and the impact it has had on teaching and learning in schools remains scarce. Though it is argued that test developers and users have the responsibility of evaluating test consequences (Kane, 2006), educational researchers’ studies on this could inform typical users of PISA scores, i.e. policymakers, for appropriate policymaking.

     This study aims to empirically investigate PISA’s consequences in terms of its impact on teaching and learning in schools in a local Chinese context. It addresses two research questions. Firstly, what are the uses of PISA in local context? Secondly, what are the consequences of these uses for teaching and learning in schools?

     The mechanism of washback (Alderson and Wall, 1993; Hughes, 1993, cited in Bailey, 1996) is employed as the theoretical framework. Washback which is considered as “the effect of testing on teaching and learning” (Hughes, 1989, p.1) could be either positive or negative (Cheng et al., 2004). It has been considerably discussed and its existence is asserted in the context of language testing or high-stakes educational assessments (Alderson and Walker, 1993; Cheng et al., 2004). Though PISA is not explicitly high-stakes for schools since it does not report individual students’ scores, the concept of washback has been introduced to discuss PISA effects (Baird et al., 2016; Maddox and Addey, 2016). Hughes (1993, cited in Bailey, 1996) notes that for participants (e.g. administrators, teachers) involved in the washback mechanism, their perceptions and attitudes towards teaching and learning would be firstly affected by test nature, the changed perceptions and attitudes may then affect what participants do in their work which would subsequently influence the learning outcomes. Washback would be complex that various factors such as test nature and educational context should be considered in researching it (Alderson and Wall, 1993). This study considers the characteristics of PISA, the specific local and national educational context in investigating PISA’s impact.

     Fangshan District of Beijing is the focused local context because it continuously participated in two cycles of PISA China trials and PISA 2015, and has been actively using PISA for improving local educational quality. As officially claimed, Fangshan has initiated three projects motivated by PISA, each of which targets one PISA assessment domain (i.e. reading, mathematics, science) (Guo et al., 2015). Sixteen participants consisting of key informants of the launch and administration of Fangshan PISA-motivated initiatives, school leaders and teachers were interviewed individually with semi-structured interviews. Preliminary findings based on thematic analysis suggest that: (1) rather than seeking “best practice” from elsewhere, Fangshan has been exploring its own way of using PISA to improve local education; (2) in the local context, PISA has begun progressively influencing teaching and learning in schools through the implementation of local PISA-motivated initiatives, and also with the power of national educational reforms which tend to be convergent with some PISA ideas to some extent.

The fieldwork site, Fangshan, is located in the southwest of Beijing. By the end of 2015, there were 56 secondary schools with 40631 enrolled students (Fangshan Bureau of Statistics and Fangshan Survey Team of National Bureau of Statistics, 2016). Semi-structured interviews (Bryman, 2012) were conducted individually with key informants of the launch and administration of PISA-motivated initiatives, and with school leaders as well as teachers. In total, 16 participants were recruited, consisting of one Fangshan local official who is responsible for the administration of PISA and research on using PISA results in Fangshan, one key informant of each of the three PISA-motivated projects, five school leaders, one reading teacher and six mathematics teachers. The local official was first interviewed, followed by the key informants of the initiated projects who were recommended by the official, to investigate the uses of PISA in PISA-motivated initiatives. With the recommendation of the local official or the project key informants, six academic secondary schools were selected, taking into consideration school location (i.e. rural/town/city) and educational programme level (i.e. lower/upper secondary). School leaders and teachers were then sampled and interviewed to investigate the school-level implementation of PISA-motivated initiatives and their perceived consequences of the uses of PISA for teaching and learning in schools. In one the selected schools, the eligible school leader who could provide insights into understanding of this school’s implementation of PISA-motivated initiatives was transferred to another school before data collection. Instead, one reading teacher in this school recommended by the school contact was interviewed, since she was involved in PISA-motivated reading initiatives. Considering the projects’ timelines, only mathematics teachers were selected. Ethical review was approved, and written consent was obtained from each participant before the commencement of each interview. Audio recordings of interviews were transcribed and then anonymized before data analysis conducted in QSR International's NVivo 11 Software. Thematic analysis approach (Braun and Clarke, 2006) was employed in data analysis to identify themes.

The preliminary findings reveal that, PISA is used as a new perspective comparing with domestic examinations for reviewing local educational quality. Meanwhile, PISA frameworks shed light on local policymakers and practitioners’ understanding of teaching and learning. Three projects closely linking schools were catalyzed by PISA for improving local education quality. To be specific, the reading project aims to increase students’ exposure to reading, especially practical reading as defined in PISA. The mathematics project and science project were then initiated successively, both of which aim to integrate PISA ideas into teaching to improve student learning. Implementation of the PISA-motivated initiatives varies across schools, affected by school leaders’ or teachers’ perceived significance of and their understanding of the ideas suggested by these initiatives. Some school-level interviewees stated that PISA ideas have been progressively influencing teachers and therefore influencing student learning, though teachers may not be definitely aware of that. National educational context including the ongoing reforms of educational assessment system and national curricula influences both the policy-level construction and school-level implementation of these local PISA-motivated initiatives. The requirements of high-stakes examinations (e.g. National College Entrance Examination) and national curricula are still most concerned for local educational practitioners, while some points of national educational reforms are aligned with PISA ideas, as perceived by some school-level interviewees. To conclude, Fangshan policymakers focus on exploring Fangshan’s own way to address local educational issues by using PISA together with domestic sources. Though PISA is not explicitly high-stakes for students, it has been progressively influencing teaching and learning through local PISA-motivated initiatives; meanwhile, it is indirectly affecting teaching and learning through its influence on national educational context. This study contributes to the understanding of PISA consequences in local contexts, and provides insights into studying the impact of influential ILSAs (e.g. PISA) on schools in other education systems.

Alderson, J. C. and Wall, D. 1993. Does washback exist? Applied Linguistics. 14(2), pp.115–129. Auld, E. and Morris, P. 2016. PISA, policy and persuasion: translating complex conditions into education ‘best practice’. Comparative Education. 52(2), pp.202–229. Bailey, K.M. 1996. Working for washback: a review of the washback concept in language testing. Language Testing. 13(3), pp.257–279. Baird, J.-A., Johnson, S., Hopfenbeck, T.N., Isaacs, T., Sprague, T., Stobart, G. and Yu, G. 2016. On the supranational spell of PISA in policy. Educational Research. 58(2), pp.121–138. Braun, V. and Clarke, V. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology. 3(2), pp.77–101. Breakspear, S. 2012. The policy impact of PISA: an exploration of the normative effects of international benchmarking in school system performance, OECD Education Working Papers, No. 71. Bryman, A. 2012. Social Research Methods. 4th ed. Oxford: Oxford University Press. Cheng, L., Watanabe, Y. and Curtis, A. eds. 2004. Washback in Language Testing: Research Contexts and Methods. Mahwah, NJ: Lawrence Erlbaum Associates. Fangshan Bureau of Statistics and Fangshan Survey Team of National Bureau of Statistics. 2016. Statistical yearbook 2016 of Fangshan District of Beijing. Beijing: Fangshan District Bureau of Statistics. Goldstein, H. 2018. Measurement and evaluation issues with PISA. In: Volante, L. ed. The PISA Effect on Global Educational Governance. New York: Routledge, pp.49-58. Grek, S. 2009. Governing by numbers: The PISA effect in Europe. Journal of Education Policy. 24(1), pp.23-37. Guo, Z., Wu, Y. and Zhang, J. 2015. Jiyu PISA shijian de quyu jiaoyu zhiliang tisheng. Teacher's Journal. 10, pp.8-12. Hughes, A. 1989. Testing for Language Teachers. Cambridge: Cambridge University Press. Johansson, S. 2016. International large-scale assessments: what uses, what consequences? Educational Research. 58(2), pp.139–148. Kane, M. T. 2006. Validation. In: Brennan, R. L. ed. Educational Measurement. 4th ed. Westport: Praeger Publishers, pp.17–64. Maddox, B. and Addey, C. 2016. The Futures and Promises of International Educational Assessment. [Online]. [Accessed 3 March 2017]. Available from: http://international-assessments.org/the-futures-and-promises-of-international-educational-assessment/. Neumann, K., Fischer, H. E. and Kauertz, A. 2010. From PISA to educational standards: the impact of large-scale assessments on science education in Germany. International Journal of Science and Mathematics Education. 8(3), pp.545–563. Rutkowski, L. and Rutkowski, D. 2016. A call for a more measured approach to reporting and interpreting PISA results. Educational Researcher. 45(4), pp.252–257.