International Large-scale Assessments in Education (ILSA) are an essential part of system monitoring in most educational systems (Martens et al., 2016). ILSA instruments gather information about the conditions and circumstances of learning and deliver comparable data on a regular basis (Jude, 2016). Consequently, policy relevant conclusions can be drawn to facilitate certain developments and support student competencies in changing learning environments (Lietz & Tobin, 2016).

Scholars have evaluated how ILSAs have impacted the scientific community by analysing related publications in scientific journals quantitatively and qualitatively. They were able to advance our knowledge about just how strongly ILSAs are represented in English-language scientific publishing and how strong certain topics dominate and others are neglected. In their systematic review of international peer-reviewed publications, Hopfenbeck et al. (2017) analysed the scientific discussion on secondary analysis, policy impact, and critiques regarding ILSA.

However, ILSA are also a prominent example of internationalization processes in education. An ongoing debate discusses the pros and cons of this approach: International tests and benchmarks are taken over into national curricula and rankings might lead to seemingly unjust comparison of educational systems (Heinemann & Lee, 2014).

Accordingly, ILSAs receive more and more attention from different actors. Policymakers, school practitioners and educational researchers aim to analyse the data, criticise the studies’ design and discuss the impact that they have on educational policymaking. Still the long-term influence and direct impacts of ILSA on educational policy are rather complex and might not easily be detected, as the cross-sectional nature of these studies can hardly be interpreted causally (Johansson, 2016).

Probably the most prominent ILSA is the OECD’s Programme for International Student Assessment (PISA) which often has been criticized for its strong influence on national educational policy (Breakspear, 2014). On a three year basis, PISA assesses students’ literacy in the major domains of reading, mathematics and science and delivers comparable indicators on learning resources and contexts (OECD, 2016). In its latest wave, PISA 2018 is being implemented in 80 countries and assesses the literacy of 15-year old students.

In Germany and the German speaking countries in Europe the public debate on ILSAs has been substantial (Waldow, 2009). PISA in particular had a strong influence in scientific, public and policy communities.

This review study analyses scientific and practitioner-oriented publications focussing on PISA in the German speaking countries over the last 20 years. It aims to give an overview of the scientific and public debate, and their focus on different topics over time. Results include quantitative and qualitative approaches, distinctions between scholarly and practitioner-oriented debates, and highlighting different interests in the public debate on education.

Data was drawn from FIS Bildung, the most comprehensive bibliographical database for German publications related to education. It includes almost a million datasets of monographs, edited books and chapters as well as journal articles. FIS Bildung is based on the cooperative work of 30 bibliographic documentation services in Germany, Austria and Switzerland. Data is being updated four times a year. The first search of the database included the terms PISA-Studie, PISA (Programme for International Student Assessment), PISA-E, PISA-E (Programme for International Student Assessment - Ergänzungsstudie) in a time window between 1998 and 2018. It resulted in 4376 publications that were then used for further analysis. Data was extracted as EXCEL file to be used in the quantitative categorization. Publications did not only appear in journals with an established peer-review process, but also in practitioner-oriented outlets without an official peer-review process. We included both forms in order to present the visibility of PISA in scholarly communities on the one hand and evaluate extent and tenor of its reception in practice-oriented communities. The inclusion criteria for the database were thus relatively broad. Contributions had to be published in German, in a scientific or practitioner-oriented journal, edited book or as a monograph and to be substantially concerned with PISA. Publications in newspapers or blogs are not covered by FIS Bildung and are thus not included in the database. For the description and analysis, we organised our data along the following categories: type of publication, year of publication, journal type. In addition we sorted publications into themes with specific keywords resulting in a ranking of keywords most commonly used in publications on PISA. They were educational policy (N = 462), equity (N=256), and school reform (N=125). With the help of these keywords, we selected publications in three dominating themes for a more in-depth review, focussing on the empirical findings, perspectives of criticism and general tenor of the discussion.

Overall, the highest amount of PISA related publications in German can be noted between 2001 and 2005, following the so called “PISA-shock” in Germany. Those articles were published in journals that mostly targeted schools and school practitioners, followed by publications in research and policy related journals. In addition to the mere quantitative analysis, we assessed articles targeting specific areas of public debate in Germany. In the category of educational policy for example, we searched for publications explicitly mentioning PISA related criticism. This revealed 15 journal articles in the field of educational policy where a critical public debate about the use of large scale assessment was initiated. Arguments used in these publications included the concept of education targeted in ILSA, criticising that cognitive tests only focus on mathematics, science and reading while neglecting a comprehensive concept on education. Other articles highlight the importance of measuring context factors as well as outcome, an approach that meanwhile has been focused more strongly as the PISA study developed over time. When analysing these publications over time, educational policy was the most common keyword in each respective year used in PISA related articles since 1998. This review shows the development of publications related to international large scale assessments as a heavily debated topic in the German education. In addition to identifying most common topics over time, gaps in the German language publications could be identified. While policy related terms can be seen dominating the discussion, publications referring to teachers or teaching for example are rather scarce. As PISA focussed on the system level, this is not surprising, however data exists that explicitly focusses on the teachers participating in the PISA questionnaires. Thus more publications could be expected informing the German speaking teacher community about results from international large scale assessments.

Breakspear, S. (2014). How does PISA shape education policy making? Why how we measure learning determines what counts in education.Centre for Strategic Education. http://simon-breakspear.com/wp-content/uploads/2015/09/Breakspear-PISA-Paper.pdf Heyneman, SP, Lee, B (2014) The impact of international studies of academic achievement on policy and research. In: Rutkowski, L, von Davier, M, Rutkowski, D (eds) Handbook of International Large-Scale Assessment. CRC Press, pp. 37–75. Stefan Johansson (2016) International large-scale assessments: what uses, what consequences?, Educational Research, 58:2, 139-148, DOI:10.1080/00131881.2016.1165559 Jude, N. (2016). The Assessment of Learning Contexts in PISA. In Kuger, S.; Klieme, E.; Jude, N. & Kaplan, D. (Eds.). Assessing Contexts of Learning. An International Perspective, 39-51. New York: Springer International Publishing. Petra Lietz & Mollie Tobin (2016) The impact of large-scale assessments in education on education policy: evidence from around the world, Research Papers in Education, 31:5, 499-501, DOI: 10.1080/02671522.2016.1225918 Martens, K., Niemann, D., & Teltemann, J. (2016). Effects of international assessments in education – a multidisciplinary review. European Educational Research Journal, 15(5), 516–522. https://doi.org/10.1177/1474904116668886 OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris, https://doi.org/10.1787/9789264266490-en. Waldow, F. (2009). What PISA Did and Did Not Do: Germany after the ‘PISA-shock.’ European Educational Research Journal, 8(3), 476–483. https://doi.org/10.2304/eerj.2009.8.3.476