Research syntheses gather, aggregate and analyse results of scientific publications on a specific subject to map past and current research and – in some cases – extract implications for research, policy and practice (Newman & Gough, 2020). They enable guidance based on stable evidence across a multitude of settings and samples, rather than on anecdotal results from single studies with only limited generalizability. The reliability and validity of the results, and therefore the scientific basis of its implications, are ensured by following the principles of open science and good scientific practice (Crüwell et al., 2019; Open Science Collaboration, 2015).

Adhering to the principles of open science while conducting research syntheses requires researchers for example to enable open/FAIR access to their research data (Stall et al., 2019), the use of open source software or the publication under open access licensing, including through ensuring transparency, replicability and reproducibility of each single step taken in conducting research syntheses. If those principles are disregarded, potential biases, blind spots or limitations of the results may be overlooked, which can lead to the incorrect transfer of the extracted implications. Therefore, each single step in conducting research syntheses has to be held to the highest standard of scrutiny regarding good scientific practice. This includes – but is not limited to – the development of (a) the initial search string and its application for the literature search, (b) the inclusion and exclusion criteria and the ensuing screening processes and (c) the coding/data extraction scheme, data analysis and syntheses processes and methods and presenting the review’s results. For (a) in conjunction with uploading the database hits to an accessible repository, the actual search terms for each single database have to be reported including the Boolean operators, the applied filters and the search date. Requirements for enabling transparency, replicability and reproducibility during screening (b) include clearly stating the inclusion and exclusion criteria and their application and reporting on resolving potential intercoder differences. In parallel to (b), the coding/data extraction schemes and potentially arising differences in the analysis and synthesis have to be reported transparently as well. In addition, if possible, all scripts applied during the analysis and visualisations should either be attached to the review or uploaded to relevant repository to enable the replicability and reproducibility of the results.

In this symposium, four research syntheses are presented with a focus on their adherence to the principles of open science and good scientific practice. While each presented project contained all relevant steps of conducting research syntheses, their goals, research questions and utilized methodology varied greatly: The first focuses on the development and application of a comprehensive yet precise search string for an overview of reviews on digital competences of young learners. Presentation two focuses on structuring and sharing the data extraction coding tool, as well as presenting the review’s results through various openly accessible means for researchers, policymakers and educators. The third presentation focuses on documenting the processes of analysis and synthesis by presenting the results of a methodological review of metasyntheses in educational research. The fourth presentation is concerned with potential biases in systematic reviews due to most included research originating in WEIRD-countries, i.e. western, educated, industrialized, rich and democratic countries.

The summarising discussion focuses on the general role of the principles of open science and good scientific practice while conducting research syntheses and how they represent requirements for evidence-based guidance of future research, policy and practice. In addition, it will highlight the potentials and challenges of transferring implications from syntheses into educational policies, practice and administration in regard to the heterogeneity of national educational systems, policies and curricula.

Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., … Schulte-Mecklenbeck, M. (2019). Seven Easy Steps to Open Science: An Annotated Reading List. Zeitschrift Für Psychologie, 227(4), 237–248. https://doi.org/10.1027/2151-2604/a000387 Newman, M., & Gough, D. (2020). Systematic reviews in educational research: Methodology, perspectives and application. Systematic reviews in educational research, 64(3), 3-22. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. https://doi.org/10.1126/science.aac4716 Stall, S., Yarmey, L., Cutcher-Gershenfeld, J., Hanson, B., Lehnert, K., Nosek, B., ... & Wyborn, L. (2019). Make scientific data FAIR. Nature, 570(7759), 27-29. https://doi.org/10.1038/d41586-019-01720-7