Research syntheses are a meanwhile important method to gather evidence on urgent questions and to support decision-making in policy and practice worldwide (Newman & Gough, 2020, p. 4). By aggregating or configurating what has been known so far, they help not only researchers in education science but also practicioners to find concise answers to a problem or to learn about promising interventions based on clearly calculated effect sizes or the careful interpretation of qualitative research results. Research syntheses have their place in today’s research landscape in education science as a scientific and trustworthy method for synthesising research results to foster evidence-based decision-making on a specific research question. Research syntheses or systematic reviews as a scientific method of their own have gained attention during the last decades (Gough et al., 2017) and are an essential part of knowledge building. Grant and Booth (2009) identified 14 types of reviews differing in scope and method, including the systematic review that combines amongst others the stages of a systematic literature search, an appraisal of the potentially relevant studies and the synthesis of the findings (Grant & Booth, 2009, p. 102). The classification of review types can be described as multidimensional (Booth et al., 2012, p. 20) and recently, Sutton et al. (2019) came up with 48 review types classified into seven groups based on common features and goals.

Irrespective of nomenclature, all research syntheses include a process, where researchers have to decide which literature they assess as basis for their syntheses or results (Boland et al., 2017, p. 25). Within the review’s context, researchers face challenges in making decisions on the literature search process, the inclusion and exclusion criteria applied to the literature retrieved and the concrete process of the synthesis to get their evidence-based results. However, how and why researchers decide on the relevance of research papers and how the steps during a systematic review process influence one another, is not fully understood. There is consent about the fact that a careful selection of relevant literature is one of the most important decisions that influence the synthesis and the concluding findings of a review (Lefebvre et al., 2021). How reviewers in educational research are influenced by their information behaviour and comprehension of relevance elucidates the process of understanding the insights behind decisions and the motivation for including relevant studies.

Relevance assessment processes are the higher-level principle when conducting research syntheses. Yet, the notion of relevance of information resources is complex and can be best described as a relation to an object or a context being expressed in a degree of appropriateness (Saracevic, 2017) or usefulness. Another way to understand the concept of relevance is introduced by Mizzaro (1997) who distinguishes four dimensions (research query or user problem, information resources, components  like context, topic or task, and time). Bringing these four dimensions together, relevance can be compared to a “point in a four-dimensional space” (Mizzaro, 1997, p. 812). In systematic review processes, researchers have to deal with degrees of relevance in view of the literature they assess. This is expressed by determining criteria that can be attributed to the foresaid dimensions. In many cases criteria can be categorised as formal relating to searching and filtering literature or as content-applied regarding the research question to be answered. But there are additional factors (e.g. subjective or environmental) that influence the reviewers’ decisions on including studies. This talk introduces a study that investigated researchers’ processes of relevance rating in research syntheses in education science. It will contribute to a better understanding of relevance decisions by researchers and their challenges when conducting syntheses.

A multi-perspective approach was undertaken to describe and analyse relevance assessment behaviour during research synthesis processes and to gain insight into the reasons that inform the reviewers’ decisions to retain or discard documents. It is also of interest in which way reviewers describe the criteria applied, explain their reasons for deciding on the inclusion of studies and refer to international standards of documenting review processes (Moher et al., 2009). Therefore, the first part of the data collection will be a qualitative text analysis (thematic analysis) of a review corpus that has been compiled via purposeful sampling out of a corpus of (systematic) reviews in educational research in Germany during 2014 until 2019 (Jäger-Dengler-Harles et al., 2021). The review texts are in English or German and cover a broad spectrum of findings in many sub-disciplines of educational research. The analysis will focus on the authors’ organisation and documentation of the review stages, especially the screening and relevance assessment processes, the type and quality of criteria for inclusion and exclusion and the rigor of application. Around 80 relevance criteria gathered from the existing literature (Schamber, 1994, p. 11) can be applied when coding the review material and identifying factors that influence decision processes. Therefore, the analysis of review documents gives a detailed insight into how reviewers describe what they find relevant. It informs about explicit as well as implicit criteria applied to rate study results for inclusion in a review.

Data show that researchers in review processes consider a variety of factors to justify their decisions on studies being rated as relevant for inclusion in the final corpus. Inspection of review texts has shown that review authors do not carefully document every review phase as well as their experience and behaviour in the relevance assessment and decision-making processes. For instance, it is often not clear what reviewers intend when they speak about aggregating results from the “German- or English-speaking discussion” among experts in education science and within this context, apply criteria related to language and geographical area (e.g. language and/or place of publication, area of investigation). But careful analysis of the review data reveals also that reviewers are not always aware of what consequences selection procedures might have for the final number of studies being included. It happens quite often that review authors notice a small number of relevant documents in the final stages, which is unexpected for them and cannot be explained at once. This talk will discuss possible pitfalls that can occur during review processes and exemplify selected issues dealing with criteria which are in need of further clarification to be completely understood by the audience. In the realm of educational research, conducting research syntheses is accepted as a powerful scientific method to aggregate research evidence. But there is also the challenge of identifying as much as possible literature relevant to the research question and at the same time of defining the individual and case-specific notion of relevance by the application of formal, content-related and other criteria made explicit and understandable to the public.

Boland, A., Cherry, M. G., & Dickson, R. (2017). Doing a systematic review: A student's guide (2. ed.). Sage. Booth, A., Papaioannou, D., & Sutton, A. (2012). Systematic approaches to a successful literature review. Sage. Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction of systematic reviews (2nd. edition). Sage. Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91–108. https://doi.org/10.1111/j.1471-1842.2009.00848.x Jäger-Dengler-Harles, I., Keller, C., Heck, T., & Rittberger, M. (2021). Methodenbericht zur Erhebung "Literaturrecherche für Dossier ForSynBiFo" aus der Studie "Forschungssynthesen zur Bildungsforschung 2014-2019 - ForSynBiFo". In Forschungsdatenzentrum Bildung am DIPF (Ed.), Forschungssynthesen zur Bildungsforschung 2014-2019 - Literaturrecherche für Dossier ForSynBiFo (ForSynBiFo) [Datenkollektion: Version 1.0]. Datenerhebung 2019-2021. (pp 1–16). DIPF | Leibniz Institute for Research and Information in Education. Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M.‑I., Noel-Storr, A., Rader, T., Shokraneh, F., Thomas, J., & Wieland, L. S. (2021). Chapter 4: Searching for and selecting studies. In Cochrane handbook for systematic reviews. The Cochrane Collaboration. Mizzaro, S. (1997). Relevance: The whole history. Journal of the American Society for Information Science, 48(9), 810–832. https://idw-online.de/de/pdfnews749034 Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. Journal of Clinical Epidemiology, 62, 1006–1012. https://journals.plos.org/plosmedicine/article/file?id=10.1371/journal.pmed.1000097&type=printable (Methods of systematic reviews and meta-analysis). Newman, M., & Gough, D. (2020). Systematic reviews in educational research: Methodology, perspectives and application. In O. Zawacki-Richter, M. Kerres, S. Bedenlier, M. Bond, & K. Buntins (Eds.), Research. Systematic reviews in educational research: Methodology, perspectives and application (pp. 3–22). Springer VS. https://doi.org/10.1007/978-3-658-27602-7_1 Saracevic, T. (2017). The notion of relevance in information science: Everybody knows what relevance is. But, what is it really? Synthesis lectures on information concepts, retrieval, and services: Vol. 50. Morgan and Claypool. https://doi.org/10.2200/S00723ED1V01Y201607ICR050 Schamber, L. (1994). Relevance and information behavior. Annual Review of Information Science and Technology, 29, 3–48. https://eric.ed.gov/?id=EJ491620 Sutton, A., Clowes, M., Preston, L., & Booth, A. (2019). Meeting the review family: Exploring review types and associated information retrieval requirements. Health Information and Libraries Journal, 36, 202–222. https://doi.org/10.1111/hir.12276