In line with the principles of open science and good scientific practice described in the description of the symposium, this contribution provides in-depth information on and results of a post-hoc analysis of a comprehensive literature search strategy for the identification of published research syntheses on digital competences of young learners (Carratero et al., 2017). Literature search strategies in systematic reviews consist of the development of the initial search string rooted in the theoretical background of the review, the searched databases and processing the database hits (Hausner et al., 2012). In addition, the efficiency of the search strategy can be determined via post-hoc analyses of the search’s precision and sensitivity (Pastore & Scheirer, 1974). In this project, two different search strategies were applied by searching for relevant phrases (i.e. “digital competence” instead of “digital” and “competence”, see Vechtomova & Karamuftuoglu, 2004) and traditional term-based searches in two languages in multiple databases. Furthermore, a manual search was conducted to identify additional papers for the screening process not found by the search strategy. Combining the search strategies, the databases and the languages resulted in a total of eight searches which led to n = 4,871 unique database hits. After title/abstract-screening and full-text-screening, n = 120 documents were included for the subsequent analyses. Subsequently, the search strategies were evaluated by determining the duplication and exclusivity rates between the searches as well as for the included and excluded papers. Additionally, we determined the document-frequencies of each single search term dependent on the inclusion decision; by determining the logarithmic difference between the document-frequencies of the search terms, information on the relevancy of the search terms for the inclusion decision was extracted. All in all, this presentation provides insights into the development of a comprehensive search strategy which aimed at finding a compromise between exhaustiveness and precision. Each central step of search strategies is presented (i.e. development search string, database search, data export) and evaluated always under the premises of replicability, reproducibility and research transparency. Implications for search strategies for research syntheses in regard to the principles of open science and good scientific practice are discussed such as reporting the search strings in their applied form, reporting on the distribution of database hits and post-hoc evaluating the search to identify potential thematic biases based on skewed distributions within the search string.

Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1. The Digital Competence Framework for Citizens. With eight proficiency levels and examples of use. Publications Office of the European Union. Hausner, E., Waffenschmidt, S., Kaiser, T., & Simon, M. (2012). Routine development of objectively derived search strategies. Systematic reviews, 1(1), 19. Pastore, R. E., & Scheirer, C. J. (1974). Signal detection theory: Considerations for general application. Psychological Bulletin, 81(12), 945. Vechtomova, O., & Karamuftuoglu, M. (2004, November). Approaches to High Accuracy Retrieval: Phrase-Based Search Experiments in the HARD Track. In TREC