Secondary analysis of large-scale international assessments requires many choices from analysts: ranging from which control variables to include, to how to handle complex survey designs (e.g., weights, missingness), to questions of how to parameterize and build measurement scales. In some cases, there are right and wrong choices, but, in many other cases, multiple possible choices could be defended. These choices create a "garden of forking paths" that analysts travel down when drawing a conclusion. It is well-known that confirmation bias can lead analysts to making choices that lead to the desired conclusion. Less understood, but equally problematic, is the invisible uncertainty in conclusions that results from these choices since standard errors do not typically include uncertainty related to such analytic choices. Universe analyses (Breznau et al., 2022; Götz et al., 2024) are a modern strategy to make visible the uncertainty that comes from analytic decisions. In a universe analysis, when the analyst is faced with a choice between two defensible choices, the analyst does the analysis twice, once with each possible choice. In essence, the goal is to take all possible pathways down the garden of forking paths and explicitly study the empirical impact of the different choices. The result is a set of thousands of slightly different, but all defensible, analytic models to answer the same research question. By looking across these models one can empirically explore the hidden uncertainty that comes from choices made in conducting the analysis. In this paper, I present a universe analysis for a highly-cited study on teacher induction practices from the TALIS survey (Reeves et al., 2022). I identify 11 analytic choices that lead to 2,048 possible ways of conducting the analysis. Across these 2,048 possible analytic models, the main result ranges from negative (but not statistically significant) to highly significant and positive. The variation in outcomes across models implies that while it is possible to show a positive impact of induction practices using TALIS data, this finding is quite dependent on the analytic model one builds. This has important implications for how we understand the trustworthiness of findings from large scale studies. For example, it is unclear that we can argue that TALIS provides support for the importance of teacher induction if such support is highly dependent on the choice of analytic model. I discuss the implications of universe analyses and ways to make secondary studies of large-scale data more trustworthy.

Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Balzer, D., Bauer, G., Bauer, P. C., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., … Żółtak, T. (2022). Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences, 119(44), e2203150119.https://doi.org/10.1073/pnas.2203150119 Götz, M., Sarma, A., & O’Boyle, E. (2024). The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse. International Journal of Psychology, 2024. https://doi.org/10.1002/ijop.13229 Reeves, T. D., Hamilton, V., & Onder, Y. (2022). Which teacher induction practices work? Linking forms of induction to teacher practices, self-efficacy, and job satisfaction. Teaching and Teacher Education, 109, 103546. https://doi.org/10.1016/j.tate.2021.103546