Highly recommended and yet neglected: Replication studies as rare phenomena in the educational sciences
Author(s):
Conference:
ECER 2017
Format:
Paper

Session Information

10 SES 08 E, Replication, Quality and Teacher As Researcher

Paper Session

Time:
2017-08-24
09:00-10:30
Room:
K5.02
Chair:
Blerim Saqipi

Contribution

Within the quantitative research paradigm the methodological importance of replication studies is frequently highlighted in literature (King 1995; Schmidt 2016; Schweizer 1989). In contrast to this proclaimed relevance only few published replication studies can be identified (Schmidt 2012, p. 247; Wagner & Huschka, 2012, S. p. ff.). Makel & Plucker (2014), for example, ascertained that only 0.13% of articles in the top 100 education journals are replications.

In other disciplines (e.g. psychology), the lack of replication studies is discussed, but this does not apply to educational sciences. Here, the gap between the methodologically-based call for replications and their factual shortage is hardly recognized. Against this background, our study focuses on this disproportion by asking: How can the remarkable contrast between the proclaimed relevance and the practical non-existence of replications in the field of educational sciences be explained?

Broadly, replication studies aim to vet or ensure the validity of previous research findings. The basic idea is to corroborate or disconfirm empirical results or an academic hypothesis by repeating a study design (Schmidt, 2012, p. 234). This is even more important given that educational science often claims the potential to alter pedagogical settings based on empirically solid findings and knowledge.

Compared to the quantitative research paradigm, the concept of replicating designs and findings is even less discussed in literature on qualitative research methods (presumably due to different methodological points of view). Yet despite possible methodological frictions (Wissenschaftsrat, 2011, p. 56), one could argue that qualitative research – similar to quantitative research – also aims at generalization, albeit in other terms than quantitative studies. In both cases, i.e. in quantitative and qualitative fields of research within educational sciences, (published) replication studies are extremely rare and there is a remarkable gap between the relevance and the actual realization of replication studies.

With reference to approaches such as system theory (Luhmann, 1990), motivation psychology and decision theory (Beach, 1990; Beach & Mitchell, 1996), we assume that causes for the gap can be located in different aspects and dimensions: For instance, it can be surmised that the research system holds explicit and implicit criteria for the development and acceptance of scientific knowledge. Additionally, individual motives of researchers relating to their scientific socialization can play a role for the (non-)engagement in replication designs. So far, explanatory factors as the named are neither theoretically nor empirically discussed with regard to the realization of replication studies.

Method

As there is little knowledge as to the barriers to carry out and/or publish replication studies, an initial qualitative approach is needed to explore underlying reasons. Therefore, we conduct expert interviews (Meuser & Nagel 1991, 2013) with key researchers, editors of relevant journals and representatives of funding organizations (i.g. DFG, BMBF). Expert interviews are a qualitative, semi-standardized interview form that explores organizationally framed insight knowledge and implicit rules of practice. Our design of guidelines for the semi-structured interviews is predicated on a study by Neuliep & Crandall (1990) who conducted a survey on attitudes: They interviewed editors of social science journals focusing on their attitudes concerning replication studies. One of their findings is that 72% of the editors see more relevance in a study that demonstrates some new effects while only 6% indicate that a replication was more important (Neuliep/Crandall 1990, p. 87). We take up those findings but take a broader view on the topic. We not only focus on publication strategies but also explore other possible impediments to replications. Compared to the study by Neuliep & Crandall, our research setting differs considerably regarding place (American vs. German science system), discipline (social and behavioral sciences vs. education sciences), interview partners (editors vs. editors among other ‘experts’) and time (1990 vs. 2017), so we do expect that our findings will differ. Comparable findings, anyhow, would indicate overall scientific rules and structures and might be of particular international interest. For the data analysis, we draw on the suggested analysis procedure of Meuser & Nagel (2013) that implies six steps: transcribing, paraphrasing, thematic coding, thematic contrasting, referring to sociological conceptions and developing theoretical generalization. Based on those findings and in a second step, we will design a standardized questionnaire in order to carry out a broad survey within the German educational science community.

Expected Outcomes

Qualitative research is an explorative, theory generating approach. Therefore, expected outcomes do not refer to specific initial hypotheses. Hence, by the means of expert interviews we intend to gain insight into expert knowledge of key persons that either act as gate keepers for scientific acceptance and success (editors and representatives of funding organizations) or hold a special expertise concerning empirical methods. We expect to find hindering and encouraging causes as well as conditions for researchers to conduct replications. With regard to qualitative research, we expect a methodological discussion, especially on the question whether quality criteria such as validation and replication (that are based on quantitative research paradigms) can and should be applied to qualitative research. Overall we aim at: • developing an explanatory approach for the remarkable disproportion between the theoretical call for replications and the actual little number of conducted replications • delivering an approach that offers a base for transfer to other disciplines • intensifying the methodological discussion on the significance of replication studies and the described discrepancy • if indicated: developing practical perspectives of encouraging researchers of educational science to more systematically focus on replication studies Our presentation offers a first analysis of the conducted expert interviews.

References

Beach, L. R. (1990). Image Theory: Decision Making in Personal and Organizational Contexts. Chichester: John Wiley & Sons. Beach, L. R., & Mitchell, T. R. (1996). Image theory, the unifying perspective. In L. R. Beach (Ed.), Decision making in the workplace. A unified perspective (pp. 1-20). Mahwah, NJ: Erlbaum. King, G. (1995). Replication, Replication. PS: Political Science & Politics, 28(3), 444–452. Schweizer, K. (1989). Eine Analyse der Konzepte, Bedingungen und Zielsetzungen von Replikationen. Archiv für Psychologie, 141, 95–97. Schmidt, S. (2016). Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences. In A. E. Kazdin (Hrsg.), Methodological Issues and Strategies in Clinical Research (4. Aufl., S. 581–596). Washington: American Psychological Association. Schmidt, S. (2012). Muss man alles wiederholen? Eine kritische Analyse des Replikationsbegriffs in der modernen Wissenschaft. In W. Ambach (Hrsg.), Experimentelle Psychophysiologie in Grenzgebieten. Würzburg: Ergon. Luhmann, N. (1990). Die Wissenschaft der Gesellschaft. Frankfurt a. M.: Suhrkamp Taschenbuch Wissenschaft. Makel, M. C., & Plucker, J. A. (2014). Facts Are More Important Than Novelty: Replication in the Education Sciences. Educational Researcher, 43(6), 304–316. Meuser, M., & Nagel, U. (1991). Experteninterviews – vielfach erprobt, wenig bedacht. Ein Beitrag zur qualitativen Methodendiskussion. In D. Garz & K. Kraimer (Hrsg.), Qualitativ-empirische Sozialforschung (S. 441-471). Opladen: Westdeutscher Verlag. Meuser, M., & Nagel, U. (2013). Experteninterviews – wissenssoziologische Voraussetzungen und methodische Durchführung. In B. Friebertshäuser, A. Langer, A. Prengel (Hrsg.), Handbuch Qualitative Forschungsmethoden in der Erziehungswissenschaft (4. Aufl., S. 457-471). Weinheim, Basel: Beltz Juventa. Neuliep, J. W., & Crandall, R. (1990). Editorial Bias Against Replication Research. Journal of Social Behavior and Personality, 5(4), 85–90. Berlin. Wagner, G. G., & Huschka, D. (2012). Datenverfügbarkeit reicht nicht, um Replikationsstudien zur Routine zu machen [Working Paper Rat für Sozial- und WirtschaftsDaten]. http://www.ratswd.de/download/RatSWD_WP_2012/RatSWD_WP_194.pdf Wissenschaftsrat (2011). Empfehlungen zu Forschungsinfrastrukturen in den Geistes- und Sozialwissenschaften.

Author Information

Christine Demmer (presenting / submitting)
Universität Bielefeld
Erziehugnswissenschaft / Educational Science
Bielefeld
Universität Bielefeld, Germany
Universität Bielefeld, Germany

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.