Given the global emphasis on education as a road to national triumph and individual success, it is not surprising that a vast amount of research concerns which teaching methods will make education fulfil its aims. Although education concerns many areas such as educational policy, organization of education, financial systems, and school leadership, there seems to be wide agreement that teaching in the end is the key factor in making educational systems successful (e.g. Barber & Mourshed, 2007; Hargreaves & Fullan, 2012; Hattie, 2003; OECD, 2016; Stiegler & Hiebert, 2009).  How teaching should be arranged in the best possible way has been targeted in a gigantic amount of investigations involving different theoretical points of departure (Hattie, 2009), and, consequently, reviews of the effectiveness/appropriateness of teaching methods have become increasingly available.

Producing such reviews appear as a logical way to proceed in order to integrate insights and findings from different studies. This line of reviewing research is seldom the target for criticism, let alone criticism that builds upon empirical analyses of the reviews themselves. On the contrary, they are often referenced as providing state-of-the-art descriptions of particular research areas. The aim of the present study is not to deconstruct such attempts to integrate research. However, we believe that teaching is a highly contextualized phenomenon, i.e. it takes its shape due to many different circumstances. Thus, we hypothesize that there are some more or less inherent problems when research about teaching methods is reviewed and more general conclusions are strived for. Results that are built upon particular studies in specific contexts are on the one hand to be integrated, and, on the other hand, used to provide suggestions for practice and future research in other contexts. Thus, there is a tension between the aim of reviews to come to some general conclusions and the contextual specifics of teaching. We approach the issue of problems to a large extent inductively, and focus on problems that are made more or less explicit by the review authors themselves. 

We will report an analysis of 75 of the most influential reviews of research on teaching methods, focusing on one overarching research question: Which reoccurring problems are pointed out in the reviews? We believe that identifying and discussing the nature of such problems can potentially advance the research area. The rationale behind choosing these reviews is that their high impact testifies to their importance to the research field. Since they are important and influential we assume that there is a high probability that they will be successful in identifying problems in the field and possibly also in presenting ideas on how such problems can be resolved. From this rather general point of departure, we have tried to inductively find out what problems that are identified in the reviews themselves.

The present study is an example of what Polanin, Maynard & Bell (2017) refer to as an overview, i.e. a review of reviews. Overviews have almost exclusively been concerned with the integration of meta-analyses. A prime example is Hattie (2009) who attempts to integrate the findings from over 800 meta-analyses of 144 different factors relating to educational attainment in what is labelled a “synthesis of meta-analyses” (see. Bergeron & Rivard, 2017; Polanin et al, 2017; Snook et al., 2009 for criticisms). In the present study we perform what can be considered as a markedly different type of overview.  Just as there is a need for different types of reviews, we argue that different kinds of overviews are needed due to their different purposes (cf Gough et al, 2012).

Prior to the analysis specific to the present study, extensive basic work had already been done. The 75 most cited research reviews on teaching methods listed in the Web of Science (WoS) between 1980 and 2017 had been identified (25 from 1980-1999, 25 from 2000-2009, and 25 from 2010-2017) through two searches in the WoS Core Collection. First, a simple search string was used (“teach* OR instr* OR curric* OR did* OR coach* OR guid* OR tut*”), secondly a combined search string was used (“teach* OR instr* OR curric* OR did* OR coach* OR guid* OR tut*” AND “review* or meta-analys* OR meta-narrative* OR meta-synthes* OR overview*”). The result lists from the two searches were matched, and by abstract reading, relevant top-cited reviews from each decade were identified for further reading and coding. Our main inclusion criterion was that the review should focus teaching methods in the K-12 context. All reviews were carefully read and encoded in a 27-feature scheme developed by the research group. During the coding process, it became apparent that underlying reviews largely discussed similar problems regardless of teaching method studied. This, in turn, led to further analysis, guided by an overall interest in inductively exploring the problems that appeared most frequently. As a first step, relevant features from the coding schemes of all the underlying 75 reviews were summarized in a table as follows: • Summary of main results • Summary of implications for practice • Summary of implications for research Our pronounced endeavor was to summarize on manifest level, i.e. without abstraction or interpretation. The subsequent inductive analysis of the summaries was inspired by Graneheim and Lundman’s (2004) qualitative content analysis approach. Each summary of results, implications for practice, and implications for research was considered a meaning unit that was labelled with codes. Next, the most frequently occurring codes were sorted into groups sharing a commonality, resulting in three overarching types of problems that we labelled: 1. The abundance of moderating factors 2. The need for highly qualified teachers 3. The research-practice gap Of these, categories 1 and 3 were so complex that further sorting into subcategories was carried out. When presenting the results, we use the term overview findings for our overarching problem areas. An overview finding can be described as a product of an accumulated analysis of individual review findings describing a phenomenon or aspects of a phenomenon (cf. Lewin et al., 2015).

Three main problems, cutting across reviews, appeared in the analysis as particularly common. 1/ The abundance of moderating factors An overview finding of high consensus is that a particular method has little or no effect per se, rather, our analysis shows that the effect depends on moderators linked to four (often interrelated) aspects: Differences in students, Differences in teachers, Differences in context, and Differences in content. We identified 30 moderator variables within these four areas (and it should be pointed out that we merely identified those explicitly addressed in the underlying reviews). Interestingly, the abundance of moderators is discussed as a methodological problem in the reviews, suggesting that it is believed that the issue of moderators is solvable in principle. 2/ The need for highly qualified teachers Overview finding 2 is linked to finding 1 and concerns the fact that moderating differences at the student level need to be recognized and compensated by the teacher organizing the instructional activities. In some of the underlying reviews this is explicitly discussed, but it is also a conclusion we draw on the basis of overview finding 1; since the effect of different methods is undoubtedly moderated by differences at the student level, the teacher's ability to adapt and balance the use of a particular method is crucial. 3/ The research-practice gap Despite the impossibility of providing “correct” answers applying once and for all, it can be argued that research over time (given certain caveats) points to the relative benefits of some methods. The methods do not seem to be put into practice by teachers, however. The research-practice gap is problematized and discussed in several of the included reviews. Our analysis shows that the causes of the gap can be related to three (often interrelated) aspects: Research, Teachers, and Context (with sub-categories).

Barber, M. & M. Mourshed (2007). How the World’s Best-Performing School Systems Come Out on Top. London: McKinsey & Co. Bergeron, P-J. och Rivard, L. (2017) How to engage in pseudoscience with real data: A criticism of John Hattie’s arguments in Visible Learning from the perspective of a statistician. McGill Journal of Education, 52 (1). Gough, D., Thomas, J., & Oliver, S. (2012). Clarifying differences between review designs and methods. Systematic Reviews, 1(28). Graneheim, U. H., & Lundman, B. (2004). Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Education Today, 24, 105–112. Hargreaves, A. & M. Fullan (2012). Professional Capital. Transforming Teaching in Every School. New York: Teachers College Press. Hattie, J. (2003) Teachers make a difference: W hat is the research evidence? Paper presented at the Building Teacher Quality: W hat does the research tell us ACER Research Conference, Melbourne, Australia. Retrieved from http://research.acer.edu.au/research_conference_2003/4/ Hattie, J. (2009). Visible learning. A synthesis of over 800 meta-analyses relating to achievement. London: Routledge. Lewin, S., Glenton, C., Munthe-Kaas, H., Carlsen, B., Colvin, C. J., Gulmezoglu, M., Noyes, J., Booth, A., Garside, R., & Rashidian, A. (2015). Using qualitative evidence in decision making for health and social interventions: an approach to assess confidence in findings from qualitative evidence syntheses (GRADE-CERQual). PLoS Med, 12(10). OECD (2016). PISA 2015 Results (Volume II): Policies and Practices for Successful Schools. PISA, OECD Publishing, Paris. https://doi.org/10.1787/9789264267510-en Polanin, J. Maynard, B. & Dell, N. (2017) Overviews in education research: A systematic review and analysis. I Review of Educational Research, 87(1), 172-203. Snook, I., O'Neill, J., Clark, J., O’Neill, A-M. and Openshaw, R. (2009) Invisible Learnings?: A Commentary on John Hattie's Book - 'Visible Learning: A Synthesis of Over 800 Meta-analyses Relating to Achievement, New Zealand Journal of Educational Studies , 44(1), 93-106.