The evidence-based movement has grown all over the world and systematic research reviews intensely requests, not at least by policymakers. The dominating research in this field has foremost built on positivistic and neo-positivistic methods. However the research landscape years back has shown research calling for transferred and changed approaches towards a more including methodology, were interpreting, narrative and critical traditions contribute to a widened methodological perspective in the research field of education, not at least in European context.  Working with research synthesis and producing research reviews is beside its own value, of value in every research project in its initial phase.

A newborn course intended for PhD students with the title ‘Research Synthesis and Methodologies – about analyzing and producing research reviews’ is the object of this study in its initial phase. The course described in detail in this study points at an elaborated one in literature, seminars and methodology. Besides methodological knowledge, the literature in this course illustrate the discussion in between researchers and their different perspectives on reviews and synthesis – from the evidence-based advocates to the critical ones that rather would see a methodology including qualitative research approaches, narratives for example and critical views in general. (See e.g. Gough, Oliver & Thomas 2012; Andrews 2005; Davies 2000; Ball 2001; Dyson, Thomson & Diacks 2001; Hammersley 2008; Clegg 2005; Zhao 1991; Kastner et al. 2012; Dixon-Woods et al. 2006; Suri & Clarke 2009; Nye, Melendez-Torres & Bonell 2016).  The course given was in cooperation with librarians (and information science) and the focus was on methodologies and ways of doing an elaborated research synthesis in accordance with a critical study of the research discussion in the field (the course literature).

This paper describes an explorative study; following seminars, reanalyzing examinations and evaluating the PhD student’s perceptions of the course in general; the need for this kind of courses, knowledge of research reviews and methodology, and knowledge of the infected discussion related to the evidence movement.  What can we learn from this case study?

The students’ perceptions of the course is pointing at a too heavy burden when it comes to the literature. Apart from this critique, the PhD students have a highly positive view of the course in every detail and all over ambition. There is however, scattered critique towards the most important methodological part of the course; it is not enough emphasized. This is interesting to note in relation to what this study also reveals; that these PhD students think that a thoroughly done research review is more or less a Sisyphus-work. It seems impossible to start a research project with mapping research already performed in the field in question. Moreover, knowledge at this PhD level about search engines and search processes is elementary or simply not there at all.

In this project, a multimethodological approach is at hand. A methodological pluralism approach is not simply about mixing methods. The attitude is not to decide on an epistemological commitment a priori, but rather to choose the methods that best suit the task (Alvesson and Deetz 2000). It is about integrating quantitative and qualitative approaches and combining methods in a way that the study successively needs to answer the research questions. It is about moving away from traditionally labeled quantitative or qualitative designs (Åsberg 2001), and methodological pluralism exists when measurement, explanation, and interpretation is considered together (Williams and Vogt 2011). This project took a multi-methodological approach combining qualitative and interpretative approaches with a quantitative bibliometric approach. At first documented, analyzed and commented seminars during the course are included in the analysis of the material. Secondly, an evaluation of the course, a questionnaire for the PhD students to answer, is included. Thirdly, the examination papers, which the PhD students wrote, are a part of the study and the analysis. The examination papers show the result of their own research review and its bibliometric methodology, but also the PhD student’s views and perceptions of the course, its aims and performance, together with their own analyze of their own review with the help of the literature in the course. The study in question, grew to become a large-scale research project in its analyzing and interpreting phase using bibliometric methods relating it to research in the field; research about educational research generally (and the lack of it), and publications in two European Journals particularly.

In the aim of exploring PhD students’ perception of the course ‘Research Synthesis and Methodologies’, two conclusions are to be drawn: At first, everyone is praising the course, the possibilities research review methodologies can offer, and its importance for the dissertation to come. Secondly, according to the results, there seems to exist a widespread opinion that research in the field in questions for an individual researcher is because of its scope impossible to grasp. Sometimes but more seldom research in the field of interest is perceived as non-existent. These perceptions are supposed to relate to knowledge of search engines, search methods and search processes. It seems that although many efforts, since among others Kulthau (1993) presented her research about difficulties in research processes, to reach a high level of information literacy, these efforts has failed. To many in this group of PhD students were unfamiliar with databases storing educational research and how to approach the review expected in the examination, when entering the course. However, taking the perceptions seriously new questions raise: Is it a widespread perception that it is impossible to map the educational research field in question in different cases? What about the educational field at large? What about research about educational research generally? The result of this study points to that research about educational research and self-reflection is called for, and bibliometric methods could be powerful in this aim followed by analyzing and theorizing the result (Bourdieu, 2004). That is research-synthesis and large-scale studies – using methodologies beyond what characterize the evidence-movement. This presupposes a marriage between information scientists and educational researchers. It has been sad before.

Alvesson, M., and S. Deetz 2000. Kritisk samhällsvetenskaplig metod. [Critical social science method]. Lund: Studentlitteratur. Andrews, R. (2005). The place of systematic reviews in education research. British Journal of Educational Studies, 53(4), 399-416. Ball, S. J. (2001). "You've been NERFed!" Dumbing down the academy: National Educational Research Forum: "A national strategy-consultation paper": A brief and bilious response. Journal of Education Policy, 16(3), 265-68. Bourdieu, P. 2004. Science of Science and Reflexivity. Chicago: The University of Chicago Press. Clegg, S. (2005). Evidence-based practice in educational research: A critical realist critique of systematic review. British Journal of Sociology of Education, 26(3). 415-428. Davies, P. (2000). The relevance of systematic reviews to educational policy and practice. Oxford Review of Education, 26(3/4), 365-377. Dixon-Woods, M. et al. (2006). How can systematic reviews incorporate qualitative research? A critical perspective. Qualitative Research, 6(1), 27-44. Dyson, A., Thomson, M., & Diack. A. (2001) Responses to Ball, S. J. (2002) You´ve been NERFed! Dumbing down the academy: National Educational Research Forum: ‘… response in JEP, 16(3), 265-268. Journal of Education Policy 17(1), 139-143. Gough, David, Oliver, Sandy & Thomas, James (red.) (2012). An introduction to systematic reviews. Los Angeles, Ca: SAGE Hammersley, M. (2008). Paradigm war revived. On the diagnosis of resistance to randomized controlled trials and systematic review in education. International Journal of Research & Method in Education 31(1), 3-10. Kastner, M. et al. (2012). What is the most appropriate knowledge synthesis method to conduct a review? Protocol for a scoping review. BMC Medical Research Methodology, 12(1) 114. Kulthau, C. (1993). Implementing a process approach to information skilss: A study identifying indicators of success in library media programs. School Library Media Quarterly 22(1) 11-18. Nye, E., Melendez-Torres, G. J. and Bonell, C. (2016). Origins, methods and advances in qualitative meta-synthesis. Review of Education, 4(1), 57–79. Suri, H., & Clarke, D. (2009). Advancements in research synthesis methods: From a methodologically inclusive perspective. Review of Educational Research, 79(1), 395-430. Williams and Vogt 2011 Zhao, S. (1991). Metatheory, metamethod, meta-data-analysis: What, why, and how? Sociological Perspectives, 34(3). 377-390. Åsberg, R. 2001. Det finns inga kvalitativa metoder- och inga kvantitative heller för den delen. Det kvalitativa-kvantitativa argumentets missvisande retorik. [There are no qualitative methods- and no quantitative either for that matter]. Pedagogisk Forskning i Sverige. [Educational research in Sweden] 6 (4) 270-292.