The paper compares the methods used in recent UK Research Assessment evaluations, including the Research Assessment Exercise (RAE) and the Research Excellence Framework (REF) periodic evaluations run by the Higher Education Funding Council for England, with the 2014 Fundação para a Ciência e a Tecnologia/European Science Foundation  Research and Development Centres evaluation in Portugal. It might be argued that the scale of the exercises and the different contexts – the UK does well in international league tables of research but even amongst European countries Portugal is towards the bottom end of the league tables and whilst both countries have experienced recent elements of austerity regimes, Portugal’s experience following the Eurozone crisis has been at the extreme end of the spectrum - make the comparison problematic.   But this can also be an advantage because just as it is useful, as Adler-Nissen and Kropp (2015) note, writing about approaches to the sociology of knowledge and their usefulness for understanding how European social science is contributing to European integration note, to draw on a number of important principles, including situatedness (the micro and meso context of academics and their institutions) and the national and regional cultural contexts which shape knowledge, so such principles are apposite for this analysis too.   An even more salient point is that the two countries have made use of very different evaluation methods and there is considerable scope for European academics and policy makers to learn from the intended/unintended consequences of both exercises.   

The main research questions and objectives of the paper are to: 

1. Compare two different ways of evaluating research excellence and distributing research funding in relation to what is being evaluated, by whom and how and the funding decisions and other consequences arising from the evaluations. 

2. Assess the strengths and weaknesses of both forms of evaluation and the main criticisms of each. 

3. Explore some of the unintended consequences of both systems for universities and researchers, given the very different cultural, social and economic features of the national contexts.

Theoretical framework

The paper considers both the intended and unintended consequences of the UK and Portuguese research evaluations.  We know that educational and science policies almost always have unintended consequences (Margetts et al 2010) and following Krücken (2014), use is made of an  adapted version of  Merton’s (1936) work on the unintended consequences of what he termed ‘purposive social action’.  As Krücken (ibid) notes, Merton’s work refers to social actors not organisations and Krücken suggests that this emphasis needs to be reframed to take account of organisational actions, since policy decisions are rarely taken by individual actors. Hence, the focus here is placed on what Krücken describes as the ‘idea of a discursive field in which remarkable change processes take place’ (2014, p 1440).   This could also be compared to Bourdieu’s (1989, 1993)’s notion of a contested field. Merton suggested five causes of unintended consequences: error, ignorance, immediate interest, basic values and self-defeating prophecy.  At least four of these seem relevant in this context, that is failing to model or not knowing about unintended consequences, focusing on the current immediate context and those taking the purposive social action having different values. This is helpful because as a recent European University Association publication observes (Pruvot and Estermann 2014), exercises to evaluate excellence, whether research or teaching, are on the increase in Europe and this seems set to continue whilst (scarce) public funding is still available.

Adler-Nissen, Rebecca, Kropp, Kristoffer (2015), A sociology of knowledge approach to European integration: four analytic principles Journal of European Integration, 37, 2, 155-173 Bourdieu, Pierre. (1988). Homo Academicus, P. Collier, translator, Cambridge: Polity Press European Science Foundation (2014) FCT R&D Units Evaluation by ESF. http://www.esf.org/serving-science/fct-rd-units-evaluation-by-esf.html Higher Education Funding Council for England, (2005). Research Assessment Exercise 2008: Guidance on Submissions. Higher Education Funding Council for England, Bristol. Higher Education Funding Council for England. (2012). Research Excellence Framework 2014 Panel Criteria and Working Methods, HEFCE, Bristol Krücken, Georg (2014) Higher education reforms and unintended consequences: a research agenda’ Studies in Higher Education 39, 8, pp 1439-1450 Lucas, Lisa. (2006). The research game in academic life, Maidenhead: Open University Press & the Society for Research into Higher Education. Margetts, H., Perri, and Hood, C. (2010). Paradoxes of Modernization. Unintended Consequences of Public Policy Reform. Oxford University Press: Oxford. Merton, Robert K (1936) The unanticipated consequences of purposive social action American Sociological Review 1,6, pp 894-904 Pruvot, Enora B, Estermann, Thomas (2014) Define Thematic Report: Funding for Excellence, European Universities Association: Brussels Richie, J and Spencer, L (1994), ‘Qualitative data analysis for applied policy research’, in Bryman and Burgess, eds., Analysing Qualitative Data, London: Routledge, p173-194. Scott, J. (2006 ). Documentary Research: Theory and Methods. City: Sage London. Srivastava, A. & Thomson, S. B. (2009).Framework Analysis: A Qualitative Methodology for Applied Policy Research. Research Note Journal of Administration and Governance, 4, 2, 72-79