English educational policy discourse emphasises the importance of 'evidence' and randomised controlled trials (RCTs) have become promoted as a 'gold standard' methodology to provide this (Haynes et al., 2012; Higgins, S. et al., 2014). Whilst endorsed by commissioners of policy research, the arrival of RCTs in England has been met with a mixed reception from the education research community.  It has re-fuelled debates reflecting historical methodological disputes, with advocates contending that they are uniquely capable of establishing causality while others consider their application in education as flawed.

In this paper we draw on our recent experiences conducting a number of large scale RCT based evaluation in England (Maxwell et al., 2014a; Maxwell et al., 2014b, Demack, 2014).   We also draw on our work on projects that have been methodologically influenced by the rise of the RCT, such as a current evaluation of the impact of an exchange programme between Shanghai and English primary mathematics teachers.  We discuss the use of RCTs internationally to inform education policy research that is increasingly claiming to be 'evidence based' and consider the historic popularity trajectory of their use.  We base our discussions on practical experiences and the challenges we met in conducting these RCT based research projects evaluating teacher focused professional development (CPD) and pupil focused literacy, transition and mentoring programmes.  

Considering RCTs in the context of teacher professional autonomy illustrates the polarised debate that surrounds their use. It can be argued that RCTs support teachers' professional autonomy by providing 'robust' evidence to respond to ideological policy initiatives.  For example, the Office for Standards in Education's (OFSTED) recent 'encouragement' to introduce ability group setting (Paton, 2014) can be countered by evidence that ability group setting is not an effective strategy for raising attainment and can result in learners falling behind (EEF, 2015).  Similarly, within an increasingly commercialised education system, RCTs can be used to help distinguish between / fend off providers of education 'products' that are jostling for market position and access to schools and/or school networks.  However, RCTs can also be seen as reflecting decades of education policy that has been dominated by a within-school, within-classroom and individual teacher focus whilst paying little to no attention to schooling more broadly.   This 'all about the teacher' focus has led to what Reay calls a paradoxical process that has seen teachers lose professional autonomy via surveillance and an increasingly prescribed curriculum whilst simultaneously (and fallaciously) being extolled as being able to transform educational failure into success (Reay, 2006 p291-292).    In other words, teachers are required to be 'inspirational' whilst being watched all the time following a narrowly prescribed curriculum in a specified manner.   RCTs might be considered as being part of the apparatus for further erosions in teacher professional autonomy.

We discuss our RCT experiences in terms of: practice, purposes, ethics, measurement and theoretical conflicts. 

Practical issues arise from the complexity of the education environment and the 'agency' of trial participants. This leads to challenges in 'fidelity' and overcoming these can have methodological and ethical implications. Further, funders' purposes and trial timescales may not allow for initial implementation dips that can occur when new innovations are introduced into educational settings (Fullan, 2001).   Measuring 'impact' through quantification of educational processes and outcomes bring further ethical and practical issues. 

We focus on three theoretical conflicts. Firstly, between what is important for learning (agency and variation) and what is required for an RCT based evaluation ( consistency and uniformity). Secondly, between the complex and recursive nature of learning and the linear pathways in RCT designs.  Finally, between a 'within-school' and 'within-classroom' focus and the schooling system more broadly.

Demack, S. (2014) Public Registration Protocol for Multiplicative Reasoning Cluster Randomised Controlled Trial. Available at www.isrctn.com/ISRCTN63650913 EEF (2015) Setting or Streaming: Negative impact for low cost, based on moderate evidence. Available at http://educationendowmentfoundation.org.uk/toolkit/ability-grouping/ (last accessed 27/1/2015). Haynes, L., Service, O., Goldacre, B. & Torgerson, D. (2012) Test,.Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. London HMSO Cabinet Office Behavioural Insights Team. Higgins, S., Katsipataki, M., Coleman, R., Henderson, P., Major, L.E., & Coe, R. (2014). The Sutton Trust-Education Endowment Foundation Teaching and Learning Toolkit. October 2014. London: Education Endowment Foundation. Fullan, M. (2001). Leading in culture of change. San Francisco: Jossey-Bass. Maxwell, B., Connolly, P., Demack, S., O'Hare, L., Stevens, A and Clague, L. (2014) Evaluation of the Summer Active Reading Programme EEF available at http://educationendowmentfoundation.org.uk/projects/summer-active-reading-programme/ Maxwell, B., Connolly, P., Demack, S., O'Hare, L., Stevens, A and Clague, L. (2014) Evaluation of the TextNow Transition Programme EEF available at http://educationendowmentfoundation.org.uk/projects/textnow/ Paton, G. (2014) Secondary schools may be forced to set pupils by ability. Daily Telegraph Sept 3rd 2014. Reay, D. (2006) The zombie stalking English schools: social class and educational inequality. British Journal of Educational Studies 54(3) pp288-307