Intersections, Commonalities, And Conflict: Developing A Harmonious Research Partnership Between Intervention And Evaluation Teams.

Author(s):

Jo Rose(presenting / submitting)Chris Brown(presenting)

Conference:

ECER 2017

Network:

15. Research Partnerships in Education

Format:

Paper

Session Information

15 SES 07, Case Study (Part 2)

Paper Session continued from 15 SES 06 and to be continued in 15 SES 08

Time:

2017-08-23

17:15-18:45

Room:

K6.16

Chair:

Philippe Masson

Contribution

This paper considers the ways in which research relationships develop between educational evaluators and those who they are evaluating, and the potential for genuine partnership within this context. We will draw on our experience of educational intervention and evaluations in general, and in particular on a recent intervention and evaluation of Research Learning Communities in English primary schools, which was funded by the Education Endowment Foundation, to conceptualise the process of partnership in an intervention/evaluation context.

The paper asks:

What are the key stages in developing and maintaining partnerships in an intervention/evaluation relationship?
In what ways might a partnership become resilient to failure?

The processes of coming together and developing collaborative work have been conceptualised by Rose and Norwich (2014). They highlight the roles of development and commitment to shared goals, of collective efficacy in working towards those goals, and of process/outcome beliefs in developing and maintaining collaborative processes. Rose and Norwich’s model informs our discussion of the partnership processes in intervention and evaluation teams. Edwards’ (2005, 2007) concept of relational agency, in which she outlines the ways in which interaction influences the development of people’s thinking, and their ways of working which they then develop in the future, will help us consider the ways in which members of intervention and evaluation teams can change as a result of coming together.

The authors have between them wide-ranging experiences of educational intervention and evaluation. This paper arises from their shared experiences on the Education Endowment Foundation (EEF) funded Research Learning Communities (RLC) project, as leads of the intervention team (Brown) and the evaluation team (Rose). The EEF had a particular model of educational evaluation, which privileges randomised control trials (RCT) as a method. Intervention and evaluation teams are brought together in the development stages, so that the intervention can be delivered in such a way to allow for an RCT evaluation. The involvement of the EEF makes this model of partnership particularly interesting: the EEF prescribes the outcome measures and scope of the evaluation, and is strongly involved in determining the design of the intervention and evaluation before leaving the intervention and evaluation teams to conduct the project.

The RLC intervention brought together small clusters of primary schools, and worked with key members of staff (“evidence champions”) in each school over the course of two years to develop research informed practice. The intervention team conducted four workshops each year with evidence champions in each cluster, to plan and support delivery of evidence-informed action research in their schools. The intervention team worked with 58 schools in 14 clusters, with another 58 schools being allocated to the control condition of the RCT. Control schools did not receive the intervention, but received a financial incentive for providing evaluation data. The evaluation team collected survey data from teachers in all schools, on their research-informed practice, and pupil attainment data from all schools which the EEF prescribed as the primary outcome measure. The process evaluation included surveys of the evidence champions, and case studies of six intervention schools and two control schools. The evaluation team also attended the initial school recruitment events organised by the intervention team at the start of the project, and attended one example of each workshop.

Method

In this paper, we take a multiple case-study approach to propose a framework of research partnership processes in educational intervention and evaluation. We reflect on our experiences in the particular substantive case of developing a research partnership in the RLC project, drawing also on our history in other interventions and evaluations with variations in the ways in which the teams come together and in which the intervention and evaluation are developed. The RLC project is indicative of several contemporary intervention/evaluation partnerships that are or have been funded by the EEF, so is a useful case for consideration. In our analysis, we will focus in particular on the role of what is shared, what is separate, and what is conflicting, between the intervention and evaluation teams, to help us understand the different stages of the partnership process and the potential points of conflict. The role played by the funder in the RLC project will be used to further consider the potential for commonalities and conflicts in the process. The pathways taken by intervention and evaluation teams can be broadly conceptualised in similar ways, with intersections between the two taking place at specific points and with specific agendas. These points of intersection will vary according to the specific model of bringing together the intervention and evaluation teams. The following stages are likely to occur in an intervention/evaluation partnership (not necessarily in this sequence), and all are potential sites of conflict: Application for funding (intervention) Application for funding (evaluation) Design and development (intervention) Design and development (evaluation) Recruitment of participants (intervention) Recruitment of participants (evaluation) Delivery (intervention) Data collection (evaluation) Analysis of process Analysis of outcomes Reporting of process and outcomes.

Expected Outcomes

There are potential intersections between the intervention and evaluation at all stages, depending on the model used to bring the intervention and evaluation teams together. There are merits and dangers of these processes being closely intertwined, or clearly distinct. An intervention that develops separately from the evaluation may be conceptually unadulterated but problematic in developing an appropriate evaluation design. Developing an intervention alongside an evaluation, however, runs the risk of twisting the concept of the intervention to fit the evaluation model. An evaluation that informs the ongoing development of an intervention with an emergent and discursive approach, can lead to greater insight into the process of the intervention, but will ultimately lead to a less robust outcome evaluation. Shifting power dynamics (in terms of design, access to participants and data, and reporting) in the partnership between the intervention and the evaluation can impact on potential for conflict at different stages of the process. This leads us to discuss what we believe the purpose of evaluation is: should an evaluation be methodologically robust in terms of attributing causal effects to outcomes, or should the evaluation inform the development of an intervention? A shared goal in terms of what the evaluation aims to achieve is a key aspect of developing a harmonious partnership between the intervention and evaluation teams. In our case study of the RLC project, the method of the evaluation was prescribed by the funder - this meant that the purpose of the formal evaluation was explicit (in terms of being a “robust outcome evaluation”) and both the intervention and evaluation teams had to comply with this to secure funding. Through the development meetings, however, we realised that we both had a different agenda to the funder, in terms of what we believed the intervention and evaluation could and should achieve.

References

Edwards, A. (2005). Relational agency: Learning to be a resourceful practitioner. International Journal of Educational Research, 43, 168–182. Edwards, A. (2007). Relational agency in professional practice: A CHAT analysis. Actio: An International Journal of Human Activity Theory, 1, 1–17. Rose, J. and Norwich, B. (2014) Collective commitment and collective efficacy: a theoretical model for understanding the motivational dynamics of dilemma resolution in inter-professional work. Cambridge Journal of Education, 44, 59-74.

Author Information

Jo Rose (presenting / submitting)

University of Bristol, United Kingdom

Chris Brown (presenting)

University College London, United Kingdom

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.