Using a Theory of Change to Evaluate Complex Initiatives: Theory and practice

Author(s):

Karen Laing(presenting / submitting)JIll Clark(presenting)Lucy Tiplady(presenting)Liz Todd(presenting)Pamela Woolner(presenting)

Conference:

ECER 2014

Network:

04. Inclusive Education

Format:

Research Workshop

Session Information

04 SES 06 B, Using a Theory of Change to Evaluate Complex Initiatives: Theory and Practice

Research Workshop

Time:

2014-09-03

15:30-17:00

Room:

B014 Anfiteatro

Chair:

Karen Laing

Contribution

The authors draw on their experience of using the Theory of Change (TOC) in very different evaluation contexts. These evaluations vary in scale, location (based in schools or in the community) and also in the exact way that ToC methodology has been utilised. The workshop presents developments in our research practices, our thinking about this methodology and in our understanding of how it may be used. Our intention is not simply to recount uses of the methodology, demonstrating its power as a tool, but to explore our understanding of ToC as a methodology and its potential to contribute to research methodology, particularly in the conduct of evaluations. We therefore propose to conduct a workshop, which will explore the theory and practice of conducting a Theory of Change evaluation.

The workshop will consist of three elements. A brief overview of the Theory of Change will be given, outlining,

What is a Theory of Change?
How is it different from other evaluation methodology?
What is involved in a Theory of Change
How to construct a Theory of Change

Following this, participants will be asked to construct their own Theory of Change for a hypothetical evaluation, exploring what kinds of questions need to be asked, and enabling them to consider the practicalities for such an approach, and aspects such as the importance of collaboration, criticality and outcomes.

Finally, the authors will discuss the use of evidence and data within a Theory of Change and facilitate a discussion aiming to address particular questions that participants have about this method.

Our experience as facilitators in a range of contexts demonstrates that this mix of content information, based on our research experience, together with opportunity to engage practically with a methodology and also space for participants to share experiences and ideas is particularly effective. Participants’ discussions build upon their diverse practices within education, drawing on shared theoretical knowledge to construct collaborative understandings which they can take back into their practice as researchers and/or educators.

Theory of Change is beginning to be used in an international context, and is growing in popularity. It would seem timely to offer this workshop to provide an insight to international researchers of a new framework to conduct research that they may find valuable.

Method

Theory of change (ToC) as a research methodology is growing in use and recognition (e.g. Batchelor, 2013), and is particularly suited to evaluations of complex school and community initiatives. As argued by early UK exponents of the methodology, Alan Dyson and Liz Todd, it is appropriate 'where initiatives are multi-strand and/or variable in form, where intended outcomes are multiple, or poorly specified, or contested, and where the context of implementation contains many uncontrolled variables' (Dyson & Todd, 2010: 124). It is important to note that these features should not simply be seen as evidence of poor research design, but instead are inevitable aspects of the real world that evaluations of necessarily complex and under-theorised initiatives must cope with. Theory of change offers a valid way for researchers to do this and makes some specific contributions to the research process. Working with actors to articulate ideas about how change is happening, or is anticipated to happen, enables examination of the theory for coherence as well as providing clarity about data to collect as evidence of immediate and intermediate steps, as well as longer term change. As noted by Dyson and Todd, this gives rise to two important features of the methodology: the potential for formative assessment for the initiative’s enactors and the possibility of negotiating outcomes to assess. In essence, adopting a theory of change methodology facilitates an iterative research process, based on dialogue and discussion. As a methodology, it is particularly suited to evaluating initiatives to support inclusion. As Dyson and Hick (2005) point out, there is no one pedagogy that can address, for example, low attainment, but that a whole school approach may be needed. This entails a complex mix of different interventions, where outcomes may not be seen for a long time. Theory of Change can be an effective method of evaluation, utilising a mixed method approach, where other methods may be inadequate. The method itself is a particularly inclusive one, as it entails working in partnership with those delivering interventions or initiatives, and can work on a multi-level basis with strategic leaders, operational staff and users,\surfacing elements of difference and agreement. It can encompass participatory approaches to research and innovative methodologies, thus engaging a variety of stakeholders in a meaningful way.

Expected Outcomes

The workshop is intended to enlighten and inform the research community. In particular, we aim to contribute to the following outcomes: For educational researchers: • Inspiring them to consider the contexts in which a Theory of Change approach may be useful • Giving them the knowledge , skills and confidence to be able to use Theory of Change in their work For Educational practitioners: • By encouraging researchers to think about using methods that are appropriate to complex evaluation, we aim to encourage better practices, and the production of better outcomes data • We hope to be able to explore with participants the potential applicability of our methods in other European contexts

References

Dyson, A. And Hick, P. (2005) ‘Low Attainment’ in A. Lewis and B. Norwich (eds) Special Teaching for Special Children? Pedagogies for Inclusion. Maidenhead. Open University press. Dyson, A and Todd, L. (2010) 'Dealing with complexity: theory of change evaluation and the fullservice extended schools initiative', International Journal of Research & Method in Education, 33(2): 119 – 134 Batchelor, S. Fisher, C. and Beynon, P.(2013) IDS Practice Paper In Brief 15: Learning about theories of change for the monitoring and evaluation of research uptake Institute of Development Studies http://www.ids.ac.uk/publication/learning-about-theories-of-change-and-embedding-reflection

Author Information

Karen Laing (presenting / submitting)

Newcastle University, United Kingdom

JIll Clark (presenting)

Newcastle University

Research Centre for Learning and Teaching

Newcastle upon Tyne

Lucy Tiplady (presenting)

Newcastle University, United Kingdom

Liz Todd (presenting)

Newcastle University, United Kingdom

Pamela Woolner (presenting)

Newcastle University

Newcastle upon Tyne

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.

Session Information

Contribution

Method

Expected Outcomes

References

Author Information

Search the ECER Programme

Navigation

Info for