The Challenge of Evaluating an ESD Programme Consistently with ESD Values: the Experience of the CoDeS-ENSI Network.

Author(s):

Michela Mayer(presenting / submitting)Christine Affolter Patrick Dillon

Conference:

ECER 2014

Network:

30. Environmental and Sustainability Education Research (ESER)

Format:

Paper

Session Information

30 SES 10 A, Effects of ESE implementations and Projects

Paper Session

Time:

2014-09-04

15:30-17:00

Room:

B119 Sala de Aulas

Chair:

Jutta Nikel

Contribution

Evaluation has been a major challenge for Environmental Education generally and especially for Education for Sustainable Development. A ‘Sustainable Education’ (Sterling, 2002) requires deep change from current educational aims and methodologies to those that are consistent with the needs of change, not only of lifestyles but also of visions of the world. Complexity, systemic thinking, participation and collaboration, situated knowledge, attention to ‘emergence’ and to ‘wicked problems’, future visioning, openness to change, are some of the many key constructs that accompany educational research on ESD. Evaluation is a fundamental component of this continuously advancing context, not only in relation to student learning but also concerning school performance and projects and programmes for ESD. National and international funding still asks for a kind of evaluation that could be defined as ‘positivistic’, concerned more with quantity than quality, and which tries to reduce to ‘objective’ and ‘measurable’ items the data collected and the results obtained. In this situation, the evaluation of a programme may conflict with the need for formative development that is consistent with the main aims of ESD.

ENSI INpA (Environment and School Initiatives International No Profit Association)vision and methodologies inspired the evaluation ‘paradigm’ used for evaluating three projects in the Comenius 3/Life Long Learning EU programmes– SEED (Tilbury et al. 2005), SUPPORT (Mayer, 2010) and CoDeS - the last of which is still in progress. The paradigm explicitly accepted for the evaluation framework has been ‘socio-critical’ (Robottom and Hart, 1993), going beyond the idea of evaluation as an assessment process and proposing instead a vision of an evaluation which proceeds through ‘negotiating values’, ‘searching for quality’, ‘taking care of unexpected issues’, and ‘constructing new meaning’. Consequently the evaluation has tried to take account not only of facts and concrete outcomes, but also the values of the participants, their visions of ESD and of the qualities they want to achieve within the project; not just products but also processes. Overall the evaluation has been concerned with the process of community building within the partners’ network and by the quality of the ‘internal learning’ of the network itself, considered as ‘a learning organization’. In this vision the project may be regarded as a tool for improving the ‘theory and practice’ of ‘education for sustainability’ not only for the project partners but also for the ESD community as a whole.

The research described is about the methodologies that have been used, and the strengths and weaknesses of these approaches in meeting the key aims of ESD.

Method

The methodology was largely inspired by the critical theory and action research tradition in the ENSI network. The validity of the results has been mainly based on the systematic use of ‘triangulation’. The collection of data has been divided between the main evaluation actors: • the ‘coordinator’ of the project, who takes care of communication with partners, monitoring of the events and of the products, and the formal reporting of the programme; • the ‘internal evaluator’, who collects the reports of the partners with responsibility for single activities or products, and offers tools – questionnaires, guide lines for reporting, etc. – for the internal evaluation of the events; • and the ‘external evaluator’, who offers to the network a different point of view on the same data, new data collection, and quality improvement tools to facilitate the processes of critical reflection between partners and ongoing constructive development through collaboration. The triangulation is a characteristic also of the data, often collected using more than one method and paying attention to different voices. The systematic comparison of different visions and opinions minimises possible biases, included the ones of the evaluators: the aim is not an impossible objectivity but rather impartiality. The participation and involvement of all partners and stakeholders in the evaluation process has been another important feature of the project: the main aim was to help the community building process through collaboration, engaging partners during the whole process, mapping emergent ideas, giving regular feedback. In the CoDeS project another innovative evaluation method was added: the internal evaluation of the three main conferences within the project was given each year to different young researchers in ESD. These young researchers were not involved in the partner institutions; they were new to the project. In this way, the evaluation process has integrated fresh research perspectives and at the same time allowed people coming from different traditions to comment on the vision of the project. There are logistical difficulties in an evaluation of this kind: the EU-schedule is difficult to integrate with effective conduct of the research; the limited financial support for project partners means that publications, work in communities, translations, opportunities of exchange etc are always tightly restricted; the shortage of time for in-depth development; and the positivistic attitude toward evaluation which is prevalent in general.

Expected Outcomes

The evaluation of the projects using these methodologies has been highly appreciated by the EU and has offered a shared understanding, both in term of products and learning outcomes between partners. Additional outcomes from this kind of evaluation are reflections on emerging issues and tensions, not only those connected to specific projects, but also collectively from the many and diverse partners (29 institutions in CoDeS, representative of real stakeholders involved in community projects). For example: • ‘Legitimating differences’ while improving reciprocal understanding. We know in ESD that differences are crucial to building something new, but collaboration notwithstanding, differences are always a great challenge. The purpose in ESD is not to overcome differences - in language, culture, visions – rather to manage them in the construction of trust and reciprocal understanding; • Internal and external to the partner networks has been the tension between ‘action’ and ‘thought’. While actions without responsible thought have driven our society to unsustainablity, there are still pressures toward ‘fast’ and ‘effective’ actions; the practice of critical thinking will help the ‘activist’ accept also the moment of reflection; • When different groups involved in ESD are asked to collaborate there is the problem of reconciling different priorities and different quality criteria. Often they remain implicit and it is only during the actions that partners discover how different are their expectations and their judgments when confronted with practice. • ESD is a topic which attracts people wanting to work for a better future. Such people are usually not strong in solving conflicts because of their ethical values. Solving conflicts is important in making SD a reality. Negotiation and conflict resolution are important matters emerging from our evaluation. An evaluative frame like the one proposed in this presentation offers possibilities to make these tensions explicit and to face them.

References

Mayer M. (2010), Partnership and participation for a Sustainable Tomorrow. External Evaluation of the SUPPORT Project, www.support-edu.org Robottom J.& Hart P. (1993) Research in Environmental Education. Engaging the debate. Victoria, Deakin University Sterling S. (2002), Sustainable Education, Schumaker Briefings no. 6, Green Books Ltd. Tilbury D., Anderson K., Cooke K. (2005) External evaluation of SEED. Final report, SEED documents, www.ensi.org

Author Information

Michela Mayer (presenting / submitting)

ENSI INpA

Roma

Christine Affolter

ENSI INpA, Switzerland

Patrick Dillon

University of Exeter

Exeter

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.