Development of an ESD indicator for teacher training in the 16 federal states of Germany - A study of evaluation.

Author(s):

Eva-Maria Waltner(presenting / submitting)Werner Rieß Dietmar Glaubitz

Conference:

ECER 2017

Network:

30. Environmental and Sustainability Education Research (ESER)

Format:

Paper

Session Information

30 SES 06 A, Operationalising of ESD for Research Purposes

Paper Session

Time:

2017-08-23

15:30-17:00

Room:

K6.08

Chair:

Leif Östman

Contribution

Background:

The UNESCO Roadmap for Implementing the Global Action Programme (GAP) on Education for Sustainable Development (ESD) points out the crucial role of teachers and teacher training (TT) in ESD (UNESCO, 2014, p.20). This is not only true on the international level, but is an equally important focus of European and transnational endeavors connected to the Sustainable Development Goals (SDGs) and ESD. The Federal Ministry for Education and Research (BMBF) is the leading ministry in charge of ESD on the federal level in Germany. It launched a high scale national monitoring of the current state of implementation of ESD. The present project is part of this national monitoring within Germany’s GAP actions.

Research question:

At the University of Education in Freiburg an evaluation study was conducted to evaluate accessibility of data and the measurability of ESD relevant TT. Through collecting novel data, the project ESD_indicator_teacher training aims to answer the following research question: How accessible is the data on teacher training in the 16 federal states of Germany? Is it possible to develop a replicable and useful ESD indicator for teacher training as part of a larger, longitudinal set of ESD indicators for a continuous monitoring?

Consequently, the research group led by Prof. Dr. Werner Rieß developed an indicator for ESD relevance in TT provided by the government. Novel data was gathered for all the federal states in Germany. Thereby, the proportion of ESD relevant TT could be calculated for the 16 federal states. In the longer term, this indicator could contribute to the establishment of a national and even international concept of ESD indicators, which in the case of Germany could be integrated in the national Sustainability Strategy. Furthermore, in the future, it could become part of the general educational monitoring reports and giving answers to the following questions: “How do stakeholders know that an increased understanding of ESD programmes and indicators is developing? […] and Are stakeholders improving their ability to assess ESD progress? “ (Tilbury, Janousek, Elias, & Bacha, p. 32).

Theoretical framework:

“Indicators are considered to provide condensed information that can be transformed into knowledge relevant for decision making” (Bormann & Michelsen, 2010, p. 510). However, important preconditions of indicators are that they need to be seen as comprehensive, credible and fair in the eyes of their recipients (s. van Ackeren & Hovestadt, 2003, p. 23f, own translation). As the discourse on how to operationalise and define ESD is still an ongoing debate in research, it is important to guarantee a high level of transparency from the outset of the design of the indicator. This is why many stakeholders, including all spokespersons for ESD and TT from the 16 federal ministries of educations, have been included in the process of development of the national ESD indicator.

Compared to the ongoing discussions on different ESD definitions, there is consensus on the essential role of the teachers in ESD implementation (s. e.g.: Seybold, 2006, p. 172, own translation) and that, “[t]eacher professional development can be an effective way to enhance and ensure the quality of schooling […]” (Lipowsky 2015, p. 28). In the case of ESD this is equally stressed by the UNESCO Roadmap that states “[e]ducators and trainers [as] powerful agents of change for delivering the educational response to sustainable development. But for them to help usher in the transition to a sustainable society, they must first acquire the necessary knowledge, skills, attitudes and values “ (UNESCO, 2014, p.20).

Method

While the aim to develop an indicator for ESD TT has been stated since the beginning of the UN Decade for ESD and first attempts for its operationalisation have been suggested (s. e.g.:UNECE, 2006), many tasks on how to generate a set of ESD indicators that are specific, measurable, attainable, realistic and timely (s. Tilbury et al., p. 35) remain uncompleted. Preceding analysis on indicators for ESD mostly drew on qualitative interviews or self-reporting by the governments (s. Bormann & Michelsen, 2010; UNECE, 2006). However, research on ESD implementation still lacks quantitative methods and data. The indicatorisation of the ESD implementation process offers many possibilities for mapping the progress of ESD. In this analysis, quantitative data was gathered to examine the proportion of ESD relevant TT offered on the federal states level. For data collection on ESD relevant TT, a two-step procedure was chosen. First, the databases of the 16 federal states of Germany were scanned with the help of a comprehensive list of 37 search terms connected to multidimensional aspects of ESD. The extensive list of search terms captured 66.935 TT with possible ESD relevance to be further analysed. The collected data was analysed using Mayring’s qualitative content analysis (Mayring, 2014). We formulated strict content-analytical rules for the scanning process of the ESD relevance, measured by defined categories on an ordinal grading scale, by scanning the descriptions of teacher formation courses with the category coding system. To guarantee intercoder reliability the material was given to a second and third coder with results ranging form .71 to .93 (Cohens Kappa/ τ- correlation coefficient). As mentioned above wide acceptance and understanding are the basics of the successful development of a set of indicators. For this reason, transparency and the consultation of other ESD experts were considered as vital in the development process of the ESD category system. As the discourse on how to define ESD and how it can be appropriately operationalised is still an ongoing debate, a transparent category system is considered a central aspect in the development of the ESD indicator for TT. In the third phase of the evaluation project, the data gathered by the described two-step procedure was analysed and interpreted using MAXQDA-software. The findings constitute the most recent research on the ESD relevance in TT and its indicatorisation.

Expected Outcomes

By the end of this project, different formulas to calculate the degrees of ESD relevance in TT were applied to the states’ coding results. Consequently, the calculated indicators present a major evaluation that covers the entire German TT- system, the proportion of ESD relevant TT, as well as distinct dimensions and characteristics of ESD relevance. As mentioned above, the indicators’ task is to provide relevant and reliable information to decision makers (s. DIPF, 2007, p. 8, own translation). As a part of a larger indicator set, the indicator on ESD in TT offers useful insights on the state of art of the implementation of ESD. As “[f]ederal agencies view indicators as essential for monitoring the status of the nation's educational system and for tracking how it changes over time. […] State policy makers also seek indicator data that can inform new improvement efforts (Oekes, 1989, p. 181). The ESD indicator on TT will not only provide an easy index that might foster e.g. benchmarking processes in the field of ESD in TT. At the same time, it assesses detailed characteristics of the TT’ ESD relevance. To conclude, the ESD indicator on TT, as part a larger set of ESD indicators “promotes comparison of performance, monitoring and data collection” (Reid et.al., 2006). It is therefore an important driver in promoting the structural ESD implementation to be put into praxis. During the whole evaluation study, a special emphasis was put on the target of developing an indicator that fulfils the criteria of being “[…] based on adequate samples, [and having] appropriate levels of reliability, good validity, and above all, positive reactivity (Fritz-Gibbon & Tymms, 2002, p. 3).

References

Bormann, I., & Michelsen, G. (2010). The Collaborative Production of Meaningful Measure(ment)s: Preliminary insights into a work in progress. European Educational Research Journal, 9(4), 510. https://doi.org/10.2304/eerj.2010.9.4.510 DIPF. (2007). Das weiterentwickelte Indikatorenkonzept der Bildungsberichterstattung. [The evolved concept of indicators for Educational Reporting]. Retrieved from http://www.bildungsbericht.de/de/forschungsdesign/pdf-grundlagen/indikatorenkonzept.pdf Fritz-Gibbon, C., & Tymms, P. (2002 January 16). Technical and Ethical Issues in Indicator Systems:: Doing Things Right and Doing Wrong Things. Retrieved from http://epaa.asu.edu/ojs/article/viewFile/285/411 Mayring, P. (2014). Qualitative Content Analysis. Retrieved from http://nbn-resolving.de/urn:nbn:de:0168-ssoar-395173 Oekes, J. (1989). What Educational Indicators? The Case for Assessing the School Context. Educational Evaluation and Policy Analysis, 11(2), 181–199. Seybold, H. (2006). Bedingungen des Engagements von Lehrern für Bildung für nachhaltige Entwicklung. [Teacher motivation and incentives in ESD ]. In Rieß Werner & Apel Heino (Eds.), Bildung für nachhaltige Entwicklung. Aktuelle Forschungsfelder und -ansätze (pp. 171–183). Wiesbaden: VS Verlag für Sozialwissenschaften. Tilbury, D., Janousek, S., Elias, D., & Bacha, J. Asia-Pacific guidelines for the development of national ESD indicators; 2007. Retrieved from http://unesdoc.unesco.org/images/0015/001552/155283e.pdf UNECE. (2006). Indicators for Education for Sustainable Development: Addendum. Reporting format. Retrieved from https://www.unece.org/…/2006/ece/cep/ac…/ece.cep.ac.13.2006.5.add.1.e.doc UNESCO. (2014). UNESCO Roadmap for Implementing the Global Action Programme on Education for Sustainable Development. Retrieved from http://unesdoc.unesco.org/images/0023/002305/230514e.pdf van Ackeren, I., & Hovestadt, G. (2003). Indikatorisierung der Empfehlungen des Forum Bildung. [Indicatorisation of recommendations from the German discussion board for Education] . Retrieved from http://d-nb.info/971373620/34

Author Information

Eva-Maria Waltner (presenting / submitting)

Universtiy of Education Freiburg

Biology and Biology Education

Freiburg

Werner Rieß

Universtiy of Education Freiburg, Germany

Dietmar Glaubitz