The globalizing world is shaped by a dynamic change which carries numerous challenges confronting the global society, such as for instance, poverty, climate change, or migration. The sustainable development goals of the UN (SDGs) aim to provide answers to these challenges. In order to monitor progress in tackling the challenges, indicators are needed for each of these goals and their sub-goals. One sub-goal of Development Goal 4 - Ensuring inclusive and equitable quality education and promote lifelong learning opportunities for all - relates to Education for Sustainable Development: SDG 4.7. Within the framework of SDG 4.7, it is to be monitored to which extent   Education for Sustainable Development (ESD) are implemented in national education policy, in the curricula, in teacher training and in the assessment of learning outcomes.

Our project “SysKo-BNE” aims at developing an outcome indicator for ESD. Within the framework of ESD, system competence is considered to be a key factor, as in view of the high degree of factual complexity of the challenges of the 21st century, for making decisions in terms of sustainable development (Rost, 2005; Rieckmann, 2017).

System competence is defined as the ability to understand areas of complex, socio-ecological reality as systems on a local and global level and, on this basis, to make prognoses and take measures for system use and regulation (Mehren, Rempfler, Ulrich-Riedhammer, Buchholz & Hartig, 2016). Socio-ecological areas of reality which are central within the education for sustainable development (ESD) as reflected in numerous publications are (1) climate and climate change, (2) natural resources (especially water and soil), (3) urbanisation and settlement development, (4) production, consumption and (alternative) energy(ies), (5) poverty and justice and (6) migration (see for example Rieckmann, 2017; European Commission, 2015). Furthermore, different aspects of multi-perspectivity play a central role for system competence in the field of ESD: (a) Considering social, ecological, economic and political or cultural dimensions, (b) systematically switching between local and global approaches and (c) taking into account the temporal perspective of intergenerational justice.

Over the course of the project so far, we have created a first version of a system competence test and refined it by using cognitive laboratories and feedback from experts.  For a psychometric pilot test of the instrument, we collected data from N = 366 15 year-olds to answer the following research questions:

(1)               Does the test reliably measure system competence?

(2)               Do the difficulties of the test items cover the range of competence of 15 year old students?

(3)               Is it possible to measure system competence with one dimension?

Test development: The first step was the item development. Firstly, question stems that focus on crucial themes of ESD were developed. The specific content of the tasks deals with (1) electronic waste, (2) climate change and coral dying, (3) megacities, (4) meat consumption and the rainforest, (5) textile production and (6) overtourism. Secondly, we developed items targeting different levels of socio-ecological system understanding within each of the testlets. Expert judgements were included for the question stems as well as for the items in order to validate both content-related and conceptual requirements. The developed items were then tested in a cognitive laboratory (Prüfer & Rexroth, 2000). In a cognitive walk-through, the task developers first specified the processes and steps that are likely to be necessary for a task solution. The students were then encouraged to think aloud while working on the items. This provided insights into the understanding of the items, solution strategies and particular difficulties which helped us to adapt the items accordingly. We have also had our test items assessed by experts in the field of ESD and system competence as well as by teachers and have incorporated their feedback into our revisions. Pilot Study: Sample. The sample of the pilot study included 366 students from three schools of the type Gesamtschule (comprehensive school) one school of the type Realschule (secondary-level school), one school of the type Hauptschule (secondary-level school) and two schools of the type Gymnasium (grammar school) in Germany. The students were 8th and 9th graders. Procedure. The test was administered in the classrooms, following a standardized instruction on how to complete the test booklets and an explanation of concept maps by external test administrators who visited school during a 90-minute period. Instruments. The instrument was structured as follows: Firstly, the students were asked about demographic data such as age, gender, language at home and number of books at home. In the following, the test items targeting system competence were presented. We use a test booklet design in which each student completes only 3 out of the total 6 test items. Analyses. We tested the conformity of the data to the one-dimensional Rasch model (see e.g., Bond & Fox, 2007), determined the psychometric properties of the items for the subsequent item selection, the reliability of the competence estimates and the match of the distributions of item difficulties and the students’ competences.

The development of our system competence test was successful in terms of yielding a high reliability, a good match of the distributions of item difficulties and the levels of students’ competence. Furthermore, only a small proportion of items had to be sorted out. First analyses of dimensionality suggest that the test can measure system competence with only one dimension. Hence, our newly developed test is a compact, easy to interpret and yet reliable instrument to measure system competence in the field of ESD.

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Erlbaum. Mehren, R., Rempfler, A., Buchholz, J., Hartig, J., & Ulrich‐Riedhammer, E. M. (2018). System competence modelling: Theoretical foundation and empirical validation of a model involving natural, social and human‐environment systems. Journal of Research in Science Teaching, 55(5), 685-711.Prüfer, P. & Rexroth, M. (2000). Zwei-Phasen-Pretesting. ZUMA-Arbeitsbericht 2000/08. Mannheim. Rieckmann, M. (2017). Education for Sustainable Development Goals: Learning Objectives. UNESCO Publishing. Rost, J. (2005). Messung von Kompetenzen Globalen Lernens [Measurement of Competences in Global Learning]. Zeitschrift für internationale Bildungsforschung und Entwicklungspädagogik, 28(2), 19-25. UNESCO (2017). Education for Sustainable Development Goals: Learning Objectives. UNESCO.