A fundamental prerequisite for developing environmental literacy for sustainability is understanding systems thinking (Kali et al., 2003). For example, developing the ability to interpret and understand the carbon cycle in terms of a system is necessary to grasp the monumental challenges posed by climate change (Shepardson et al., 2012). Although international school curricula, including countries like Sweden, promote the learning of the carbon cycle, science education research shows that understanding complex earth systems is challenging for pupils as it requires integrating knowledge from different levels of organisation and content areas (Düsing et al., 2019). Obstacles that pupils encounter include perceiving components of the system as separate “entities” rather than connecting them, or struggling to relate the system to everyday life (Assaraf & Orion, 2005). Systems thinking about earth systems requires mastering a range of skills, such as identifying the components of the cycle, through to thinking temporally about predictive implications of a system. Assaraf and Orion (2005) have articulated a framework of systems thinking abilities that consists of three hierarchical levels, namely Analysis (skills for identifying components of a system), Synthesis (skills for relating system components) and Implementation (skills for perceiving hidden system dimensions).

A large body of empirical evidence has confirmed the learning benefit of including pictorial elements in educational materials, and that careful design of multimedia resources that consider human cognitive processes has great influence on learning outcomes (Mayer, 2014). At the same time, Asshoff et al. (2010) claim that visually representing the complexity of natural processes such as the carbon cycle should provide more interactive and dynamic opportunities for learners. Therefore, it is rather surprising that the complexity of the carbon cycle is typically depicted and taught via static and often highly conventionalised diagrams. Little work has investigated how systems thinking can be supported through interactive, adaptive visualizations that also integrate aspects of canonical representations familiar to pupils and teachers.

This study forms part of a larger research program developing and testing an adaptive visual learning environment, termed Tracing Carbon, which supports pupils’ systems thinking skills in the context of the carbon cycle. Tracing Carbon comprises three modules, each integrated with interactive visual tasks  and respective quiz questions aimed at probing abilities related to the three hierarchical levels (1-Analysis, 2-Synthesis, 3-Implementation). The current study purpose was to explore pupils’ interaction and performance with Tracing Carbon, guided by the following research questions. How do pupils:

• Interact with the Tracing Carbon learning environment when performing tasks?
• Perform on the quiz questions in terms of assigned hierarchy and difficulty levels?
• Assess the difficulty of quiz questions in terms of assigned hierarchy and difficulty levels?

A sample of 63 pupils aged 14-15 years from two Grade 8 classes engaged with the interactive visual learning environment about the carbon cycle as a part of a biology class. Tracing Carbon consists of interactive tasks and quizzes organized in three modules structured in chapters. In this study, the pupils had access to the first module (global aspects of the carbon cycle) and the first half of the second module (forest ecosystem), altogether comprising three chapters and three sets of quizzes. In each chapter, pupils first engaged with visual interactive tasks, followed by a quiz. After completing each quiz item, pupils also assessed the perceived difficulty of the item on a scale ranging from 0 (very easy) to 10 (very hard). Log file data automatically captured by the system provided information about the learning process, such as students’ mouse/pointer interaction with a particular graphical feature, or the number of mistakes pupils made while responding to the quizzes. Collectively, all pupils responded to quiz questions representing all three hierarchy levels. Additionally, quiz questions in each hierarchy level were assigned as “easy” or “hard”. One type of visual interactive task in the system prompted pupils to draw arrows between components of the carbon cycle. Each arrow corresponds to a process that transfers carbon atoms between carbon reservoirs, such as when carbon atoms in carbon dioxide molecules are transferred to the biosphere through photosynthesis in plants. In a “simple” task, consisting of four reservoirs, the most common error (made by 39 pupils) was to draw arrows from Fossil fuel reserves to Land. In a “complex” task, consisting of 12 reservoirs, the most common mistake (made by 50 pupils) was to draw an incorrect-connection arrow from Decomposers to Plants. Additionally, we performed GLM repeated measures analyses of variance with number of incorrect answers and difficulty assessment by pupils as dependent variables. We found for both variables significant main effects of assigned difficulty levels (easy vs. hard, F (1, 45) = 17.60; p < .001; η2 = .28 and F (1, 45) = 35.84; p < .001; η2 = .44, respectively). Questions assigned as hard resulted in a higher number of incorrect answers and a higher level of assessed difficulty by pupils. We also observed significant interaction effects for both dependent variables.

Analysis indicates that overall, the quiz items designated by the researchers as “easy” were associated with fewer mistakes and a lower perceived difficulty rating than quiz items designated as “hard”. This supports the validity of the quiz item design and integration in Tracing Carbon. However, quiz items designed to engage the second level (Synthesis) in the applied hierarchical systems thinking framework (Assaraf & Orion, 2005) deviate from this pattern. This calls for a deeper consideration of what makes a synthesis-level quiz item easy or hard. The required cognitive abilities might be expected to be more complex for quiz items designed to test for higher levels of the hierarchical systems thinking framework. Nevertheless, the findings do not indicate a corresponding consistent difference in the number of errors or perceived difficulty between quiz items related to the three levels. This result suggests that measurement of hierarchy level understanding is complex and cannot be simply reflected by number of errors alone. In addition, qualitative analysis could help shed light on what types of errors pupils made in the questions and if there is a link between type of mistakes made in interactive tasks and type of mistakes made in quiz questions. Analysis of interaction data from log files reveals multiple errors related to both drawing erroneous arrows and in the wrong direction. However, the errors were not evenly distributed among the possible errors and could therefore be related to misunderstandings that are commonly found in the literature. For example, the very common incorrect connection made between decomposers and plants could be related to consistently reported erroneous conceptions where many learners believe that trees obtain their energy and building blocks from the soil, rather than from carbon dioxide and solar radiation (e.g. Wennerstam et al., 2020).

Assaraf, O. B. Z., & Orion, N. (2005). Development of systems thinking skills in the context of earth system education. Journal of Research in Science Teaching, 42(5), 518-560. Asshoff, R., Ried, S., & Leuzinger, S. (2010). Towards a better understanding of carbon flux. Journal of biological education, 44(4), 175-179. Düsing, K., Asshoff, R., & Hammann, M. (2019). Students’ conceptions of the carbon cycle: Identifying and interrelating components of the carbon cycle and tracing carbon atoms across the levels of biological organisation. Journal of Biological Education, 53(1), 110-125. Kali, Y., Orion, N., & Elon, B. (2003). The effect of knowledge integration activities on students’ perception of the earth’s crust as a cyclic system. Journal of Research in Science Teaching, 40, 545-565. Mayer, R. E. (2014). The Cambridge handbook of multimedia learning (2nd ed.). Cambridge University Press Shepardson, D. P., Niyogi, D., Roychoudhury, A., & Hirsch, A. (2012). Conceptualizing climate change in the context of a climate system. Environmental Education Research, 18(3), 323-352. Wennersten, L., Wanselin, H., Wikman, S., & Lindahl, M. (2020). Interpreting students’ ideas on the availability of energy and matter in food webs. Journal of Biological Education, 1-21.