Diagnostic assessments are substantial to teachers’ professional lives. They diagnostically assess to choose tasks for the class or for individuals, to adapt their teaching during and in between lessons, and to grade their pupils by the end of term (Herppich et al., 2018; Urhahne & Wijnia, 2021). However, teachers assess a variety of pupils’ characteristics at differing levels of diagnostic accuracy – e.g., higher levels in academic achievement and lower levels in social skills, self-concepts, and motivation (Urhahne & Wijnia, 2021). Thus, assessments are diagnostically distorted, and accuracy development is required to make suitable and effective educational decisions that benefit pupils’ learning success (Herppich et al., 2018).

In this framework, diagnostic accuracy is defined in lens model terms as the so-called ‘rank-structure-component’ – the correlation of teachers’ assessment with pupils’ actual competence level (Brunswik, 1958; Schrader, 2013 ger. ‘Rangordnungskomponente’). Low accuracy signifies that teachers’ diagnostic assessments and pupils’ actual competence level do not correlate to a high degree. These distortions are due to an inaccurate and inconsistent use of valid and non-valid information, the so-called cues (Karelaia & Hogarth, 2008; Urhahne & Wijnia, 2021). E.g., a teacher could use the non-valid cue ‘migration background’ to assess a pupil’s argumentative text competence. Thus, it appears particularly relevant to promote teachers’ diagnostic accuracy by training their consistent use of valid cues to assess students’ characteristics (Loibl et al., 2020; Nestler & Back, 2013; Südkamp et al., 2012). Therefore, in the present study, we attempted to influence teachers’ diagnostic behaviour regarding their cue utilisation to increase their diagnostic accuracy. To do so, we based our research upon the theoretical framework of Loibl and colleagues (2020, p. 3). 

In their framework for ‘Explaining Teachers’ Diagnostic Judgments by Cognitive Modeling’ (DiaCoM), they explicate that a teacher’s person characteristics influence the use and interpretation of situation characteristics. This subjective filter informs diagnostic thinking and behaviour. Precisely, in a diagnostic situation, teachers selectively perceive the cues they are presented with. What cues teachers perceive and how they are interpreted depends on their own mindset and affect, their previous knowledge, e.g., on a student’s performance, and their beliefs. 

To regulate teachers’ subjective filter when assessing, researchers have addressed the moderating function of metacognitive awareness, with its two components knowledge and regulation, on diagnostic accuracy (Xu & Brown, 2016). Nevertheless, there are few insights into its potential to induce behavioural change to develop teachers’ diagnostic accuracy: Thiede and colleagues (2015, 2018) delivered longitudinal interventions of professional development to teachers. Their interventions focused on metacognitive knowledge, as the acquisition of knowledge on professional development, but they also entailed elements of metacognitive regulation, as model application. Those participants who highly implemented metacognitive knowledge and regulation into their lessons increasingly used more diagnostically relevant information and, thus, achieved higher diagnostic accuracy. Similarly, Beck and colleagues (2021) emphasise the application of cognitive strategies, i.e., metacognitive regulation, during assessments. Further, they hypothesise that interventions of feedback on cue utilisation, i.e., metacognitive knowledge, are relevant to increasing teachers’ diagnostic accuracy.

To explore the role of metacognitive awareness in accuracy development, we used computer-based assessment tests and examined how metacognitive knowledge as individualised feedback and metacognitive regulation as strategy training impact diagnostic accuracy. Specifically, we hypothesised that …

-       feedback on diagnostic accuracy and cue utilisation after an assessment task increases diagnostic accuracy [H1].

-       groups that receive feedback and strategy training (EG3) change more in diagnostic accuracy across time than those who receive feedback only (EG1) [H2].

-       the higher pre-service teachers’ diagnostic accuracy in the pre-test, the lower their increase to the post-test [H3].

A randomised laboratory study with two measurements of four groups was conducted (CG, EG1-EG3): EG1 received individualised feedback, EG2 was administered strategy training to reflectively evaluate diagnostic behaviour, EG3 received feedback and strategy training, and one control condition (CG). Diagnostic accuracy was the dependent variable, operationalised as the correlation of respondents’ assessments with pupils’ actual competence level (Brunswik, 1958; Karelaia & Hogarth, 2008). The higher the correlation, the higher the diagnostic accuracy. Pupils’ actual competence level was rated by Müller and Busse (2023) using the software Comproved. Participants were pre-service teachers in their bachelor’s and master’s degree. In total, the study will have a minimum sample of 200 participants. Students spent 90 minutes engaging in the experiment: about 30 minutes for pre- and post-test each, about 10 minutes for feedback, and about 20 minutes for strategy training. The post-test was followed by a manipulation check. CG, EG1, and EG2 received lacking materials after the manipulation check to equalise chances of accuracy development and the duration of the experiment. Participants assessed the argumentative text competence of 40 pupils in random order on a grade scale from zero to 15 points, as is common in the German educational system (15 = A+, 14 = A, 13 = A-, 12 = B+, etc.). All text data stemmed from the project ‘Education of Pre-Service Teachers to Promote Text Competence via Feedback’, financially supported by the Mercator-Institute (WeLiKe Feedback , project number: 1111109). With the feedback, we aimed to induce dissonance, operationalised as a distortion of at least one non-valid cue by 0.1 or more. Feedback contained participants’ diagnostic accuracy (low, rather low, average, rather high, high) and information on how the presented cues structure, coherence (valid), migration background, and sex (non-valid) were used to assess. Strategy training was based around Strike and Posner’s (1992) four conditions of conceptual change – dissatisfaction, intelligibility, plausibility, and productivity – and, thus, consisted of four reflective tasks, which respondents answered in free-form text. We used Mayer and colleagues’ (2009, pp. 33–35) task-operators to define the learning objectives: Respondents describe their diagnostic process, they explain how they can change their diagnostic behaviour to assess more justly, they plan what they will pay attention to in the following assessments, and they develop strategies for more accurate assessments. We calculated hierarchical linear models (HLM) with pilot data to analyse the impact of feedback and strategy training on pre-service teachers’ diagnostic accuracy.

Results of this study indicate that feedback alone may not suffice to impact diagnostic accuracy in EG1, nor does strategy training in EG2. Solely, the intervention that combines both, the feedback and the strategy training condition, seems to result in an effect that is statistically significant. The cue ‘migration background’ is used more accurately from pre- to post-test. To gain further insights and draw conclusions on the relevance of feedback for accuracy-related learning, we will continue data collection from April to July 2025. Further analyses will continue to explore metacognitive awareness, as the combination of feedback and strategy training, as a moderator of diagnostic accuracy (Xu & Brown, 2016).

Beck, J., Dutke, S., & Utesch, T. (2021). Using Brunswik’s Lens Model to Identify and Reduce Teacher Biases in the Informal Assessment of Student Motivation. The Brunswik Society Newsletter, 36, 14–17. Brunswik, E. (1958). Perception and the Representative Design of Psychological Experiments (Vol. 9). University of California Press. Herppich, S., Praetorius, A.-K., Förster, N., Glogger-Frey, I., Karst, K., Leutner, D., Behrmann, L., Böhmer, M., Ufer, S., Klug, J., Hetmanek, A., Ohle, A., Böhmer, I., Karing, C., Kaiser, J., & Südkamp, A. (2018). Teachers’ assessment competence: Integrating knowledge-, process-, and product-oriented approaches into a competence-oriented conceptual model. Teaching and Teacher Education, 76, 181–193. Karelaia, N., & Hogarth, R. M. (2008). Determinants of Linear Judgment: A Meta-Analysis of Lens Model Studies. Psychological Bulletin, 134(3), 404–426. Loibl, K., Leuders, T., & Dörfler, T. (2020). A Framework for Explaining Teachers’ Diagnostic Judgements by Cognitive Modeling (DiaCoM). Teaching and Teacher Education, 91, 103059. Mayer, H. O., Hertnagel, J., & Weber, H. (2009). Lernzielüberprüfung im eLearning. Oldenbourg. Müller, N., & Busse, V. (2023). Herausforderungen beim Verfassen von Texten in der Sekundarstufe. Zeitschrift für Erziehungswissenschaft, 26(4), 921–947. Nestler, S., & Back, M. D. (2013). Applications and Extensions of the Lens Model to Understand Interpersonal Judgments at Zero Acquaintance. Current Directions in Psychological Science, 22(5), 374–379. Schrader, F.-W. (2013). Diagnostische Kompetenz von Lehrpersonen. BzL - Beiträge zur Lehrerinnen- und Lehrerbildung, 31(2), 154–165. Strike, K. A., & Posner, G. J. (1992). A Revisionist Theory of Conceptual Change. (pp. 147–176). SUNY Press. Südkamp, A., Kaiser, J., & Möller, J. (2012). Accuracy of teachers’ judgments of students’ academic achievement: A meta-analysis. Journal of Educational Psychology, 104(3), 743–762. Thiede, K. W., Brendefur, J. L., Carney, M. B., Champion, J., Turner, L., Stewart, R., & Osguthorpe, R. D. (2018). Improving the accuracy of teachers’ judgments of student learning. Teaching and Teacher Education, 76, 106–115. Thiede, K. W., Brendefur, J. L., Osguthorpe, R. D., Carney, M. B., Bremner, A., Strother, S., Oswalt, S., Snow, J. L., Sutton, J., & Jesse, D. (2015). Can teachers accurately predict student performance? Teaching and Teacher Education, 49, 36–44. Urhahne, D., & Wijnia, L. (2021). A review on the accuracy of teacher judgments. Educational Research Review, 32, 100374. Xu, Y., & Brown, G. T. L. (2016). Teacher assessment literacy in practice: A reconceptualization. Teaching and Teacher Education, 58, 149–162.