This three-year study aims to design and validate the Kazakhstani Teacher Observational Protocol (KTOP) aimed at gauging the degree to which teachers/ instructors in Kazakhstan successfully implement reform-based practices in high school and undergraduate STEM lessons.

The 2011-2020 Educational Strategy for Kazakhstan outlined the goal of further development of the ‘training system and professional development’ with the primary objective being to improve teachers' learning and professional mastery in the Kazakhstani school system. Education reform has been considered necessary as the secondary curriculum was perceived as overly dense, with a focus on rote learning that resulted in only providing superficial knowledge rather than deep mastery of topics (Fimyar, Yakavets, & Bridges, 2014).

 Education reform across contexts has focused on teachers' critical role in improving education quality (Schleicher, 2016). The question remains as to how initiated changes are validated or measured. Teacher observation instruments have been developed to measure effective teaching (Mantzicopoulos, et al., 2018). In addition to being used to provide standardized data to monitor education, these instruments can help to determine if professional development programs or the use of reformed curricula are producing changes in teaching practices (MacIsaac et al., 2001). Classroom observation instruments can be constructive tools for aiding in the evaluation of teachers and designing professional development (Evenhouse et al., 2018). Classroom observation allows observers to gather information on student and teacher behaviours and the classroom environment within an authentic setting, which is one way that links theory and practice in order to better understand the classroom environment (Snyder, 2012). Observations have been used to gather data on teachers’ integration of technology (Helmer et al., 2018); student-teacher interactions (Darling-Hammond, 2006); and explicit subject-area learning (Waxman et al., 2009). In Kazakhstan, the fast pace and wide range of reforms raises questions about teachers’ readiness to implement reforms in the classroom. Moreover, historically “open lessons” where teachers and administrators observe lessons may be rehearsed without systematic criteria for evaluation.  

 To address these issues, this paper reports on the pilot study where researchers collected data to determine how well the KTOP is suited to the reform-based practice in Kazakhstan. The reform-based practice was conceptualized by way of the following six subscales: lesson design and implementation, methods/teaching strategies, communicative interactions, student-teacher relations, assessment interactions, and integration of content and language in teaching.

     .     The specific questions addressed:

RQ1: What is the overall level of inter-rater reliability of the Kazakhstan Teaching Observational Protocol (KTOP) instrument?

RQ2: What is the level of inter-rater reliability of the KTOP instrument’s (a) lesson design and implementation (5 items), (b) methods/teaching strategies (5 items), (c) classroom culture (communicative interactions; 6 items), (d) classroom culture (student-teacher relationships; 5 items), (e) assessment interactions (3 items), and (f) integrating content and language in teaching (2 items) subscales?

RQ3: How well targeted is the KTOP instrument for identifying the higher- and lower-reform-based teaching practice in the Kazakhstani high school and higher educational contexts?

The Protocol was adapted from the Reformed Observation Teaching Protocol (RTOP) (Sawada et al., 2002). The same 21st-century reform practices on which the RTOP is based are included in many current international educational reform efforts, including those of Kazakhstan (e.g., Yakavets & Dzhadrina, 2014). In the development of this tool, the team considered the best attributes of other tools such as the COPUS (Smith et al., 2013) COPED (Wheeler et al., 2019), and CLIL (deGraaff et al., 2007) with additional sections added to consider the teaching of STEM in English and inclusive teaching practices as these are both current priorities in Kazakhstani educational reforms. Instrumentation A total of 26 items were used in the overall KTOP instrument. The 26 items were comprised of the six separate scales mentioned in RQ2. Each item was anchored by 0 = Never occurred and 4 = Very descriptive. Sampling This study employed convenience sampling with schools that volunteered to participate in trialling the observation instrument in their classes. A total of 25 unique lessons delivered by 13 unique teachers from five distinct schools were of focus for the current study. Lesson observations took place between February and May 2022. On average, the 13 unique teachers who participated in the current study had acquired 10.78 (SD = 6.44) years of teaching experience. The teachers were all drawn from educational institutions in the Astana, Kazakhstan region (three high schools and two universities). The lessons observed included the following eight different subject areas: physics, calculus, math, chemistry, ICT, computer-aided engineering, algebra, biomechanical engineering, and general math. All classes were taught in English. For each lesson, two trained observers were assigned to judge each of the 25 lessons. For this study, of the total 50 judgements, individual observers contributed to a different total number of lessons.

Overall Level of Inter-Rater Reliability (RQ1) The intra-class correlation for the 26-item instrument was estimated at .72 (N = 650) which can be considered acceptable (0.50 < icc ≤ .749; lower 95% CI = .667, upper 95% CI = .755). Level of Inter-Rater Reliability of the KTOP Instrument’s Six Subscales (RQ2) Table 1 Inter-Rater Reliability of the Six KTOP Subscales Abbr. Scale ICC L 95% CI U 95% CI LDAI Lesson design and implementation .689 .558 .782 MTS Methods/teaching strategies .683 .549 .778 CC(CI) Classroom culture (communication interactions) .701 .588 .783 CC(STR) Classroom culture (student-teacher relationships) .577 .398 .703 AI Assessment interactions .730 .574 .829 ICLIT Integrating content and language in teaching .785 .622 .878 Note. Abbr. = scale abbreviation; L/U 95% CI = upper/lower 95% confidence interval for the ICC estimate; acceptable ICC underlined, good ICCs in bold. All ICC subscales were considered moderate, with the exception of the ICLIT, which exhibited a slightly higher level of “good” inter-rater reliability (ICC = .785). KTOP Targeting (RQ3) In terms of targeting, the items did not function optimally. As explained, each item had five levels (0-4). However, the lowest level (zero) was not ‘reached’ by students for 21 of the total 26 items. However, items did discriminate well, with only one item exhibiting negative item-rest point biserial correlation (MTS.8, r = -.040) and only one item, CC(CI).16 was only slightly underfitting to the Rasch model with an outfit value of 1.61 (p < .05). Given the positive results of the pilot study, the tool will be translated into Kazakh and used with a larger sample of teachers/instructors, as the psychometric analysis in the current study is very much underpowered.

Darling-Hammond, L. (2006). Constructing 21st-century teacher education. Journal of Teacher Education, 57(3), 300-314. de Graaff, R., Koopman, G. J., Anikina, Y & Westhoff, G. (2007). An Observation Tool for Effective L2 Pedagogy in Content and Language Integrated Learning (CLIL), International Journal of Bilingual Education and Bilingualism, 10(5), 603-624, DOI: 10.2167/ beb462.0 Evenhouse, D., Zadoks, A., Silva de Freitas, C. C., Patel, N., Kandakatla, R., Stites, N. & DeBoer, J. (2018). Video coding of classroom observations for research and instructional support in an innovative learning environment. Australasian Journal of Engineering Education, 23(2), 95-105. Fimyar, O., Yakavets, N. and Bridges, D. (2014). Educational Reform in Kazakhstan: the contemporary policy agenda. In D. Bridges (Ed.) (2014) Educational reform and internationalisation: the case of school reform in Kazakhstan. Cambridge: Cambridge University Press. MacIsaac, D., Sawada, D., & Falconer, K. (2001, April). Using the Reformed Teaching Observation Protocol (RTOP) as a Catalyst for Self-Reflective Change in Secondary Science Teaching. Paper presented at the American Educational Research Association, Seattle, WA Mantzicopoulos, P., Patrick, H., Strati, A., & Watson, J. S. (2018). Predicting kindergarteners' achievement and motivation from observational measures of teaching effectiveness. The Journal of Experimental Education, 86(2), 214-232. Sawada, D., Piburn, M. D., Judson, E., Turley, J., Falconer, K., Benford, R., & Bloom, I. (2002). Measuring reform practices in science and mathematics classrooms: The reformed teaching observation protocol. School Science and Mathematics, 102(6), 245-253. Schleicher, A. (2016). Teaching excellence through professional learning and policy reform: Lessons from around the world, International Summit on the Teaching Profession. Paris: OECD Publishing. http://dx.doi.org/10.1787/978926452059-en van der Lans, R. M., van de Grift, W. J., & Van Veen, K. (2018). Developing an instrument for teacher feedback: Using the Rasch model to explore teachers' development of effective teaching strategies and behaviors. The Journal of Experimental Education, 86(2), 247-264. Waxman, H. C., Padrón, Y. N., Franco-Fuenmayor, S. E., & Huang, S. L. (2009). Observing classroom instruction for ELLs from student, teacher, and classroom perspectives. TABE Journal, 11(1), 63-95. Wheeler, L. B., Navy, S. L., Maeng, J. L., Whitworth, B. A. (2019). Development and validation of the Classroom Observation Protocol for Engineering Design (COPED). Journal of Research in Science Teaching, 56, 1285-1305. Yakavets, N., & Dzhadrina, M. (2014). Educational reform in Kazakhstan: Entering the world arena. In D. Bridges (ed.), Educational reform and internationalisation: The case of school reform in Kazakhstan (pp. 28-52). Cambridge, UK: Cambridge University Press.