Even though there is many existing research of characteristics of instructional quality (e.g. Praetorius et al., 2018) and of characteristics of school quality (e.g. Holzberger et al., 2020), research in these two fields has developed relatively independently from each other. Consequently, over the last decades researchers have focussed primarily on identifying and testing the followings “paths”: On the classroom level the field of interest is which teaching practices influence student achievement (e.g. Lipowsky et al., 2009);  In the domain of school effectiveness scholarships continued to examine efforts to illuminate effects of school quality also on student achievement (e.g. Gustafsson et al., 2018; Vanlaar et al., 2015); The third “path” of interest is the influence of the higher level (e.g. school characteristics) on the lower level (e.g. classroom characteristics) (e.g. Blömeke et al., 2016; Creemers et al., 2000; Gärtner, 2016). In the field of educational effectiveness research researchers create models depicting the effects of these paths (Reynolds et al., 2014). These studies have in common that they analyse characteristics on one level (i.e., school or classroom) as predictors for a dependent characteristic (e.g. student achievement). The present study seeks at focussing the dynamic nature of relationships among school characteristics, instructional quality and student achievement by analysing interaction effects between school and their teachers/classrooms on student achievement. Heck and Hallinger (2014) published an initial approach.

There are already a multitude of different methodological frameworks and approaches to capture instructional quality (e.g. Lipowsky et al., 2009; Schlesinger et al., 2018) and school quality (e.g. Institut für Qualitätsentwicklung, 2011; OECD, 2005). However, most of the research is observation-based and therefore it has to deal with challenges like limited expertise of personnel and school leaders who serve as observers (Hill & Grossman, 2013). In addition, the quality of observation-based frameworks varies. Praetorius and Charalambous (2018) describe that the 12 analysed observation-based frameworks of instructional quality differ largely in their process of development, their theoretical underpinnings and for this their operationalisations and measurements as well as the existing evidence on reliability and validity.

The American Educational Research Association et al. (2014) state that “it has become common practice to use the same test for multiple purposes” (p. 195). The reuse creates the possibility that the research conducted will suffer disadvantages: First, the instrument does not fully fit the theory-based model and second, one inherits psychometric weaknesses of the existing instruments (e.g., lack of evidence for reliability or validity). Therefore, the need arises to develop instruments to investigate interaction effects between school and their classrooms in mathematics. Such instruments should fulfil the psychometric standards and represent the theory-based model. In the context of education Archer et al. (2014) already state: “Real improvement requires quality measurement” (p. 1). 

The aim of the present study is to develop two questionnaires to measure the attitudes of principals on characteristics of school quality and the attitudes of teachers on characteristics of instructional quality. The author of this paper, together with collaborators (R. Alexandrowicz, K. Krainer and A. Vohns) have conducted two systematic reviews following the PRISMA statement (Moher et al., 2009) to determine which characteristics have an impact on student achievement. The insights from these reviews help to formulate a theory-based model mapping both the expected direct effects of both levels (school and classroom) on student achievement as well as potential interaction effects between the school and classroom level. Based on the reviews, a taxonomy of teaching characteristics and a taxonomy of school characteristics were created to categorize and structure the heterogeneous research fields. These taxonomies form the basis for the development of the two questionnaires. 13 different sources (e.g. Bellens et al., 2019; Cai et al., 2019; Kelcey et al., 2019; Lipowsky et al., 2009; Vanlaar et al., 2015) provided potential items for characteristics of instructional quality. We relied on five sources for the identification of potential items for the questionnaire of school quality (e.g. Institut für Qualitätsentwicklung, 2011; Kyriakides et al., 2015; Vanlaar et al., 2015). After identifying possible items the further development of the questionnaires followed the procedure of Campanelli (2009). In order to test and refine an early version of the questionnaires, it was pretested using “informal methods” (Campanelli, 2009, p. 178). The refined version was tested again in a pilot study with a sample of eleven teachers and three principals. In this stage of development the method of “Think-aloud” (Faulbaum et al., 2009) and descriptive analyses were used. The third pretest was larger in scope. With a sample of 108 teachers and 74 principals, it is possible to apply a polytomous Rasch model on the data. More specifically, the Partial-Credit-Model (PCM) (Masters, 1982) offers several means to assess the model fit and the following techniques were applied: Likelihood-Ratio-Test (LRT), Graphical Model Check, Person-Item-Plot (PI-Plot) and Item fit indices (Smith et al., 2008; Wright & Panchapakesan, 1969). All calculations have been performed with R (R Core Team, 2020), using the eRm-package (Mair et al., 2020) for conducting Rasch analyses.

Developing the two questionnaires lead to a number of important findings. At an early stage, we chose a four-category response format ("Not at all important" - "Rather not important" - "Important" - "Extremely important") in order to avoid the problem of a midpoint. However, the pretests revealed that the category ("Not at all important") was superfluous and therefore the response format was reduced to three categories. Moreover, the present study (development of questionnaires) uses PCM to analyse the quality of the categories. This analysis showed problems for the category “Rather not important”, which was then renamed to “Less important”. Items were also too “easy” from a psychometrically point of view. As a results, the final response categories are "Less important" - "Important" - "Extremely important". A closer analysis of the individual items highlighted weaknesses in item formulation. For example, items containing the word ”appropriate” were problematic, probably due to the fact that people have very different interpretations of when something is appropriate or not. Already in the second pretest, the method of “Think-aloud” and descriptive analyses showed social-desirability bias for some questions. Respondents did not use the categories “Not at all important” or “Rather not important”. To mitigate this mathematically, we introduced questions in the form of “chose the most/least important item” to obtain weighing factors. Furthermore, items that were found to be redundant or inappropriate for this study were removed. This made it possible to achieve an overall length that accommodates the time and mental workload of the respondents, thus ensuring test economy. In conclusion, we may state that "less is more" applies to both the response categories and the items.

Campanelli, P. (2009). Testing Survey Questions. In E. D. de Leeuw, J. J. Hox, & D. A. Dillman (Eds.), EAM book series. International handbook of survey methodology (pp. 176–200). Psychology Press. Creemers, B. P. M., Scheerens, J., & Reynolds, D. (2000). Theory development in SER. In C. Teddlie & D. Reynolds (Eds.), The International Handbook of School Effectiveness Research (pp. 283–298). Taylor & Francis Group. Holzberger, D., Reinhold, S., Lüdtke, O., & Seidel, T. (2020). A meta-analysis on the relationship between school characteristics and student outcomes in science and maths – evidence from large-scale studies. Studies in Science Education, 56(1), 1–34. https://doi.org/10.1080/03057267.2020.1735758 Kyriakides, L., Creemers, B. P. M., Antoniou, P., Demetriou, D., & Charalambous, C. Y. (2015). The impact of school policy and stakeholders’ actions on student learning: A longitudinal study. Learning and Instruction, 36, 113–124. https://doi.org/10.1016/j.learninstruc.2015.01.004 Lipowsky, F., Rakoczy, K., Pauli, C., Drollinger-Vetter, B., Klieme, E., & Reusser, K. (2009). Quality of geometry instruction and its short-term impact on students’ understanding of the Pythagorean Theorem. Learning and Instruction, 19(6), 527–537. https://doi.org/10.1016/j.learninstruc.2008.11.001 Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097 Praetorius, A.‑K., & Charalambous, C. Y. (2018). Classroom observation frameworks for studying instructional quality: looking back and looking forward. ZDM Mathematics Education, 50(3), 535–553. https://doi.org/10.1007/s11858-018-0946-0 Schlesinger, L., Jentsch, A., Kaiser, G., König, J., & Blömeke, S. (2018). Subject-specific characteristics of instructional quality in mathematics education. ZDM Mathematics Education, 50(3), 475–490. https://doi.org/10.1007/s11858-018-0917-5 Smith, A. B., Rush, R., Fallowfield, L. J., Velikova, G., & Sharpe, M. (2008). Rasch fit statistics and sample size considerations for polytomous data. BMC Medical Research Methodology, 8, 33. https://doi.org/10.1186/1471-2288-8-33 Vanlaar, G., Kyriakides, L., Panayiotou, A., Vandecandelaere, M., McMahon, L., Fraine, B. de, & van Damme, J. (2015). Do the teacher and school factors of the dynamic model affect high- and low-achieving student groups to the same extent? a cross-country study. Research Papers in Education, 31(2), 183–211. https://doi.org/10.1080/02671522.2015.1027724 Wright, B., & Panchapakesan, N. (1969). A Procedure for Sample-Free Item Analysis. Educational and Psychological Measurement, 29(1), 23–48. https://doi.org/10.1177/001316446902900102