This workshop discusses the methodological framework of recent empirical studies (Marsh et al., 2009; Televantou et al., 2015) that have addressed the impact of correcting for measurement error in student-level measures (i.e., student achievement) on compositional effects’ estimates. A compositional effect is revealed when students’ outcomes are associated with the aggregated characteristics of their peers in the school or the classroom, after controlling for pre-existing differences at the student level. Research findings often support what is taken to be the conventional wisdom, suggesting a positive, but weak effect of class- or school-aggregated achievement on students’ academic outcomes. Thus, for example, they suggest a positive association between the peers’ average achievement and a student’s academic achievement. Still, there is remarkably little agreement on this matter.
The present workshop begins by explaining how failing to account for measurement error at level 1 distorts derived estimates of school/class compositional effects. On the basis of illustrative analysis using large-scale data from education, it demonstrates how the use of doubly latent models can help overcome this problem. The demonstration shows in an empirical way how measurement error bias systematically leads to a positive bias in the school-/class. composition effect estimates. This suggests that non-existent, or even negative, school composition effects may misleadingly be estimated as positive and statistically significant—an artifact of the inadequacy in the underlying statistical methodology.

Further, the workshop considers academic self-concept toward mathematics as an educational outcome (Dicke et al., 2018; Televantou et al., 2021; 2023), showing how doubly latent models can be used to investigate the so-called big-fish-little-pond effect (BFLPE). The BFLPE suggests that average achievement at the classroom level or the school negatively predicts academic self-concept, despite the positive effect of achievement on self-concept at the individual level, and it is a robust finding concerning controls for measurement in compositional analysis (Dicke et al., 2018; Televantou et al., 2021).
The workshop concludes with a discussion of related theoretical, substantive, and methodological issues and with some guidelines for future research.
The methodology proposed by Marsh et al. (2009) can be relevant for any researcher concerned about how individuals may be affected by their interaction with other individuals within similar settings and whenever the variables involved in the analysis are subject to unreliability (e.g., in econometrics and health, in organizational psychology, social psychology, etc.). Specifically, in relation to school (and teacher) effectiveness studies, these models can be applied whenever the substantive interest lies in the investigation of the impact that the characteristics of the fellow students in a school (or a class) have on an individual’s outcomes.

The conventional approach to the investigation of compositional effects is multilevel analysis. Multilevel modeling effectively considers the hierarchical structure of educational data (e.g., students nested within schools, with Level 1 representing individual-level variables nested within Level 2 or group-level variables). The methodological framework typically used until recently to control for unreliability due to measurement error was one of single-level confirmatory factor analysis and Structural Equation Modeling (SEM). SEM research is concerned with issues related to the factor structure: how multiple indicators are related to the latent variables (factors) they are intended to represent, the assessment of measurement error, and the investigation of relationships among the latent variables after controlling for measurement error (Marsh et al., 2009).The problem with using these models in educational settings is that, conventionally, they fail to take potential clustering in the data into account. These two dominant approaches in educational research, multilevel modeling and structural equation modeling, have been integrated into a single framework. Using the Big-Fish-Little-Pond-Effect hypothesis as their substantive basis—a classic compositional effect widely investigated in the field of educational psychology—Marsh et al. (2009) demonstrated a 2x2 taxonomy of multilevel structural equation models. Marsh, et al., used the term “manifest” in relation to measurement error or sampling error when no adjustments are made for the corresponding source of error and “latent” when measurement or sampling error is adjusted for.In this way, the doubly manifest model is the conventional multilevel model that makes no adjustments for measurement or sampling error, while the doubly latent model accommodates both measurement error at level 1 and level 2 as well as sampling error in the higher-level aggregates. The models control for measurement error using multiple indicators and for sampling error, assuming latent rather than manifest aggregation.

The proposed workshop seeks to familiarize the attendees with the literature on the mixed findings regarding the magnitude and direction of school/class compositional effects on students’ individual outcomes. It aspires to spur discussion on the validity of empirical results from past and current research that evaluates compositional effects based on sub-optimal models failing to control for measurement error. Meanwhile, it demonstrates the robustness of the BFLPE to different modeling specifications and datasets used. Importantly, this workshop aims to equip educational researchers with the methodological knowledge that allows them to quantify the amount of bias in the compositional effect estimates that could be attributable to a failure to control for measurement error. Hence, by the end of this session, attendees will have achieved the following outcomes: I. Gain a comprehensive understanding of the significance of correcting for measurement error when testing compositional effects in educational contexts. II. Have been presented with research questions that could potentially be answered using large-scale educational survey data and doubly latent models. III. Understand how to perform relevant statistical analyses in the Mplus statistical package. IV. Be equipped with information on further resources for continued learning on the topics presented.

Dicke, T., Marsh, H. W., Parker, P. D., Pekrun, R., Guo, J., & Televantou, I. (2018). Effects of school-average achievement on individual self-concept and achievement: Unmasking phantom effects masquerading as true compositional effects. Journal of Educational Psychology, 110(8), 1112–1126. https://doi.org/10.1037/edu0000259 Marsh, H.W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A.J.S., Abduljabbar, A.S. and Köller, O. (2012) Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47 (2), pp. 106-124. 10.1080/00461520.2012.670488 Marsh, H.W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B. and Nagengast, B. (2009) Doubly-latent models of school contextual effects: Integrating Multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44 (6), pp. 764-802. Televantou, I., Marsh, H. W., Dicke, T., & Nicolaides, C. (2021). Phantom and big-fish-little-pond-effects on academic self-concept and academic achievement: Evidence from English early primary schools. Learning and Instruction, 71, 101-399. Televantou, I., Marsh, H. W., Kyriakides, L., Nagengast, B., Fletcher, J. & Malmberg, L-E. (2015). Phantom effects in school composition research: consequences of failure to control biases due to measurement error in traditional multilevel models. School Effectiveness and School Improvement. 26(1), 75-101. https://doi.org/10.1080/09243453.2013.871302 Televantou, I., Marsh, H. W., Xu, K. M., Guo, J., & Dicke, T. (2023). Peer Spillover and Big-Fish-Little-Pond Effects with SIMS80: Revisiting a Historical Database Through the Lens of a Modern Methodological Perspective. Educational Psychology Review, 35(4), 100.