The literature on school effects is abundant with school contextual effects on academic achievement of students (see Ma, Ma, & Bradley, 2008). School socioeconomic composition is, perhaps, the most popular school contextual variable. Socioeconomic status (SES) affects students’ academic achievement at different levels of an educational system (e.g., student level, school level, and school district level) (e.g., Ma, Yuan, & Luo, 2016). Student SES is often measured through parents’ education, occupation, and income; and school SES is often measured through the aggregation of SES among students within a school. School SES has been declared to have large, persistent effects on academic achievement of students (e.g., McConney & Perry, 2010; Perry & McConney, 2010; Willms, 2010).

These researchers employed multilevel modeling or hierarchical linear modeling (HLM) to investigate school contextual effects, often with students nested within schools as the data hierarchy. However, one critical variable that is strongly correlated with academic achievement is commonly absent in their models. That is a prior measure of academic achievement of students. The effects of school SES on students’ academic achievement can be seriously misestimated by the absence of this prior measure (see Marks, 2015; Pokropek, 2015; Televantou et al., 2015). These researchers demonstrated that in the absence of students’ prior academic achievement there are statistically significant effects of school SES on academic achievement (at the school level), but in the presence of students’ prior academic achievement the statistically significant effects of school SES on academic achievement (at the school level) tend to disappear. They coined this phenomenon with a general term of fake compositional effects or statistical artifacts or phantom effects.

With a focus on school SES, the present study investigates school contextual effects as a potential source of phantom effects in the school effectiveness research literature. Specifically, the purpose of the present study is to develop an general analytical framework or approach that effectively examines to what extent the effects of school context (e.g., school SES) on schooling outcomes (e.g., academic achievement of students) are phantom effects. Data for the present study come from the 2015 Program for International Student Assessment (PISA) with students nested within schools. The PISA 2015 emphasizes science education. With measures of students’ science achievement and individual background (including student SES from which school SES can be created) as well as school context and school climate, PISA data are appropriate for the purpose of the present study. Specifically, to examine the potential phantom effects of school SES, the strategy is to create a prior measure of science achievement with various degrees of correlation with the measure of science achievement available in the PISA 2015 database. With the multilevel models fitted with and without these prior science achievement measures, the behaviors of school SES can be examined in terms of its (contextual) effects on science achievement of students. This analytical framework can facilitate a discussion on the following research questions.

1. In the absence of any prior science achievement measures, how strong are the effects of school socioeconomic composition (i.e., school SES) on science achievement of students?
2. In the presence of various prior science achievement measures, how strong are the effects of school socioeconomic composition (i.e., school SES) on science achievement of students?
3. What student-level variables (descriptive of student and family characteristics) and school-level variables (descriptive of context and climate characteristics), when presence in the multilevel model, can significantly reduce the likelihood of phantom effects?

The combination of empirical answers to these research questions can provide evidence to address the issue of to what extent the effects of school SES on science achievement of students are phantom effects and what researchers can do about this phenomenon.

Data We employ PISA 2015 data from Europe and North America to develop and validate our analytical framework. The outcome measure is science achievement. Independent variables describes student and family characteristics at the student level and context and climate characteristics at the school level. School (mean) SES is an aggregated measure of SES within each school. Generation of Prior Measures Using R, ten prior measures of science achievement are created with correlation as .05, .15, .25, .35, .45, .55, .65, .75, .85, and .95 with the PISA measure. Each prior measure is normally distributed and share the same measurement scale as the PISA measure. Model without Prior Measures A multilevel model is developed with students nested within schools. At the student level, students’ science achievement scores are the dependent variable, with no independent variables. Y_ij= β_0j+ϵ_ij where Y_ij is the score of science achievement for student i in school j, and ϵ_ij is the error term. The coefficient, β_0j, represents the average score of science achievement for school j. At the school level, β_0j is modeled by school SES β_0j= γ_00+γ_01 〖SchSES〗_j+ u_0j where γ_00 is the (adjusted) grand mean measure of science achievement, γ_01 represents the effects of school SES on the measure of science achievement, and u_0j is an error term. The results provide answers to the first research question. Model with Prior Measures A more complex multilevel model is developed with the addition of the prior measures of science achievement at the student level. Each prior measure is added separately. There is no change to the school-level model. Y_ij= β_0j+β_1j 〖PM〗_ij+ϵ_ij β_0j= γ_00+γ_01 〖SchSES〗_j+ u_0j With ten prior measures of science achievement at different correlations with the PISA measure of science achievement, the behavior of school SES in relation to the PISA measure of science achievement can be observed. The results provide answers to the second research question. Model with Student and School Characteristics Finally, student and family characteristics are introduced to the student level. Y_ij= β_0j+β_1j 〖PM〗_ij+∑▒〖β_pj X_pij 〗+ ϵ_ij where X_pij are student characteristics. The coefficients, β_pj, represent the effects of student characteristics on the PISA measure of science achievement in school j. At the school level, β_0j is modeled by school contextual and climate characteristics β_0j= γ_00+γ_01 〖SchSES〗_j+∑▒〖γ_0q Z_qj 〗+ u_0j where Z_qj are school characteristics. The coefficients, γ_0q, represent the effects of school characteristics on the PISA measure of science achievement. The results provide answers to the third research question.

The initial results indicate a potential correlation coefficient between the prior measures of science achievement and the PISA 2015 measure of science achievement as the “cut-off” point or critical point, above which phantom effects associated with school SES are likely to occur. This cut-off point depends on the presence of independent variables (as control variables) at both student and school levels. This result implies that with proper inclusion of important student-level variables and school-level variables, the likelihood of the phantom effects associated with school SES can be reasonably reduced. In other words, the cut-off point can be reasonably raised to prevent the occurrence of the phantom effects associated with school SES (i.e., making the phantom effects associated with school SES harder to occur). Therefore, the initial examination of our analytical framework suggests the effectiveness of this analytical framework. Final results will be ready to be presented in the conference.

Ma, X., Ma, L., Bradley, K. (2008). Using multilevel modeling to investigate school effects. In A. A. O’Connell & D. B. McCoach (Eds.), Multilevel modeling of educational data (pp. 59-110). Charlotte, NC: Information Age. Ma, X., Yuan, J., & Luo, X. (2016). Achievement related within-school socioeconomic gaps in science subjects in China: Evidence on existence, consistency, and compensation. School Effectiveness and School Improvement, 27, 511-533. Marks, G. N. (2015). Are school-SES effects statistical artefacts? Evidence from longitudinal population data. Oxford Review of Education, 41, 122-144. McConney, A., & Perry, L. B. (2010). Socioeconomic status, self-efficacy, and mathematics achievement in Australia: A secondary analysis. Educational Research for Policy and Practice, 9, 77-91. Perry, L. B., & McConney, A. (2010). Does the SES of the school matter? An examination of socioeconomic status and student achievement using PISA 2003. Teachers College Record, 112, 1137-1162. Pokropek, A. (2015). Phantom effects in multilevel compositional analysis: Problems and solutions. Sociological Methods & Research, 44, 677-705. Televantou, I., Marsh, H. W., Kyriakides, L., Nagengast, B., Fletcher, J., & Malmberg, L. E. (2015). Phantom effects in school composition research: Consequences of failure to control biases due to measurement error in traditional multilevel models. School Effectiveness and School Improvement, 26, 75-101. Willms, J. D. (2010). School composition and contextual effects on student outcomes. Teachers College Record, 112, 1008-1037.