Abstract

Home possessions index, the key component of the socioeconomic status measure in the Programme for International Student Assessment (PISA), has been on the researcher’s radar for the reasons of its cross-cultural and cross-cycle comparability since the last decade (Rutkowski & Rutkowski, 2013; Pokropek, Borgonovi & McCormick, 2017; Rolfe, 2021). Although PISA continuously changes items and scaling methodology of the home possessions (HOMEPOS) index, its reliability did not improve and even dropped for some of the countries including the Nordic (see, e.g., OECD, 2012, 2017). Within the Nordic context, one explanation for this could be new migration waves since 2000, causing further change in demographic contours of the countries. With that, possession of some items may cease to be the indicator of family wealth for a certain subgroup of individuals. In PISA, the question of comparability of HOMEPOS items across native and migrant student sub-populations within countries has not yet been addressed. This gap in knowledge may represent a challenge to draw valid inferences on the educational inequalities of student sub-populations, as analysis must be sensitive to the impacts of changing ethnicity landscapes.

To close this gap, we used data from PISA 2018 to test globally and locally for the measurement invariance of the HOMEPOS scale within the item response theory (IRT) paradigm. The within-country comparability analysis was performed across three 15-year-old student sub-populations of Norway and Sweden: with native, first-generation and second-generation immigrant backgrounds. The obtained results will be used to test how items with differential item functioning (DIF) or non-invariant parameters affect the HOMEPOS scale validity as a predictor of academic achievement for ethnic minorities.

Preliminary results show that the validity and reliability of the HOMEPOS scale are threatened. Several items in Sweden and Norway were found to have nonuniform DIF or metric non-invariance. This means that the relationship between HOMEPOS and student learning outcomes is not comparable between student ethnic majority and minorities. The findings call for caution for researchers who investigate this relationship and use the scale to advise on the policy-making processes targeted at students with non-native background. 

Introduction 

The contextual indicator of family wealth in PISA (Vyas & Kumaranayake, 2006), HOMEPOS, is by far the most extensive composite scale across international large-scale assessments (ILSAs) that is used to derive inferences on educational (in)equity for different subgroups of individuals. Furthermore, educational researchers in the Nordic countries have displayed an increasing interest in studying equity of learning outcomes for children with non-native ethnic background. On the other side, there has been critique of adopting identical SES constructs, including home possessions items, from developed countries to measure SES in developing countries (Kim, Cho & Kim, 2019). And with the high immigration of students from developing countries, their financial capital may not be captured by the items that are common indicators of family income for the native students. Hence, HOMEPOS index may not be comparable across native and immigrant students and its validity as a strong predictor of academic achievement for non-native students may be threatened.

To the best of our knowledge, there has been no study investigating the measurement invariance or comparability, of HOMEPOS across native and non-native student sub-populations within the Nordic context. Hence, our study aims to address this gap and provide specific recommendations on how to use the scale for drawing inferences on minority-majority differences in HOMEPOS and its relations to other constructs. 

Data and sample To address our research aim, we included 15-year-old students from Sweden (N=5336) and Norway (N=5575) who participated in PISA 2018. Our focus was on the Nordic countries, where only Norway and Sweden have sufficient sample sizes of students in both second- and first-generation immigrant sub-populations. Moreover, both the countries experienced the immigration wave in the latest decade. Hence, these countries are suitable to investigate whether we can derive common inferences for HOMEPOS scale for the Nordic region. However, if common features are scarce, this may posit even further concerns for the valid use of the scale not only within the Nordic but within the broader European context. For the purpose of analysis, we used 22 categorical items of HOMEPOS index that are common across countries-participants. We used the PISA variable “IMMIG” to define three subgroups of students, namely, Native, Second-generation and First-generation immigrants. Method of Analysis The extent to which an IRT model measuring home possessions of 15-year-old students in PISA 2018 exhibited measurement invariance or non-differential item functioning across three subgroups of students was examined using Mplus v.8.4 (Muthén & Muthén, 1998–2017). We investigated the comparability of HOMEPOS scale with two methods: multi-group IRT modeling for studying global measurement invariance and multiple indicator multiple cause (MIMIC) modeling for testing locally for uniform and nonuniform DIF. Uniform DIF is analogous to scalar non-invariance. It happens when the item is more easily endorsed for one group than another, in other words, when one group requires more of the latent trait to have a 50-50 chance at the higher response category. Hence, in case of uniform DIF, the means of a construct are not comparable across groups. Nonuniform DIF is analogous to metric non-invariance. This means that the item is systematically more related to the trait at a specific item difficulty location for one group. When the item has metric non-invariance, we cannot compare neither the relationship between this item and, e.g., achievement across groups, nor the means. In case of multiple nonuniform DIF-items, the construct cannot be used for valid inferences across the groups under question. Until recently, MIMIC models were used to test only for uniform DIF (Woods, 2008; 2009). However, this practice has been challenged by conceptualizing MIMIC models as mediation and moderated mediation models to test for both uniform and nonuniform DIF (Montoya & Jeon, 2020). In this study, we apply this novel approach.

Our preliminary findings show that there are multiple HOMEPOS items exhibiting uniform DIF or scalar non-invariance in both Sweden and Norway. This was observed across all three student sub-populations, or between native and second-generation or native and first-generation immigrant students. Scalar non-invariance means that the means of HOMEPOS construct cannot be compared across these student subgroups. Scalar non-invariance does not posit a threat to the validity of inferences as long as the means of the construct are not compared. However, in addition to that we found several items with nonuniform DIF or metric non-invariance in Norway and Sweden. This means that researchers cannot use HOMEPOS index to compare its association with, e.g., learning outcomes between native and immigrant student sub-populations. If we don't have comparable measures of SES between ethnic majority and minority students, educational policy would be incapable of taking measures to ensure equity and equality between these groups, or to help students with immigrant background. This is especially important now, as the pandemic most probably has increased the gap between native and non-native student sub-populations. To conclude, investigating DIF and measurement invariance is a key step to establish the valid use of HOMEPOS scale and to understanding how it works. This study found multiple HOMEPOS items that showed differential item functioning, many of them across all three student sub-populations. This finding has direct implications for the validity of inferences drawn on educational inequalities at the national and school level. The study may be of interest to educational researchers not only within the Nordic context, but within the broader European context which demographic contours are everchanging. Our further planned analysis will shed the light on how to use the HOMEPOS index for drawing inferences on its relations to academic achievement for student ethnic majority and minorities.

Kim, S. W., Cho, H., & Kim, L. Y. (2019). Socioeconomic status and academic outcomes in developing countries: A meta-analysis. Review of Educational Research, 89(6), 875–916. Montoya, A. K., & Jeon, M. (2020). MIMIC Models for Uniform and Nonuniform DIF as Moderated Mediation Models. Applied Psychological Measurement, 44(2), 118-136. Muthén, L.K. and Muthén, B.O. (1998-2017). Mplus User’s Guide. Eighth Edition. Los Angeles, CA: Muthén & Muthén. Organisation for Economic Co-operation and Development (2012). PISA 2009 Technical Report. Paris, France: OECD Publishing. Organisation for Economic Co-operation and Development (2017). PISA 2015 Technical Report. Paris, France: OECD Publishing. Pokropek, A., Borgonovi, F., & McCormick, C. (2017). On the cross-country comparability of indicators of socioeconomic resources in PISA. Applied Measurement in Education, 30(4), 243-258. Rolfe, V. (2021). Tailoring a measurement model of socioeconomic status: Applying the alignment optimization method to 15 years of PISA. International Journal of Educational Research, 106, 101723. Rutkowski D. & Rutkowski L. (2013). Measuring socioeconomic background in PISA: One size might not fit all. Research in Comparative and International Education, 8(3), 259–278. Vyas, S., & Kumaranayake, L. (2006). Constructing socio-economic status indices: how to use principal components analysis. Health policy and planning, 21(6), 459-468. Woods, C. M. (2009). Evaluation of MIMIC-model methods for DIF testing with comparison to two-group analysis. Multivariate Behavioral Research, 44, 1-27. Woods, C. M., Oltmanns, T. F., & Turkheimer, E. (2008). Illustration of MIMIC-model DIF testing with the schedule for nonadaptive and adaptive personality. Journal of psychopathology and behavioral assessment, 31(4), 320.