Background: Young people’s economic, cultural and social status (ECSS) is one of the most prevalent constructs used for studying equity of educational outcomes. National, regional and international large-scale assessments have furthered the quantitative research concerning the relationship between economic, cultural, and social background indicators and educational outcomes (Broer et al., 2019; Lietz et al., 2017; OECD, 2018).

However, there are observed theoretical and analytical limitations in the use of existing ECSS indicators from large-scale assessments for the purpose of monitoring equity in education (Osses et al., Forthcoming). Theoretical limitations relate to inconsistencies in how the ECSS construct is defined and operationalised, which pose significant challenges for comparing results between large-scale assessments and limit the usability of findings in addressing policy issues concerning equity in education. For example, Osses et al. (2022), demonstrated that using alternative approaches for constructing an ECSS indicator leads to different judgements concerning education systems in terms of equity of learning achievement. 

Analytical limitations relate to the validity and reliability of ECSS indicators used in large-scale assessments. Whilst studies often explore reliability, cross-national invariance and other psychometric properties of ECSS indicators, information about the performance of alternative indicators is not provided. In fact no studies were found that compare the performance of alternative ECSS indicators constructed by large-scale assessments; Oakes and Rossi (2003) is an example from health research. 

Objective: This paper focuses on analysing the properties of two ECSS indicators constructed using alternative theoretical and analytical approaches, applied to the same student sample. Evidence on validity is provided to evaluate the relative merits and the comparability of the two indicators for monitoring equity in education.

Method: This study analyses the properties of students’ ECSS indicators constructed by PISA and TIMSS with the aim of providing evidence concerning the validity and comparability of these two indicators. The novelty of the methodological approach lies in estimating both indicators for the same sample of students – those in PISA 2018, and thus analysing the merits of each analytical approach. 

Indicators are analysed in terms of its content – ie, evaluating alignment between the theoretical construct, the indicators and the items chosen for its operationalisation – and its internal consistency. Indicators’ internal structure is investigated using confirmatory factor analysis and item response modelling in relation to model fit and the precision with which the indicators measure the ECSS construct – that is, targeting and reliability. The use of plausible values as a strategy to reduce error in making inferences about the population of interest is also explored.

Preliminary results show that the TIMSS-like indicator constructed using PISA 2018 data may benefit from better defining the underlying construct and of theoretical support to provide evidence for evaluating the adequacy of indicators chosen in its operationalisation. In terms of internal consistency, results indicate that items in the TIMSS-like indicator are “too easy” for the PISA population of interest and, although response data show a reasonably fit to the measurement model, the chosen items provide an imprecise measurement of students’ ECSS. 

Three key conclusions emerge from preliminary results. First, large-scale assessments should devote more time to clearly define and provide theoretical support for the construct of students’ ECSS. Second, items used in summary indicators of ECSS should be carefully inspected, not only in terms of their reliability but also in terms of the adequacy of response categories and fit to measurement model. Third, the use of plausible values should be considered in order to avoid bias and improve precision of population estimates. The PISA indicator is currently being analysed.

This work extends the analysis in Osses et al. (2022) to investigate the properties of two alternative ECSS indicators constructed with the same student sample using PISA 2018 data. The first indicator corresponds to the PISA Economic, Social, and Cultural Status index (hereinafter, PISA_ESCS). The second indicator is constructed by recoding PISA data to obtain variables that are identical to those used in the TIMSS Home Educational Resources scale for grade 8 students (hereinafter, PISA_HER) and following the procedures detailed in the TIMSS 2019 technical report (Martin, von Davier, & Mullis, 2020). Two main aspects of validity (AERA et al., 2014) are evaluated: evidence on indicators’ content and internal structure. Evidence on indicators’ content Evaluating alignment between the construct, indicators and items chosen for its operationalisation allows determining whether scores can be interpreted as a representation of individuals’ ECSS. This is typically referred to as evidence of content relevance and representation (AERA et al., 2014; Cizek, 2020; Messick, 1994). To investigate content relevance and representation, a review of published documentation of PISA and TIMSS was undertaken in relation to theoretical underpinning, conceptualisation and operationalisation of each indicator. Evidence on indicators’ internal structure The modelling approach of each indicator is analysed in relation to the appropriateness of analytical steps followed in its construction. The PISA_ESCS is the arithmetic mean of three components, highest parental education and occupation, and home possessions – the latter being a latent variable indicator constructed using Item Response Modelling – IRM (OECD, 2020). The PISA_HER is the application of an IRM to three items: highest parental education, study support items at home, and number of books at home. Internal structure of indicators is investigated using the analytical tools provided by the modelling approach used in PISA and TIMSS in relation to model fit and the precision with which the indicators measure the ECSS construct – that is, targeting and reliability. Confirmatory factor analysis – with a specification and constraints that matches the indicator construction method used by PISA (OECD, 2020), is used to investigate the internal structure of PISA_ESCS, including model fit and reliability. IRM is used to investigate the internal structure of PISA_HER and of the home possessions scale – a component in the PISA_ESCS. Within IRM analysis, item targeting, model fit, and reliability of estimates are investigated. The use of plausible values, as opposed to weighted likelihood estimates, is also explored (OECD, 2009; Wu, 2005).

Indicators’ content: PISA and TIMSS published documentation provide different levels of depth in the theoretical argument underpinning the ECSS construct. Although indicators used in both summary scales are typically used in operationalisations of ECSS, neither assessment specifies a conceptual model relating theory to construct operationalisation. Indicators’ internal structure: Preliminary results relate to PISA_HER indicator; PISA_ESCS indicator is currently being analysed. Items in the PISA_HER scale are relatively easy for PISA students, with most thresholds located in the lower region of the scale – ie, below the mean latent attribute of 1.63. PISA_HER items fit well together (ie, have similar discrimination) and response data fit the partial credit model (mean squared statistic close to 1). However, the person separation index of the PISA_HER index is low (0.36). Using plausible values in relation to ability estimates is common practice in PISA and TIMSS, where the interest is on reducing error in making inferences about the population of interest. However, contextual information is typically analysed using an IRM approach with the use of point estimates (eg, WLE) to produce students’ scores. Preliminary results indicate that the analytic outcomes might be quite different if plausible values are used. Preliminary results from this study suggest that ECSS indicators in PISA and TIMSS require a sounder definition and operationalisation of the ECSS construct, which should be supported by theory and empirical evidence. The analytical steps in constructing the summary indicator – ie, the measurement model, should reflect the underlying theory. For example, if the construct is theorised to be a latent variable, then the summary indicator should be constructed using a latent variable model. As large-scale assessments aim at making inferences about the population of interest, rather than about individual students, using plausible values is an approach that should be explored in constructing contextual indicators. 

AERA, APA, & NCME. (2014). Standards for Educational and Psychological Testing. AERA. Broer, M., Bai, Y., & Fonseca, F. (2019). Socioeconomic Inequality and Educational Outcomes. Evidence from Twenty Years of TIMSS. SpringerOpen. Cizek, G. J. (2020). Validity: An Integrated Approach to Test Score Meaning and Use. Routledge. Hooper, M., Mullis, I. V. S., Martin, M. O., & Fishbein, B. (2017). TIMSS 2019 Context Questionnaire Framework. In I. V. S. Mullis & M. O. Martin (Eds.), TIMSS 2019 Assessment Frameworks. Boston College, TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2019/frameworks/ Lietz, P., Cresswell, J., Rust, K. F., & Adams, R. J. (2017). Implementation of Large‐Scale Education Assessments. John Wiley and Sons. Martin, M. O., Mullis, I. V. S., Foy, P., & Arora, A. (2012). Methods and Procedures in TIMSS and PIRLS 2011. TIMSS & PIRLS International Study Center, Boston College. https://timssandpirls.bc.edu/methods/index.html Martin, M. O., von Davier, M., & Mullis, I. V. S. (2020). Methods and Procedures: TIMSS 2019 Technical Report. TIMSS & PIRLS International Study Center. Messick, S. (1994). Validity of Psychological Assessment: Validation of Inferences from Persons’ Responses and Performances as Scientific Inquiry into Score Meaning. Educational Testing Service. https://files.eric.ed.gov/fulltext/ED380496.pdf Oakes, M., & Rossi, P. (2003). The measurement of SES in health research: Current practice and steps toward a new approach. Social Science & Medicine, 56(4), 769–784. OECD. (2001). Knowledge and Skills for Life—First results from the OECD Programme for International Student Assessment (PISA) 2000. https://www.oecd-ilibrary.org/education/knowledge-and-skills-for-life_9789264195905-en OECD. (2009). PISA Data Analysis Manual: SAS Second Edition. https://www.oecd.org/pisa/pisaproducts/pisadataanalysismanualspssandsassecondedition.htm OECD. (2017). PISA 2015 Assessment and Analytical Framework: : Science, Reading, Mathematic, Financial Literacy and Collaborative Problem Solving, revised edition. PISA, OECD Publishing. https://doi.org/10.1787/9789264255425-en OECD. (2018). Equity in Education: Breaking down barriers to social mobility. OECD Publishing. OECD. (2019). PISA 2018 Results (Volume II): Where All Students Can Succeed. https://www.oecd.org/pisa/publications/ OECD. (2020). Chapter 16. Scaling procedures and construct validation of context questionnaire data—PISA 2018. https://www.oecd.org/pisa/publications/