Cross-Cultural Differences in Opportunities-to-learn and Achievement in Reading: a Methodological Study

Author(s):

Dominique Lafontaine(presenting / submitting)Svenja Vieluf Baye Ariane Monseur Christian

Conference:

ECER 2014

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 01 A, Findings from Large-scale Assessments: Roles of Curricula and Opportunities to Learn

Paper Session

Time:

2014-09-02

13:15-14:45

Room:

B010 Anfiteatro

Chair:

Tjeerd Plomp

Contribution

Theoretical background

International comparative studies provide rich data to explore the links between instructional quality and educational opportunities and students’ outcomes in a broad variety of education systems. While IEA studies (TIMSS, PIRLS) traditionally encompass analyses of curriculum, instruction and opportunities-to-learn in their investigation, PISA (Programme for International Student Assessment) only recently included such measures in the students’ background questionnaires.

There is strong and long-lasting evidence that opportunities-to-learn are among the best predictors of students’ achievement (Schmidt & Maier, 2009); OTL is one of the most important malleable variables for educators and policy makers (Oakes, Gamoran & Page, 1992). However, most of the comparative studies reporting a strong impact of OTL were about maths and science. No results were available so far about reading OTL from previous large scale assessments in secondary education.

For primary education, Lundberg (1994), analyzed OTL variables for grade 4 in IEA-RLS (1991) and found “a rather small impact on student achievement, when factors related to home and community resources were partialled out” (p. 185). These disappointing results could be explained by the fact that OTL measures were self-reported measures (by teachers). Moreover, higher values of OTL were observed in non-European or south-European countries and lower values in Nordic and West European countries. This might indicate a general response bias (compliance and/or cultural bias such as extreme response style).

Response styles are commonly defined as “consistent and stable tendencies in response behaviour that are not explainable in terms of question content or what a given question aims to measure” (Yang, Harkness, Chin & Villar, 2010). The main response style biases are acquiescence (and disacquiescence), extreme and intermediate response style and social compliance. Response styles may vary according with several characteristics of the respondents, such as gender, age, education. Numerous studies in cross-cultural psychology have tried to link response style with the respondents’ culture, refering to Hofstede’s model, opponing individualistic and collectivist cultures. On that basis, one has often contrasted Western and Asian countries, sometimes leading to overgeneralization. In Europe, many studies have contrasted Northern and Southern countries (Van de Vijver & Poortinga, 1997; Van Herk, Poortinga & Verhallen, 2004).

Aims of the study

In PISA 2009 (reading as a major domain), two OTL scales focusing on classroom reading activities - more precisely on reading material (8 items) and strategies used for teaching reading (9 items) - were included in the students’ booklets.

Using the PISA 2009 data from 43 education systems, the present study aims at exploring to what extent is reading achievement of 15 year-olds explained by variations in reading curriculum (OTL). The study has two distinct facets:

A methodological facet:

Is it possible to measure reading OTL in a cross-cultural context in a valid and reliable way? Are the OTL measured constructs stable or invariant across countries? If not, which patterns can be observed? If response style biases are at play, how can they be addressed through innovate ways of data analyses?

A content facet

To what extent are OTL related to reading achievement? How much of the variation in student and school performance can be explained by OTL in a less-content related domain such as reading literacy (which is not taught as a subject in secondary education)?

The communication will focus on the methodological facet, only providing a broad overview of the correlations between the OTL constructs and reading achievement.

Method

Data from PISA 2009 (OECD, 2010) have been used. Fifty-six education systems participated in this cycle of PISA. However, missing rates were above 10% in 13 countries for the OTL variables. Consequently, data from only 43 education systems have been included for the following analyses. Those countries cover a broad geographical area, including Western and Eastern Europe, Middle East, North of Africa, North and South America and Asia. For each country, representative samples of 15 year-olds have been drawn; the total number of students included in the analyses amounts 426 000 students. Methodological facet Several methods were successively implemented in order to test the structure and invariance of the data and to improve the OTL measures in an international context. 1. A Confirmatory Factorial Analysis; four factors were extracted. 2. A Multiple Group Confirmatory Factor Analysis (MGCF) was used to test the structural invariance of the four factor structure across countries; configural, metric and scalar invariance have been checked for. 3. As correlations of the four factors with reading achievement were low and response style biases were suspected, a bidimensional IRT model was implemented, aiming at disentangling response bias from OTL dimensions. A two and a three dimensions models were tested. 4. Structural invariance of the two and several three dimensional models were tested; the three dimensional model with the best fit and which better met the criterion for invariance was kept for further analyses exploring the content (relationships between OTL dimensions and reading achievement).

Expected Outcomes

1. CFA Four factors were extracted: interpretation of literary texts; non-continuous texts (graphs, diagrams, maps); traditional literature course (poetry, memorize by heart); use of functional reading materials. 2. MGCFA, four factors model The Multiple Group Confirmatory Factor Analyses showed that configural and metric invariance were met, but scalar invariance was not. However, as correlations of the four factors with reading achievement were surprisingly low and response style biases were suspected, further data analysis solutions were explored. 3. IRT within-item bidimensional scaling All the OTL items were allocated to the first dimension, and some OTL items were allocated to the second dimension. Those latter items were selected taking into account what PISA assesses (framework) and what is known about the effectiveness of reading instructional strategies. We interpreted dimension 1 as a general component, capturing some basic OTL, but also general response style, whereas dimension 2 captures the effective OTL variables which can potentially enhance reading comprehension. Results were promising: dimension 1 (response style bias) was negatively correlated with reading achievement in all the countries (median -0.17). Dimension 2 had a robust correlation with reading achievement in most of the countries (median 0.62). 4. MGCFA, bi and tri-dimensional within-item IRT Despite the promising results, the analyses aimed at testing the invariance of the bi-dimensional model gave unsatisfactory results: fits were not good, configural, metric and scalar invariance were not met. Eventually, we came up with a tri-dimensional model (response style, OTL for non-continuous texts, OTL for fiction texts) which met configural and metric invariance and showed robust correlations with reading achievement in all countries. Conclusion: Response style in answering OTL questions (or other self-reported measures using Likert scales) have to and can be addressed in comparative studies. When response biases are fixe, strong correlations between OTL and reading achievement are observed.

References

Lundberg (1994). The Teaching of Reading. In W. B. Elley (ed.). The IEA Study of Reading Literacy: Achievement and Instruction in Thirty-Two School Systems. (pp. 149-193). Pergamon. Oakes, J. , Gamoran, A. & Page, R. N. (1992). Curriculum differentiation: Opportunities, outcomes, and meanings. In P. W. Jackson (Ed.). Handbook of research on curriculum (pp. 570-608). New York: Macmillan. Oecd (2010), PISA 2009 Assessment Framework. Key competencies in reading, mathematics and science. Paris: Oecd. Schmidt, W.H. and Maier , A. (2009) The Reading and Math Wars, in Sykes, G., Schneider, B., Plank, D. N. (eds), Handbook of Education Policy Research, Routledge Van de Vijver, F. & Poortinga, Y. (1997). Towards an Integrated Analysis of Bias in Cross-Cultural Assessment. European Journal of Psychological Assessment, 13-1, 29-37. Yang, Y., Harkness, J. A., Chin, T.-Y, & Villar, A. (2010). Response Styles and Cultures. In J.A. Harkness, M. Braun, B. Edwards, T.P. Johnson, L. Lyberg, P. Ph. Mohler, P.-E. Pennell, T. W. Smith (eds). Survey Methods in Multinational; Multiregional and Multicultural Contexts. (pp. 203-233). Hoboken: Wiley & Sons.

Author Information

Dominique Lafontaine (presenting / submitting)

University of Liege

Education

Liege

Svenja Vieluf

DIPF, Frankfurt-am-Main

Baye Ariane

University of Liege, Belgium

Monseur Christian