An Average Is Just an Average: What About Countries’ Low- and High-Performing Students?

Author(s):

David Miller(presenting / submitting)Yuan Zhang Frank Fonseca Varsha Radhakrishnan

Conference:

ECER 2016

Network:

11. Educational Improvement and Quality Assurance

Format:

Paper

Session Information

11 SES 03, Paper Session

Paper Session

Time:

2016-08-23

17:15-18:45

Room:

OB-E2.14

Chair:

Andra Fernate

Contribution

Analyses of results from large-scale international assessments typically focus on average student performance; in particular, how a country’s average performance compares to that of other countries and how it has changed over time (Coughlan, 2015; McKnight & Valverde, 1998; Sedghi, Arnett, & Chalabi, 2013; Shepherd, 2010). While such a focus is useful, it provides little insight into a country’s success in educating its low-and high-performing students. Helgason (1997) suggested that one single indicator is not sufficient in international benchmarking. A comprehensive picture of student performance and cross-national comparisons of student performance require a set of well-balanced measures, and as Scheerens and Hendriks (2004) noted, equity is a key factor to examine in evaluating the quality of an education system. Both low- and high-performing students need an appropriate and challenging education if they are to become contributing members of society (Badescu, D’Hombres, & Villalba, 2011; Barone & van de Werfhorst, 2011). Thus, countries that are committed to fostering equity and opportunity and technological and economic competitiveness should attempt to maximize the learning potential of both their low- and high-performing students and monitor their progress through the education system.

Published reports from large-scale international assessments, including PISA and TIMSS, have included tables with percentiles of achievement that show, for example, how scores at the 10^th and 90^th percentiles compare across countries. However, prior research has not systematically examined and statistically tested these gaps in achievement between low-and high-performing students and whether these achievement gaps have narrowed or widened over time.

Using fourth- and eighth-grade mathematics data from the 2003 and 2011 Trends in International Mathematics and Science Study (TIMSS), this analysis will address the following research questions:

What is the extent of the variation seen across countries in the mathematics achievement of low- and high-performing students?
What is the extent of the variation seen across countries in the size of students’ within-country achievement gaps in mathematics?
Across countries, has the mathematics achievement of low- and high-performing students changed over time?
Across countries, has the size of students’ achievement gaps in mathematics changed over time?

A plethora of research on the effects of schooling starting with the landmark release of the Coleman Report in the United States (Coleman et al., 1966) and the Plowden Report in the United Kingdom (Peaker, 1971; Plowden, 1967) has suggested that the majority of the variance in academic achievement could be explained by a student’s experiences and socioeconomic background prior to entering school and that differences in the quality of schools and teachers has only a small positive impact on student outcomes. However, subsequent research by Heyneman and Loxley (1983) found that in low-income countries, school-level factors were more important than student-level characteristics such as family socioeconomic status in determining academic achievement. Stemming from this theoretical framework, a fifth research question examined in this analysis is the following:

Using country-level data, what is the relationship between income inequality and mathematics achievement gaps?

Prior research has not specifically examined the relationship between country-level income inequality and gaps in the mathematics achievement of low- and high-performing students. We hypothesize that, at the country level, the more unequal the income distribution is, the larger the mathematics achievement gap among students. Furthermore, we hypothesize that the correlation between income inequality and mathematics achievement gaps will be stronger among industrialized OECD countries and weaker among less developed countries.

In keeping with the mission of EERA and ECER, this presentation will highlight results from European countries, recognizing wider, transnational contexts with their social, cultural and political similarities and differences.

Method

Using 2011 TIMSS data, we will address the first research question by examining cross-national differences in average mathematics performance at the 10th and 90th percentiles. TIMSS 2011 data will also be used to address the second research question, in which we examine the size of the within-country performance gaps in mathematics. Doing so will allow us to differentiate those countries that have a more equitable distribution of student performance (i.e., a relatively small point difference between mathematics scores at the 10th and 90th percentiles) and those countries that have a relatively large performance gap between low- and high-performing students. As another way of evaluating the cross-national variation in mathematics performance, we will present a graph that plots scores at the 10th percentile (shown on the x axis) and the 90th percentile (shown on the y axis). When arranged in this way, countries will generally appear in one of four quadrants: (1) top right: low and high performers both scoring relatively high, (2) bottom left: low and high performers both scoring relatively low, (3) bottom right: low performers scoring relatively high and high performers scoring relatively low, and (4) top left: low performers scoring relatively low and high performers scoring relatively high. The third research question will be addressed using TIMSS data from two time points: 2003 and 2011. Across countries, we will examine changes over time in average mathematics scores and in mathematics scores at the 10th and 90th percentiles. TIMSS 2003 and 2011 data will also be used to address the fourth research question. Across countries, we will examine whether the point difference between mathematics scores at the 10th and 90th percentiles significantly narrowed or widened during this 8-year time period. In testing the fifth research question, income inequality will be measured using the Gini coefficient, which can be derived from sources including the OECD, World Bank, and CIA World Factbook. This is a measure of statistical dispersion that represents the income distribution of a country's residents. A value of 0 represents perfect income equality, while a value of 100 represents absolute income inequality. Analyses will be carried out using the TIMSS International Data Explorer (IDE) tool, including the percentiles function and the newly released gap analysis test. The TIMSS IDE is available on the website of the National Center for Education Statistics (http://nces.ed.gov/surveys/international/ide/). SPSS statistical software will be used for correlation analyses.

Expected Outcomes

There were considerable cross-national differences in average mathematics performance at the 10th and 90th percentiles. For example, while at both grades the average scores of students in England and Finland did not statistically differ in 2011, low-performing students scored about 35 points higher in Finland than in England while high-performing students scored about 20 points higher in England than in Finland at both grades. In 2011, the size of the mathematics achievement gaps varied substantially across countries and the gaps, on average, tended to be smaller at fourth grade than at eighth grade (209 compared to 228). For example, the Netherlands at fourth grade (135) and Norway and Finland at eighth grade (166) had a more equitable distribution of student performance, while Romania and Turkey had relatively large performance gaps at both grades (range from 253 to 292). Examining countries’ average achievement over time can mask change that may be occurring with low- and high-performing students. For example, in Italy at fourth grade and in the Republic of Macedonia at eighth grade, average mathematics scores did not statistically change from 2003 to 2011. However, the scores of low-performing students increased by 20 points in Italy at fourth grade and decreased by 37 points in the Republic of Macedonia at eighth grade. At fourth grade, there were three countries (e.g., Hungary) where the mathematics achievement gap significantly widened from 2003 to 2011 and three countries (e.g., Slovenia) where the gap significantly narrowed. At eighth grade, however, there were 16 countries where the size of the gap statistically changed from 2003 to 2011, and in all cases the gaps widened over time. Preliminary analyses show that the correlation between country-level income inequality and mathematics achievement gaps was about 0.30, though it was considerably higher (about 0.57) for the subset of OECD countries.

References

Badescu, M., D’Hombres, B., & Villalba, E. (2011). Returns to education in European countries: Evidence from the European Community Statistics on Income and Living Conditions (EU-SILC). Joint Research Centre, European Commission. Luxembourg: Publications Office of the European Union. Barone, C. & van de Werfhorst, H. G. (2011). Education, cognitive skills and earnings in comparative perspective. International Sociology, 26(4), 483-502. Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfall, F. D., & Vork, R. L. (1966). Equality of educational opportunity. Washington, DC. Department of Health, Education, and Welfare. Coughlan, S. (2015). Asia tops biggest global school rankings. BBC, available at http://www.bbc.com/news/business-32608772. Helgason, S. (1997). International benchmarking experiences from OECD countries. Paper presented at Conference organized by the Danish Ministry of Finance on: International Benchmarking, Copenhagen. Heyneman, S. P., & Loxley, W. A. (1983). The effect of primary-school quality on academic achievement across twenty-nine high- and low-income countries. American Journal of Sociology, 88 (6), 1162-1194. McKnight, C. C. & Valverde, G.A. (1998). Explaining TIMSS Mathematics Achievement: a preliminary survey. In Kaiser, G. (Ed.), International comparison in mathematics education (48-67), Florence, KY, USA: Taylor & Francis. Peaker, G. F. (1971). The Plowden children four years later. London: National Foundation for Educational Research in England and Wales. Plowden, B. (1967). Children and their primary schools: A report of the Control Advisory Council for Education (England), Vol. 1. London: Her Majesty’s Stationery Office. Scheerens, J., & Hendriks, M. (2004). Benchmarking the quality of education. European Educational Research Journal, 3(1), 101. Sedghi, A. Arnett, G., & Chalabi, M. (2013). PISA 2012 results: which country does best at reading, maths and science? The Guardian, available at http://www.theguardian.com/news/datablog/2013/dec/03/pisa-results-country-best-reading-maths-science. Shepherd, J. (2010). World Education Rankings: Which Country Does Best at Reading, Maths and Science? The Guardian, available at http://www.theguardian.com/news/datablog/2010/dec/07/world-education-rankings-maths-science-reading.

Author Information

David Miller (presenting / submitting)

American Institutes for Research (AIR)

Washington

Yuan Zhang

University of Pittsburgh, United States of America

Frank Fonseca

American Institutes for Research (AIR), United States of America

Varsha Radhakrishnan