Background

Educational freedom is a key concept in various educational systems worldwide. Amongst others, the free choice of instructional learning support materials, such as textbooks[1] (e.g., England, Finland, Hungary, and Sweden; Exhibit 7 in Mullis, Martin, Minnich, Stanco, & Arora, 2012) originates from the idea of ensuring educational freedom. In many educational systems, amongst others in Flanders (i.e., the Dutch speaking part of Belgium), textbook is the prime factor in primary education through which educational practitioners give rise to their day-to-day mathematics teaching content and practices (Mullis et al., 2012; Törnroos, 2015), and in this may be assumed to affect educational outcomes.

The position of textbooks in education is closely related to two main aspects largely affecting students’ educational outcomes: the implemented curriculum and opportunity to learn (OTL). The implemented curriculum, referring to the educational processes (Mullis & Martin, 2013), includes issues like how do schools and teachers make sure that all students receive education that optimally suits their abilities and interests, and how do we treat students coming from less advantaged background in class. OTL, as part of the implemented curriculum, plays a major role in education, referring to the idea that the extent to which students have been educated in certain topics should be considered in evaluating students’ educational outcomes (McDonnell, 1995). Students’ OTL depends on four aspects (Stevens, 1993): (a) content coverage, (b) content exposure, (c) content emphasis, and (d) quality of instruction. As shown by Törnroos (2005), the textbook is one of the main factors determining students’ OTL and as a consequence shaping the implemented curriculum.

In Flanders, textbooks play a major part in the content and sequence of teachers’ daily educational practices. This central role is combined with (a) teachers’ free textbook choice, (b) publishers’ freedom of publishing textbooks, and (c) absence of any information nor (governmental) guidelines on (the quality of) textbooks.

However, it may easily be argued that a free, rational, and conscientious choice may only be guaranteed when educational stakeholders have sufficient information available on textbook quality (e.g., provided by government). Additionally, publishers might optimize their textbooks based on this information. Furthermore, the inability to investigate causal effects to date may not serve as an excuse for not comparing textbooks at all. Together with indispensable nuances, research on the relation between textbooks and educational outcomes may give us clues of important levers for textbook improvements.

Research aims and questions

This study aims at giving insight in the strengths and weaknesses of different mathematics textbooks in Flanders by empirically investigating their relation with achievement and (social and ethnic) equity. The following research questions are formulated:

RQ1    Is there a relation between textbook and students’ math achievement?

RQ1a  Is there a relation between textbook and overall math achievement?

RQ1b  Is there a relation between textbook and math achievement in the different cognitive and content domains?

RQ1c  Is the relation between textbook and math achievement different between the different cognitive and content domains, respectively?

RQ1d Does the relation between textbook and achievement vary across the achievement distribution?  (i.e., Is the relation between textbook and achievement different for (very) low, average or (very) high achieving students?)

RQ2     What is the role of textbook in educational equity? (i.e., Does textbook matter in the relation between student SES/ethnicity and math achievement?)

[1] When we refer in this proposal to textbook, we also refer to the accompanying educational support material provided along with that particular textbook (e.g., workbook, worksheets, tests, website, learning platform, etc.).

Methods Data. We made use of data of TIMSS 2015 Grade 4 of Flanders (5,230 students, 276 classes, and 145 schools). Measures. Math achievement. Next to students’ overall math achievement, we examined achievement in three different content (i.e., data display, geometric shapes and measures, and number) and cognitive (i.e., knowing, applying, and reasoning) domains. Achievement was represented in the analyses by means of the five plausible values representing students’ underlying math ability, made available by TIMSS 2015. Equity. Equity is looked at as the absence of a relation between SES/Ethnicity with achievement. Textbook. By means of a national option in Flanders’ teacher questionnaire, teachers indicated the textbook predominantly used (publisher stated between brackets), choosing between five options. Textbook was taking into account in the analyses by means of dummy variables. Confounders. Confounders were taken into account in the analyses: student input characteristics (gender, SES, and ethnicity), school input characteristics (group composition in SES and ethnicity, and educational network), and process variables (student self-confidence in math and school emphasis on academic success). Analyses. We used multilevel modelling, with students nested within schools. Firstly, we looked at the relation between textbook and MATH (RQ1a). After the estimation of a Null model, we estimated the relation between textbook and MATH (Model 1) not taking into account any confounders. In Model 2a, we controlled for student background characteristics. Model 2b additionally included school background characteristics. A final model (Model 3) additionally included process variables. Secondly, we investigated the relation between textbook and achievement in the different content and cognitive domains, respectively, (RQ1b), making use of multivariate multilevel modelling. Difference tests for all significant coefficients of the correlation between textbook and achievement were performed in the analyses for RQ1c to get insight in the extent to which the relation between textbook and achievement differed according to the different domain. Thirdly, to get a grasp on the relation between textbook and achievement for the entire achievement distribution (RQ1d), quantile regression (QuantReg) analyses were performed, investigating the correlation between textbook and math for several quantiles (τ) of the distribution: 10th, 25th, median, 75th, and 90th. Finally, moderation models were estimated to investigate the extent to which textbook might counter existing inequity (RQ2), looked at by means of the relation between SES/LANG and achievement. Cross-level moderation terms were modelled by means of interaction terms between SES/LANG and textbook.

Results First of all, results show that differences in the relation between textbook and achievement exist, both with regard to overall math as to content and cognitive domains, even after controlling for student and school background factors, and process variables. Results suggest that students instructed with the least effective textbook are lacking behind two months of additional education compared to students instructed with the most effective textbook. Secondly, we found no differences between textbooks in achievement across the three different cognitive and content domains, respectively. Thirdly, results of quantreg suggest that the correlation between textbook and achievement is not significantly different at different quantiles of the distribution and compared to mean achievement. However, a general tendency can be seen in which the correlation between textbook and achievement is stronger and more frequently significant for the lower quantiles compared to the higher quantiles of the distribution, suggesting that textbook matters more for low than for high achieving students. Finally, no significant interactions between textbook and SES/LANG are found, not even after consecutively leaving out the control variables of Models 3, 2b, and 2a, indicating that textbook does not play any stimulating nor buffering role in the relation between SES/LANG and achievement. Discussion and conclusion Apart from the main limitation of this study -its correlational design- this study showed that (a) differences between textbooks in their relation with achievement exist, across content and cognitive domains, and mostly in the lower quantiles of the achievement distribution, and (b) no relation of textbooks with (in)equity exist. In this, this study shows the valuable contribution in investigating educational support materials’ relation to educational outcomes.

References McDonnell, L. M. (1995). Opportunity to learn as a research concept and a policy instrument. Educational Evaluation and Policy Analysis, 17, 305-322. Mullis, I. V. S., Martin, M. O., Minnich, C. A., Stanco, G. M., & Arora, A. (2012). TIMSS 2011 encyclopedia: Education policy and curriculum in mathematics and science (Volume 1). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Mullis, I. V. S., & Martin, M. O. (2013). TIMSS 2015 assessment frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: http://timssandpirls.bc.edu/timss2015/frameworks.html. Stevens, F. I. (1993). Applying an opportunity-to-learn conceptual framework to the investigation of the effects of teaching practices via secondary analyses of multiple-case-study summary data. Journal of Negro Education, 62, 232-248. Törnroos, J. (2005). Mathematic textbooks, op¬portunity to learn and student achievement. Studies in Educational Evaluation, 31, 315-327.