Abstract

Romanian education has served some students well, as evidenced by successful performance in math and science Olympiads (OECD, 2000). Consequently, teachers are encouraged to focus on high-performing students with little incentives to address the needs to those who were less advantaged in this race. So, the success of teachers and schools is limited in Romanian education to the achievement of high performers. The focus on academic excellence in national examinations including Olympiads leads to the prevalence of private tutoring which accentuates the inequalities to the benefit of those students who come from wealthier families. The system currently allows only a minority of its students to excel.

According to the results of Romania in TIMSS 2011 – one of the large-scale assessments that allow for comparison of student achievement among educational systems, the system currently enables 5% of its students enrolled in 8th grade to excel (Mullis et al., 2013). Those at the top demonstrates the same level of knowledge and skills as their international counterparts. However, too many of the eighth graders (29%) do not master the basics necessary for continuation of learning in such a way that these acquisitions will avoid failure at the end of compulsory school. These results are confirmed by the latest participation in PISA. 40% of young Romanians do not master the basic competencies necessary for full participation in society (OECD, 2016). 

 Furthermore, “outcomes indicate that many Romanian children do not achieve their potential” (Kitchen et al., 2017, p.38) and this coincides with the lack of consistency in approaches related to rising educational outcomes of all students, although there are several initiatives to improve participation in education, especially with students at-risk.

To contribute to this discussion, this present study explores the eighth grade students' achievement in mathematics in TIMSS 2011 (Trends in International Mathematics and Science Study), taking a comparative perspective within Romania. Large-scale assessment studies in education allow for comparison of student achievement among educational systems, with the aim of understanding how input and processes could be improved.

In the search for explanations at the micro and macro levels, the mathematics achievement of students in different domains such as Algebra, Geometry, Data and Chance and Numbers are compared. Our interest in this study is to understand more which students, the lower or the higher achievers, need more support and resources according to their potential and which are the domains of mathematics where the discrepancies are made manifest. 

The crucial questions are: Which students are achieving the closest to what is expected from them to achieve? And at which level of the TIMSS test scores is the system failing the most?

Finding answers to our questions is extremely important for the country since this type of analysis is unique at the national level, pointing out the great discrepancies between students and the fact that a large portion of our students do not reach their potential. Providing such results is likely to support a structural reform evidence-based.

The study uses the Romanian eighth graders' results in TIMSS 2011, mathematics. The sample included about 4700 students nested within schools, the unit of analysis being the students. TIMSS mathematics items were organized around two dimensions: content dimension, specifying the subject matter to be assessed; and cognitive dimension, specifying the thinking processes to be assessed. To measure the content dimension, 217 mathematics items were administered in TIMSS 2011, 8th grade, and all these items are used for this analysis. Their distribution on content domains was: Algebra, 70 items; Geometry, 43; Data and Chance, 43; and Number, 61. TIMSS 2011 includes multiple-choice items where the student chooses the correct answer from four response options, and constructed-response items where the student is required to provide a written response. For this TIMSS assessment, the goal was that at least half of the total number of score points represented by all the questions should come from multiple-choice questions. The program and software used for this profile analysis is PROFILEG, developed by the econometrician Norman Verhelst. Based on the Verhelst (2018), profile analysis is used for to detect systematic deviations at group level, from the predictions of a measurement model, in this case for binary credit items of TIMSS 2011. The estimated deviations, being the difference between the expected and the observed subscore in each category of items and for each respondent, are the focus of profile analysis. When considered jointly for all categories of items, we are talking about deviation profiles, the final analysis being a comparison of these aggregated results across groups. The expected subscore is expected given the total score on the test, being a function of the item parameters only (Verhelst, 2018). A calibration of the items is necessary, being performed in this study as well. The deviation profile of a respondent, of interest here, is defined as the difference between his/her observed profile and the conditional expected profile given the test score, under the measurement model used. The average deviation profile per content domain (Algebra, Geometry, Data and Chance, and Number) will be presented for eight groups of achievement scores, from the lowest to the highest, with a focused attention on the two lowest and the two highest groups. Considering that, TIMSS 2011 uses test items as multiple-choice items and constructed-response items, for every respondent the results obtained will be split as well by these two types of items.

Our hypothesis for this study was that the lowest achieving groups are achieving lower than expected and the highest achieving groups (the Olympiads students usually) are achieving higher than expected. We will present here the results for one content domain, namely Algebra. What we find, in general lines, is that for the two lowest groups, their deviation profile as the difference between the observed and the expected profile, indicates that the students in the lowest group are performing on average significantly worse than expected on the Algebra items. Moreover, for the two highest groups, their deviation profile as the difference between the observed and the expected profile indicates that the students in the highest group are performing on average significantly better than expected on the Algebra items. What makes the main difference in both cases are the constructed-response items, considering that the lowest achieving groups are performing on average significantly worse than expected only on the constructed-response Algebra items; while the students in the highest two groups are performing on average significantly better than expected mainly on the constructed-response Algebra items. An important step towards an equitable education would be to identify what contributes to the current gap presented in students’ achievement. These and future results will guide policymakers in their evidence-based decisions. Occupational and economic disadvantages are seen in relation to developmental trajectories (Bynner & Parsons 1997) and related to mathematics deficits (Kaufmann et al. 2013). Despite growing body of research on the data provided by large scale-assessments, relatively little work has been done on extreme segments in population distribution. Refining the ability to surface structural inequities as evidence for what our students experience represents one way to prevent the reproduction of an inequitable system and, at the same time, a moral obligation for the upholders of quality education.

Addey, C., Sellar S., Steiner-Khamsi,G., Lingard, B. & Verger, A. (2017). The rise of international large-scale assessments and rationales for participation, Compare: A Journal of Comparative and International Education, DOI: 10.1080/03057925.2017.1301399 Burns D and Darling-Hammond L (2014) Teaching Around the World: What Can TALIS Tell Us? Stanford, CA: Stanford Center for Opportunity Policy in Education. Chung J (2015) International comparison and educational policy learning: Looking north to Finland. Compare: A Journal of Comparative and International Education 45(3): 475–479 European Commission (2013). PISA 2012 performance and first inferences regarding education and training policies in Europe. Retrieved from: http://ec.europa.eu/education/policy/strategic framework/doc/pisa2012_en.pdf Fenwick, T., Mangez, E., and Ozga, J. (Eds.). (2014). Governing knowledge: Comparison, knowledge-based technologies and expertise in the regulation of education. New York: Routledge. Griffin P, McGaw B and Care E (2012) Assessment and Teaching of 21st Century Skills. Dordrecht: Springer. Jackson, K., & Cobb, P. (2010). Refining a vision of ambitious mathematics instruction to address issues of equity. Paper presented at the Annual Meeting of the American Educational Research Association, Denver, CO. Kamens, D. H. and McNeely, C. L. (2010). Globalization and the growth of international educational testing and national assessment. Comparative Education Review, Vol. 54(1), 5-25. Kitchen, H., et al. (2017), Romania 2017, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris. http://dx.doi.org/10.1787/9789264274051-en OECD (2013). PISA 2012 results: What students know and Can Do. Student performance in Mathematics, Reading and Science. Volume I. Retrieved from: http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-I.pdf OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264266490-en Plomp, T., & Nieveen, N. (Eds.). (2013). Educational design research. Enschede, the Netherlands: SLO. Rutkowski, L., Gonzalez, E., Joncas, M. & von Davier, M. (2010). International large-scale assessment data: Recommendations for secondary analysis and reporting. Educational Researcher, 39(2), 142-151 Rutkowski, L., Gonzalez, E. & von Davier, M. (2009). What are plausible values and why are they useful? In IERI Monographs Series: Issues and methodologies in large-scale assessments (vol 2) von Davier, M., Sinharay, S., Oranje, A., & Beaton, A. (2007). The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 1039–1055). Amsterdam: Elsevier