Conceptual framework

A common educational policy in many countries involves standardized assessment of student outcomes, which can be used in different ways (Eurydice, 2009), most often for accountability purposes. Knowing that student achievement depends strongly on family background, schools’ average test scores are likely to reflect social composition of the student body rather than the quality of instruction. This is why new procedures have been proposed to obtain a more reliable measure of school contribution to student learning, value-added modelling (VAM) being the most prominent one.

VAM has its roots in educational effectiveness research (Reynolds et al., 2014; Scheerens, 2015) and encompasses many different methods with a common focus on controlling for students’ prior achievement when assessing their performance in a later time point. This enables to compare the improvement of students irrespective of their absolute achievement level (OECD, 2008; Tekwe et al., 2004).

A wide-spread approach to VAM relies on hierarchical linear modelling. Some models use gains (difference between two consecutive test scores) as outcome variable, whereas others use current scores as outcome variable and prior achievement as a covariate. These so-called covariate adjustment models are more common because they do not need vertically scaled tests. Researchers in education tend to prefer random effects models over fixed effects models. The literature suggests that random effects models provide more precise value-added (VA) estimates, especially for smaller class sizes (Everson, 2017). In addition, models differ in whether they do or do not include other predictors than prior achievement, such as family background. Adding sociodemographic variables into the model usually leads to significant changes in VA estimates and is mostly preferred (Tekwe et al., 2004; Leckie & Goldstein, 2017), although it may unintentionally lower expectations of disadvantaged groups of students. On the other hand, including SES into the Tennessee model (TVAAS), which controls for more than one prior achievement, had only limited influence on resulting VA scores (Ballou, Sanders, & Wright, 2004). Simple models are generally regarded as better than more complex ones if they are “good enough” to provide reliable VA scores (Tekwe et al., 2004). The choice of a particular model inevitably involves political considerations and reflects specific context of a given education system (OECD, 2008).

The Czech Republic has followed the general trend towards standardized assessment of student outcomes, although the public opinion is largely against testing. Two centralized exams are currently used in practice. Since the school year 2010/2011 students in secondary schools are expected to pass a centrally administered part of their final exam, which is one of the preconditions for secondary school leaving certificate. In the school year 2016/2017 a centrally administered entrance exam for secondary schools was introduced. Scores from these two assessments can possibly be used for value-added measurement in the future. However, VAM is still a rather neglected topic among Czech educational researchers, most likely due to its limited relevance for current educational policy and practice.

Objective

The aim of the present study is to provide an overview of approaches to VAM in selected European countries based on a comparative analysis. Conclusions drawn from the analysis are intended as input for a discussion about value-added measuring in the Czech Republic.

Research questions

Which VA models have been proposed, piloted and/or used in real educational contexts of other European countries?

What are the (declared) reasons for specific model choices?

Do the models correspond to recommendations derived from research literature?

What implications could be drawn for the case of the Czech Republic?

The study will be based on a comparative analysis of value-added models used in selected European countries. Of interest are here countries where value-added of schools is officially reported as part of school accountability measures (e.g., England, France) as well as countries where value-added measurement was piloted on real data from national assessments (e.g., Norway, Poland). Attempts to measure value-added of schools within research projects will not be considered. The focus of the study is on application of sophisticated statistical techniques in real settings where a balance between simplicity and precision of value-added estimates is one of the crucial problems to be solved. First, information about models used in selected countries will be collected from official websites, government reports, documents and other available sources. Some information can be acquired from texts written in English and French, but texts in other national languages will also be included in the analysis. Work with these documents will be facilitated by automatic translation services, such as Google Translate. Previous experience proved that this was feasible. In addition, local experts can be consulted by e-mail. Second, collected information about models will be compared according to a set of predefined criteria, such as fixed or random effects, other predictors than prior achievement (yes/no), number and type of contextual variables included in the models, declared reasons for model choice, etc. Third, results of the second step will be compared with recommendations derived from research literature and discussed with regards to a possible application of value-added modelling in the Czech Republic.

Value-added models used in European countries tend to control not only for prior achievement but also for students’ sociodemographic characteristics. This is in line with research literature, which suggests that models with sociodemographic variables provide more precise value-added estimates. However, the actual choice of sociodemographic variables included in the models depends on available data rather than on methodological considerations, so there is a risk of omitting important variables. The Czech Republic is a country where socioeconomic background strongly affects educational outcomes. This has been repeatedly documented by international studies, such as PISA or TIMSS, as well as by national research projects. Omitting socioeconomic variables from the model would most probably lead to biased value-added estimates. However, data about student background are not routinely collected for the purposes of educational statistics and attempts to collect such sensitive data can meet strong resistance from the public. Before any value-added measurement could be introduced, it will be necessary to open a public discussion about collecting data on student background and safeguard that they will not be misused.

Ballou, D., Sanders, W., & Wright, P. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37–65. Eurydice (2009). National testing of pupils in Europe: Objectives, organisation and use of results. Brussels: Education, Audiovisual and Culture Executive Agency. Everson, K. C. (2017). Value-added modelling and educational accountability: Are we answering the real questions? Review of Educational Research, 87(1), 35–70. Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992–2016: ‘Contextual value-added’, ‘expected progress’ and ‘progress 8’. British Educational Research Journal. doi: 10.1002/berj.3246. OECD (2008). Measuring improvements in learning outcomes. Best practices to assess the value-added of schools. Paris: OECD Publishing. Reynolds, D., Sammons, P., De Fraine, B., Van Damme, J., Townsend, T., Teddlie, Ch., & Stringfield, S. (2014). Educational effectiveness research (EER): A state-of-the-art review. School Effectiveness and School Improvement, 25(2), 197–230. Scheerens, J. (2015). Theories on educational effectiveness and ineffectiveness. School Effectiveness and School Improvement, 26(1), 10–31. Tekwe, C. D., Carter, R. L., Ma, C.-X., Algina, J., Lucas, M. E., Roth, J., … Resnick, M. B. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1), 11–36.