The use of international large-scale assessment (ILSA) data has become ubiquitous among educational researchers and policy makers. Results from these large-scale assessments are often cited in different contexts to refer to differences between groups, provide indicators of proficiency of students, make distinctions between contextual conditions, and so on. But conclusions and inferences drawn from using these data are not without some estimable amount of statistical uncertainty. ILSAs use complex sampling and assessment designs to deliver the assessment to the population of interest. These designs are referred to as multiple-matrix sampling designs. Under these designs, they administer relatively small amounts of overlapping material selected from larger representative pools of items, to relatively small samples from the target population. This is done in order to maximize the amount of information collected from the population, while minimizing individual burden, and minimizing the number of people that need to participate in the assessment. This is beneficial in terms of reducing operational costs of these assessments. As a consequence, when using these designs, there is a loss of precision in the information collected. This resulting uncertainty is calculable and expressed in the form of standard errors. There are two main sources of these standard errors: the sampling design and the assessment design. This section of the symposium will present an overview of the principles, methods and procedures that are used to calculate these standard errors of the statistics, mainly sampling replication and plausible values. It will also present the set of procedures and formulas used in the calculation of standard errors. The goal is to provide a didactic guide of which procedures to use when, rather than provide mathematical explanations and derivations of the formulas.

Gonzalez, E. (2014) “Calculating Standard Errors of Sample Statistics when Using International Large-Scale Assessment Data” In Strietholt, R., Bos, W., Gustafsson, J.-E., & Rosén, M. (Eds.) Educational Policy Evaluation Through International Comparative Assessments. Münster: Waxmann.