Large-scale surveys of student competencies require a high standard of comparability in terms of the competencies and attitudes assessed at various levels of the educational system, across various sub-populations, and at various points of measurement over time. These large scale studies suffer from constraints such as sample size, individual response time and high expectations with regard to the breadth of content coverage. The focus on the assessment of competencies often results in a rotated booklet design which enables increased coverage of competencies, while not placing too high a burden on any particular student (Adams, Wu & Carstensen, 2007). In order to make achievement scores comparable across points in time often data collection designs are chosen that establish a vertical link between test forms by use of “common items” (Von Davier et al., 2008). One common procedure for placing item parameters on a common scale is to apply the fixed-parameters method using the standard unidimensional Rasch model (such as the PCM; Masters, 1982), common items whose parameters are known are anchored or fixed to their pre-calibrated item parameters, estimated in a baseline test (t0). A second quite similar calibration method using the same Rasch measurement model is to treat data fromdifferent time points asgroups of independent „virtual persons“ by applying the concurrent calibration method. A third possibility is the application of multidimensional Rasch models and to treat each time point as a separate latent dimension. These models are implemented as special cases of the Multidimensional Random Coefficient Multinomial Logit (MRCML) models (Adams, Wilson & Wang, 1997).

The advantages and shortcomings of the methods mentioned are illustrated and compared using data from the German achievement study KESS (Competencies and Attitudes of Students). The study follows one entire cohort of students (N ≈ 9.000) in the metropolis of Hamburg from grade 4 to grade 8 by three measures every two years that include student attainment data as well as background information from questionnaires filled in by the students, their parents, teachers and schools. For the analysis to be presented data obtained from reading literacy tests of the first two measurements is used. The impact of the different measurement approaches will be explored by looking at important background variables of the students such as school track, gender, migration background and socio-economic status.

The paper is based on work in progress, but it is expected that by the direct comparison of three Rasch approaches will help to understand competence development in the field of evidence based testing. Results from data simulation studies do indicate that multidimensional Rasch modelling is the more appropriate method to estimate individual change when using the plausible value technique. In contrast the application of a standard unidimensional Rasch model for separate and concurrent calibrations of the data appears to be a powerful tool in order to identify valid and reliable anchors.