Comparing Mathematics Competencies Across Assessments
Author(s):
Ursula Itzlinger-Bruneforth (presenting / submitting) Jan Steinfeld (presenting)
Conference:
ECER 2017
Network:
09. Assessment, Evaluation, Testing and Measurement
Format:
Paper

Session Information

09 SES 06 A, Issues in Linking Large-Scale-Assessments

Paper Session

Time:
2017-08-23
15:30-17:00
Room:
W3.11
Chair:
João Marôco

Contribution

Less than ten years ago, Austria introduced national assessments in grades 4 and 8, assessing the degree to which national education standards are reached by students (Bildungsstandardüberprüfung, “BIST-Ü”) and a standardized high school leaving exam (“Matura”) in academic schools grade 12, regulating access to University.  The introduction of these assessments has been accompanied by heated discussions about educational goals, loss of academic freedom and the fear of “dumbing down” in the assessed subjects, specifically in schools leading to university access. On the other hand, students, parents and teachers complained about lack of preparation for the national high school leaving exam in grade 12, fearing high fail rates. Four years after the first national assessment in grade 8 in mathematics (2012), a considerable portion of the cohort has undergone the national high school leaving exam in 2016. Despite the tests being fundamentally different, we want to analyse the results for their possible relation: can we assume an overarching mathematics competence, allowing validation of the results of one assessment with the other? For both assessments, validity evidence is more arbitrary than stringent; so possible evidence for the interpretation (Hartig et al. 2008; Fellinger et al. 2016) coming from other, independent sources would represent a welcome finding. From a more practical viewpoint, the prediction power of one test four years ahead of the other could be of great interest for teachers, students and parents.

The aforementioned differences in the assessments comprise of slightly different mathematics frameworks, differences in the item formats used as well as different scoring routines. The most important difference is probably a characteristic of the test itself: the Matura represents a high stakes test (students are allowed university access or not), the BIST-Ü assessment, not giving any official certificate to students, is low-stakes for the test-takers. However, these two assessments represent the only standardized assessments available in Austria, which may be linked at least at school level.

Method

For this analysis, we use the data of the national assessment of educational standards (BIST-Ü) in mathematics in 2012, along with data of the national high school leaving exam (Matura) in 2016. The students of both assessments belong to the same cohort. For reasons of data privacy, we cannot link students directly; but we can use the data from those schools in Austria (n=262 academic schools) which offer instruction in both grades 8 and 12, and link the performances of the respective schools. This student population represents about one quarter of all students of a cohort. These 262 schools are distributed over all geographic regions of Austria as well as urban/rural strata, but the student population is different from the total student population in some important characteristics (most notably gender and migration background). Any comparison between assessment results in grade 8 for the entire population and the subpopulation used for this analysis therefore should use weights to control for the differences in the populations. An additional source for noise are students dropping in and out of the cohort due to repetition or skipping of grades but the effect is deemed to be comparatively small. Merging the schools by an unique identifier, we may relate the results on school level without errors. However, the attrition rate may be quite different between schools; a comparison of these rates over all schools offering both grades is still outstanding .

Expected Outcomes

In the light of the aforementioned differences, the results of first analysis are surprisingly high; a correlation of r(262)=.72 could be established. In a further step we controlled for students who stated in a background questionnaire their intention to stay in a school leading to Matura. Despite a decrease both in population and in variation in the results, the correlation between grade 8 results and Matura stays the same (r(262)=.72). Analysis which is still outstanding concerns the results by gender. In most assessments, boys outperform girls in mathematics in Austria. This is also true for the grade 8 assessment; the difference between boys and girls results is even bigger in the schools leading to Matura (AHS) than in other school forms (APS) (Schreiner & Breit, 2012). For upper secondary schools leading to Matura, Neuwirth (2015) showed that the resulting gender-gap is also a statistical problem. While more girls stay in Matura-leading schools, boys more often leave this type of schools. Keeping in mind this different gender representation in the different school forms, we want to analyse whether this gender gap widens, closes or stays the same between grade 8 and grade 12.

References

Fellinger, R., Kiefer, T., Robitzsch, A., & Trendtel, M. (2016). Aspekte der Validierung. In S. Breit & C. Schreiner (Hrsg.), Large-Scale Assessment mit R. Methodische Grundlagen der österreichischen Bildungsstandardüberprüfung (S. 363-398). Wien: facultas. Hartig, J., Frey, A., & Jude, N. (2008). Validität. In H. Moosbrugger & A. Kevala (Hrsg.), Testtheorie und Fragebogenkonstruktion (S. 134-163). Berlin: Springer. Neuwirth, E. (2015). Was die Ergebnisse der Zentralmatura wirklich zeigen. Abgerufen am 21.12.2016, von http://www.neuwirth.priv.at/forblogs/BLErgebnisse.html) Schreiner, C., & Breit, S. (Hrsg.). (2012). Standardüberprüfung 2013 Mathematik, 8. Schulstufe. Bundesergebnisbericht. Salzburg: BIFIE. Zugriff unter https://www.bifie.at/system/files/dl/01_BiSt-UE_M8_2012_Bundesergebnisbericht.pdf

Author Information

Ursula Itzlinger-Bruneforth (presenting / submitting)
BIFIE
BISTA
Salzburg
Jan Steinfeld (presenting)
MoE, Austria

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.