The Predictive Accuracy and the Predictive Validity of Academically Oriented Teachers' Recommendations for Students of Different Socio-economic Status
Author(s):
Denisa-Felicia Dudas (presenting / submitting) Wilfried Bos
Conference:
ECER 2014
Format:
Paper

Session Information

09 SES 02 B, Investigating Conditions of School Choice and Selection

Paper Session

Time:
2014-09-02
15:15-16:45
Room:
B012 Anfiteatro
Chair:
Monica Rosén

Contribution

Teachers’ recommendations for students’ educational careers and school track choices play an important role in students’ future academic performance. This is the case in Germany when it comes to teachers’ track recommendations at the end of primary school. On the basis of the students’ grades, primary school teachers recommend a secondary track (Nichtgymnasium or Gymnasium). These recommendations not only depend on the diagnostic but also on the predictive accuracy of teachers’ judgments. However, in some federal states students are allowed to select a different track than that recommended by the primary school teacher. Yet, recent research shows that students from families of a lower socio-economic-status (SES) more frequently attend less demanding secondary tracks and achieve lower test-scores compared to socio-economically privileged students (e.g., Boone & Van Houtte 2013; OECD 2013). Although there is ample evidence on the accuracy (e.g., Martin & Shapiro 2011) and on the predictive validity (e.g., Dollmann 2011) of teacher judgments, research has yet to evaluate both concepts together by taking medium- and long-range prediction studies into account.

Consistent with the evaluative framework for diagnostic screenings and batteries by Jenkins, Hudson and Johnson (2007) merely reporting predictive validity coefficients – efficacy – provides weak evidence for the predictive (classification) accuracy – effectiveness – of teachers' track recommendations. For this reason, the present study aims to expand the research perspective by theoretically and empirically describing the concept of predictive accuracy, particularly in contrast to the predictive validity and with special regard to teachers' academically oriented recommendations as loosely organized selecting measures. This research proposal focuses on three central questions: (1) How accurate are selecting measures that are almost exclusively based on teachers’ track recommendations when predicting students’ future academic achievement? (2) Are selecting measures more or less accurate when predicting the academic achievement of students from different socio-economic backgrounds on the basis of teachers’ track recommendations? (3) How is the predictive validity of selecting measures influenced by the interplay between teachers' track recommendations and students’ SES?

Method

The sampling frame for this study is the Competencies and Attitudes of Students Study (KESS), a longitudinal study in the federal state of Hamburg (Germany) that covers one entire cohort of students (N = 6 497; 52,3% female). Dependent measures include students’ performance in reading at the end of grade 8 and predictor variables include teachers’ recommendations for academically and non-academically oriented secondary tracks at the end of grade 4, students’ prior achievement in reading, students’ SES, the interaction term "academically oriented teachers' recommendations * students’ SES" and other characteristics such as language background and gender. Targets of this selecting procedure are all students considered to be at least proficient in reading, where at least proficient is equal to the test scores at the 25th percentile of the distribution of reading achievement at more demanding secondary tracks (Gymnasium). Descriptive statistics permit us to examine the sensitivity and the specificity of this selecting procedure in differentiating between students who are likely to exceed the given criterion measure for reading proficiency and students who are likely to fall below it. Sensitivity refers to the accuracy of teachers' recommendations to predict students who perform at or above the 25th percentile (i.e., "true positives"), whereas specificity refers to their accuracy to predict students who perform at or below the 25th percentile compared to the distribution of reading achievement at more demanding secondary tracks (i.e., “true negatives"). First, we will analyse how well the distribution of academically (N = 3 416) versus non-academically (N = 3 081) oriented teachers' recommendations corresponds with students’ performing levels above (N = 3 497) or below (N = 3 000) the 25th percentile in reading at more demanding secondary tracks. Besides this, our data includes information on the degree to which academically (N = 1 935) versus non-academically (N = 1 665) oriented teachers' recommendations accurately predict high (N = 1 688) and low SES students (N = 1 912) who are either proficient to advanced (N = 1 264; N = 712) or non-proficient (N = 424; N = 1 200) in reading. Second, to determine the association between academically oriented teachers' recommendations, students' prior achievement in reading, students' SES and the interaction term "academically oriented teachers' recommendations * students’ SES" on the one hand, and reading performance, on the other, we make use of binary regression models with full-information maximum likelihood (FIML) estimators.

Expected Outcomes

When all students were included the selecting measure failed to predict accurately those who performed at or beyond the criterion measure required for reading proficiency (75,1%) versus those who did not obtain proficiency status (76,0%). We also found that this selecting procedure overestimates high SES students, predicting those who performed at or above reading proficiency (86,8%) far more accurate than those who failed to reach the criterion measure needed for reading proficiency (42,9%). Conversely, it underestimates low SES students, predicting non-proficient readers (84,9%) far more accurate than proficient or advanced readers (58,3%). These findings indicate good sensitivity (0.87) and extremely low specificity (0.43) for high SES students as well as good specificity (0.85) and poor sensitivity (0.58) for low SES students. Binary logistic regression models show that the odds of reaching proficiency status if teachers express a positive bias towards high SES students through their academically oriented recommendations are approximately 1.20 the odds of reaching proficiency status relative to low students even when statistically controlling for prior knowledge. In interpreting this study's findings, we note that the levels of sensitivity and specificity are at best fair (e.g., Fuchs et al. 2007), resulting in a high percentage of undetected proficient and better than proficient as well as non-proficient readers. A selecting measure that produces a large number of undetected proficient to advanced readers diminishes schools' efforts to work effectively by depriving successful students of the instruction that they require. According to the extent to which later reading achievement was mediated by students' SES, we further conclude that this selecting procedure is not only ineffective but also inefficient. By evaluating the consequences of academically oriented teachers' recommendations, we demonstrated that unstandardized selecting procedures at the transition from primary to secondary school do not meet the meritocratic ideal that drives contemporary educational policy.

References

Boone, S. & Van Houtte, M. (2013): Why are teacher recommendations at the transition from primary to secondary education socially biased? A mixed-methods research. In: British Journal of Sociology of Education 34 (1), S. 20–38. Dollmann, J. (2011): Verbindliche und unverbindliche Grundschulempfehlungen und soziale Ungleichheiten am ersten Bildungsübergang. In: Kölner Zeitschrift für Soziologie und Sozialpsychologie 63 (4), S. 431–457. Fuchs, L.S., Fuchs, D., Compton, D.L., Bryant, J.D., Hamlett, C.L. & Seethaler, P.M. (2007): Mathematics screening and progress monitoring at first grade: Implications for responsiveness to intervention. In: Exceptional Children 73 (3), S. 311–330. Jenkins, J.R., Hudson, R.F. & Johnson, E.S. (2007): Screening for at-risk readers in a response to intervention framework. In: School Psychology Review 36 (4), S. 582–600. Martin, S.D. & Shapiro, E.S. (2011): Examining the accuracy of teachers' judgments of DIBELS performance. In: Psychology in the Schools 48 (4), S. 343–356. OECD (2013): PISA 2012 Results: Excellence through Equity. Giving Every Student the Chance to Succeed (Volume II). Paris (OECD Publishing).

Author Information

Denisa-Felicia Dudas (presenting / submitting)
Institute for School Development Research
Dortmund
Institute for School Development Research (IFS), Germany

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.