Formal Teacher Education and Student Ratings: Valid Measures of Teacher Competence?

Author(s):

Stefan Johansson(presenting / submitting)Eva Myrberg(presenting)

Conference:

ECER 2017

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 12 A, Teaching and Teacher Characteristics: Findings form large scale assessments

Paper Session

Time:

2017-08-25

09:00-10:30

Room:

W3.11

Chair:

Trude Nilsen

Contribution

An earlier, predominant view of student achievement as first and foremost affected by out-of-school factors such as individual family background or neighbourhood characteristics is challenged by school effectiveness research. Particularly, there is compelling evidence that teachers account for a significant portion of variance in achievement between classrooms Hanushek & Rivkin, 2012; Nye, Konstantopoulos, & Hedges, 2004). Results on the meaning of teacher competence are, however, far from conclusive and it has been disputed whether teacher effectiveness manifested in classrooms can be traced back to observable variables (Jackson, Rockoff & Staiger, 2014).

During latter decades schools and teachers have increasingly been subjects to evaluation of different kinds. Quality control and audit are regular elements in school systems that have adopted market principles. Evetts (2011) distinguished between organizational and occupational professionalism. Organizational professionalism is incorporating accountability, target-setting and performance review. Occupational professionalism is the more traditional historic form. It includes discretionary decision-making in complex cases, collegial authority, occupational control of the work and is based on trust in the practitioner by both clients and employers.

As result of a neo-liberal development involving deregulation, marketization and an organizational view on professionalism much interest from research is devoted to assessed competence. Teacher performance as measured by, for example, students or employers is sometimes used for designing in-service training, determining wage levels or to guide assignment of positions. For example, Kyriakides et al. (2014) concluded that students in grade four are able to provide valid data on the classroom behaviour of their teachers and that school stakeholders can use such data for school improvement. On the other hand, Blömeke, Olsen, and Suhl (2016) found that the correlation between student ratings of instructional quality and student achievement differ between countries.

As regards the significance of formal competence a growing body of studies indicate that teacher effectiveness is dependent on relevant teacher education with respect to specialization for teaching particular subjects in specific grades (Darling-Hammond 2014; Kleickman et al., 2013). Likely, effects of qualifications appropriate for grade level and subject may vary in importance depending on subject domain grade level. For science and mathematics, and particularly at secondary level, a sizeable proportion of empirical results from Europe and the US support the importance of specialized teachers (Baumert et al., 2010; Goe, 2007). Results for reading at primary level are meager (Snow, Griffin & Burns, 2005) though some evidence from the US indicates effects of relevant teacher education on reading achievement in lower grades (Darling-Hammond, 2014). Croninger et al. (2007) suggested that more precise measures of teachers’ preparation tend to be better predictors of student achievement than more conventional and broader measures, such as general certification status. In one of few studies that investigate effects of formal teacher competence on reading achievement in a Swedish context, and with data from PIRLS 2001, Johansson, Myrberg, and Rosén (2015) estimated substantial effects of teacher education relevant for subject and grade on 3^rd graders’ reading achievement levels.

Assessed competence may or may not be related to formal competence. In addition, their respective relationships to student achievement need further clarifications. Due to large contextual differences between educational systems (e.g., Blömeke, Olsen, & Suhl, 2016), country specific analyses are warranted.

Against this background, the purpose of this study is to investigate 1) the relationship between formal teacher competence and student ratings of teacher competence, 2) the relationship between the two different measures of competence and reading achievement. Teaching experience and student SES were used as controls in the analyses.

Method

The empirical base consists of data from IEA’s regularly recurring reading study, Progress in International Reading Literacy Study 2011 (PIRLS). Well over 30 countries have participated in PIRLS, which assesses student reading ability in fourth grade. In 2011, a national extension in Sweden provided unique data on aspects of teachers’ education. Six education variables were considered for the current study: 1) type of teacher education, 2) emphasis on reading pedagogy 3) emphasis on Swedish language, 4) pedagogical content knowledge in Swedish language (PCK) 5) number of semesters studying Swedish language, and 6) focus on primary school-years during initial. These six indicators define the latent variable “Tcomp” in the present study. Furthermore, seven items from the student questionnaire were used to operationalize the latent variable student assessed teacher competence (SAcomp). Among these items, some were national options. Items concern, for example, whether the teacher is easy to understand, helps when student has difficulties, explains what to do to become a better reader, knows what the teacher expect (4-point likert-scale). Besides these two variables, teaching experience–expressed by number of years taught, student socio-economic status (SES) and students’ reading achievement results were used in the analyses. The information regarding SES is collected from the parent questionnaire. The main method of analysis is multilevel Structural Equation Modeling (SEM) with latent variables (e.g., Hox, 2002). The latent teacher qualification variable was fitted at teacher level while the variable SAcomp could be formulated at both student and teacher level (classroom means). In the first step measurement models of these latent variables was formulated. Due to the many categorical items in variable Tcomp, the WLSMV (robust weighted least squares) estimator was required. Hence, the model was fitted at one-level only because Mplus 7.4, did not allow WLSMV for two-level modeling. The measurement model of Tcomp obtained acceptable fit. In a next step, factor scores of Tcomp were saved and merged to the student level dataset. PIRLS data has a nested observational structure, students being clustered in classrooms, and so on. We could thereby assign the factor scores of the individual teachers to their student. The measurement model of SAcomp obtained good fit. The analyses were conducted using the computer program Mplus (Muthén & Muthén, 1998-2012).

Expected Outcomes

The structural modeling began with a model specifying relationships between Tcomp, SAcomp and student reading achievement. SAcomp was neither significantly related to achievement (at student and teacher levels) nor correlated with Tcomp. However, Tcomp was positively related to student achievement (.20 t=2.45). At student level, a relationship beween SAcomp and achievement was specified, but no significant relation was estimated. When teaching experience was entered into the model, the effect of Tcomp on achievement remained about the same, and teaching experience did not itself have any relation to student achievement, as did not SAcomp. The correlation between teacher qualification and experience was modest, yet significant (.17). Interestingly, teaching experience had a significant relation to SAcomp (.21 t=2.77). This result indicated that experienced teachers were rated at higher levels by their students, which was not the case for the more qualified teachers. To shed further light on the characteristics of Tcomp and SAcomp, we introduced student SES. At the classroom level, SES accounted for substantial variance in student achievement (r2=.70). The effect of Tcomp on achievement became non-significant, showing that Tcomp was unevenly distributed between classrooms. In the case of SAcomp, a negative relation to SES at the teacher level was found (-.22 t=3.44). This means that students in low-SES schools rated their teachers higher, than did students in high-SES schools, although the more qualified teachers were allocated to the high-SES schools. The preliminary findings showed that teacher competence seems important for student achievement. However, more competent teachers were not rated higher. Further, in low-SES schools students tended to appreciate their teacher more. These between class differences may be indications of a differential treatment effect. For example, teachers may give feedback to their students differently depending on their expectations.

References

Baumert, J., Kunter, M., Blum, W., Brunner, M., Voss, T., Jordan, A., . . . Tsai, Y.-M. (2010). Teachers’ Mathematical Knowledge, Cognitive Activation in the Classroom, and Pupil Progress. American Educational Research Journal, 47, 133-180. Blömeke, S., Olsen, R. V., & Suhl, U. (2016). Relation of student achievement to the quality of their teachers and instructional quality. In T. Nilsen & J-E. Gustafsson (Eds.), Teacher quality, instructional quality and student outcomes, IEA Research for Education 2. Croninger, R. G., King Rice, J. K., Rathbun, A., & Nishio, M. (2007). Teacher qualifications and early learning: Effects of certification, degree, and experience on first-grade student achievement. Economics of Education Review, 26, 312-324. Darling-Hammond, L. (2014). Strengthening teacher preparation: The holy grail of teacher education. Peabody Journal of Education, 89, 547-561. doi:10.1080/0161956X.2014.93900 Evetts, J. (2011). A new professionalism? Challenges and opportunities. Current Sociology, 59(4), 406-422. doi:doi:10.1177/0011392111402585 Goe, L. (2007). The Link Between Teacher Quality and Student Outcomes: A Research Synthesis. Washington DC. National Comprehensive Center for Teacher Quality. Hanushek, E. A., & Rivkin, S. G. (2012). The distribution of teacher quality and implications for policy. The Annual Review of Economics, 4, 131-157. doi:10.1146/annurev-economics-080511-111001 Hox, J. (2002). Multilevel Analysis - Techniques and Applications. New Jersey: Lawrence Erlbaum Associates. Jackson, C. K., Rockoff, J. E., & Staiger, D. O. (2014). Teacher effects and teacher-related policies. Annual Review of Economics, 6, 801–825. doi:10.1146/annurev-economics-080213-040845 Johansson, S. Myrberg, E., & Rosén, M. (2015). Formal teacher competence and its effect on pupil reading achievement. Scandinavian Journal of Educational Research, 59(5) 564-582. doi:10.1080/00313831.2014.965787 Kleickmann, T., Richter, D., Kunter, M. Elsner, J., Besser, M. Krauss, S., & Baumert, J. (2013). Teachers’ content knowledge and pedagogical content knowledge: The role of structural differences in teacher education. Journal of Teacher Education, 64(1), 90-106. doi:10.1177/0022487112460398 Kyriakides, L., Creemers, B. P. M., Panayiotou, A., Vanlaar, G., Pfeifer, M., Cankar, G., & McMahon, L. (2014). Using student ratings to measure quality of teaching in six European countries. European Journal of Teacher Education, 37(2), 125-143. doi:10.1080/02619768.2014.882311 Muthén, L. K., & Muthén, B. O. (1998-2012). Mplus User's Guide. Los Angeles, CA: Muthén & Muthén. Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How Large Are Teacher Effects? Educational Evaluation and Policy Analysis, 26, 237-257. Snow, C., Griffin, P., & Burns, M. (2005). Knowledge to support the teaching of reading. Preparing teachers for a changing world. The National Academy of Education. San Francisco: Jossey-Bass.

Author Information

Stefan Johansson (presenting / submitting)

Department of Education and Special Education, University of Gothenburg, Sweden

Eva Myrberg (presenting)