International Comparisons of Student Perceptions on Teacher’s Classroom Management: Improving Comparability with the Anchoring Vignettes Method

Author(s):

Hana Vonkova(presenting / submitting)Gema Zamarro Vera DeBerg

Conference:

ECER 2015

Network:

09. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 12 D, Assessing Students' and Teachers' Roles and Perceptions

Paper Session

Time:

2015-09-11

09:00-10:30

Room:

3002. [Main]

Chair:

Jelena Radišic

Contribution

Teacher quality is a key element of student academic success and improving it is a major concern among parents, educators, and policy makers around the world. One of the main and most studied methods to estimate teacher quality is the so called value-added methods, which estimate a teacher’s contributions to student performance based on test scores. Several concerns have been raised with the use of value-added teacher quality measures, including their limited reliability in discriminating among teachers in a specific year (see Glazerman et al, 2011; Kane & Staiger, 2008) and the fact that not all subjects are covered by state tests, making it impossible to have value-added teacher quality estimates for subjects not covered. As a result, researchers and policymakers are increasingly interested on alternative measures of teacher quality to be used in combination with value-added measures. A candidate for such measures are student perception surveys on teacher practices (see, e.g., Berk, 2005; Bill & Melinda Gates Foundation, 2013; Glazerman et al., 2011).

Students are first hand witnesses of what occurs in the classrooms and so they are in a privileged position to provide feedback on teaching practices. Research on the reliability of student perceptions on teacher effectiveness via self-administered questionnaires is, although limited, encouraging. The Measures of Effective Teaching (MET) project in the U.S. found that teacher quality measures based on student self-reports are predictive of student achievement gains (see Bill & Melinda Gates Foundation, 2012). Although this is a significant and important result, more research is needed to get a better understanding of the properties and comparability of teacher quality measures based on student self-reports.

One major concern with self-assessments is that they may depend not only on the objective situation but also on respondents’ use and interpretation of the different scales in the provided questions. The anchoring vignette method was introduced in the social sciences by King et al. (2004) precisely to adjust for such heterogeneity in reporting behavior and obtain comparable responses. Since its introduction, the anchoring vignette method has been largely used in social-science research in areas such as health, work disability, life satisfaction, job satisfaction and satisfaction with contacts (see Peracchi & Rossetti, 2012; Vonkova & Hullegie, 2011; Bago d’Uva et al., 2008, Angelini, Cavapozzi, & Paccagnela, 2011; Kapteyn, Smith, & van Soest, 2007; Angelini, Cavapozzi, Corazzini, & Paccagnella, 2012; Kapteyn, Smith, & van Soest, 2010, Kristensen & Johansson, 2008, Bonsang & van Soest, 2012). However, studies that make use of the anchoring vignette method in education are rare. Exceptions include Buckley & Schneider (2007) and Buckley (2008) who used anchoring vignettes for the comparison of parents' satisfaction measures in charter and public schools.

In this paper, we use data from PISA 2012, which includes students self-reports and anchoring vignettes for an important dimension of teacher quality: teacher’s classroom management. Teacher’s classroom management is one of the dimensions often measured in student perception surveys and it was found to be one of the most predictive of student achievement gains (Bill and Melinda Gates foundation, 2012). Specifically, we study a) the heterogeneity in student’s assessments of teacher’s classroom management reporting behavior across countries, b) the use of the anchoring vignettes method for comparison of measures of teacher’s classroom management level across countries before and after adjustment for the heterogeneity in reporting behavior, and c) possible sources for such heterogeneity in reporting behavior. As far as we are aware, this is a first study of the use of the parametric model of the anchoring vignettes method to adjust student perceptions on a dimension of teacher quality and its potential consequences for comparisons across countries.

Method

Responses to self-assessment questions in surveys may reflect not only the objective situations we seek to measure but also differences in scale usage. The anchoring vignette method has been proposed to avoid misleading results caused by differences in respondents' scale usage. The method uses additional information to reveal the potential differences – these being evaluations of hypothetical persons described in short stories, called anchoring vignettes. The differences between the respondents' vignette evaluations may be interpreted as the differences in their scale usage since the respondents evaluate the same stories. This information is then used to adjust self-assessments for the heterogeneity in reporting behavior. This paper focuses on the study of teacher quality measures based on student’s assessments on teacher classroom behavior. PISA 2012 included four scale items that measure the classroom management of students’ mathematics teachers. Specifically, the PISA 2012 student questionnaire asked students to use the four-point scale to self-report the extent they agree with the following four statements: a) my teacher gets students to listen to him or her, b) my teacher keeps the class orderly, c) my teacher starts lessons on time, and d) the teacher has to wait a long time for students to quiet down. While these questions aim to measure students’ perceptions of teacher classroom management, comparing these raw percentages can be troublesome. As previously mentioned, this is because any differences in these ratings may be due to differences in how students interpret the rating scale, rather than actual differences in teacher classroom management. In order to be able to correct for discrepancies on reporting behavior, the PISA 2012 study included three hypothetical scenarios (vignettes) related to teacher’s classroom management. Students were asked to rate to what extent they agree that the teacher is in control of his/her classroom: Vignette 1 (High level): The students’ in Ms. class are calm and orderly. She always arrives on time to class. Ms. is in control of her classroom. Vignette 2 (Medium level): The students’ in Ms. class frequently interrupt her lessons. She always arrives five minutes early to class. Ms. is in control of her classroom. Vignette 3 (Low level): The students’ in Mr. class frequently interrupt his lessons. As a result, he often arrives five minutes late to class. Mr. is in control of his classroom.

Expected Outcomes

We show that the differences between adjusted and unadjusted distributions of actual classroom management in countries are big. It reflects the big differences in the scale usage. Countries like Norway, the Netherlands, Shanghai-China, New Zealand, United Kingdom, Florida (USA), Australia, Germany and the Czech Republic highly improve their position among countries after the adjustment of the scale usage heterogeneity. It suggests that they have high standards (perceive a given classroom management level as worse compared to other countries) and their actual classroom management is better than they report. In contrary, there are countries which position highly worsens after the adjustment. These countries are, for example, Indonesia, Jordan, Qatar, Thailand, Malaysia, Romania, United Arab Emirates, Bulgaria, Brazil and Turkey. Using a word map, we demonstrate that the scale usage is geographically related – for example, some North and Central European countries have high standards while some Southeast Asia countries have low standards when evaluating the classroom management. We also focus on possible sources of the reporting heterogeneity. The correlations between the classroom management level and external variables like public expenditure per student highly increase after the adjustment for the heterogeneity in reporting behavior. We conclude, that the anchoring vignette method shows a good potential to enhance the validity and international comparability of measurement in education.

References

Angelini, V., Cavapozzi, D., & Paccagnella, O. (2011). Dynamics of reporting work disability in Europe. Journal of the Royal Statistical Society A,174(3), 621-638. Angelini, V., Cavapozzi, D., Corazzini, L., & Paccagnella, O. (2012). Age, health and life satisfaction among older Europeans. Social Indicators Research, 105(2), 293-308. Bago d’Uva, T., van Doorslaer, E., Lindeboom, M., & O’Donnell, O. (2008). Does reporting heterogeneity bias the measurement of health disparities? Health Economics, 17(3), 351-375. Berk, R. A. (2005). Survey of 12 strategies to measure teaching effectiveness. International Journal of Teaching and Learning in Higher Education, 17(1), 48-62. Bill and Melinda Gates foundation (2012). Asking students about teaching: Student perception surveys and their implementation. MET project Policy and Practice Brief. Bill and Melinda Gates foundation (2013). Ensuring fair and reliable measures of effective teaching: Culminating findings from the MET project’s three-year study. MET project Policy and Practice Brief. Bonsang, E., & van Soest, A. (2012). Satisfaction with social contacts of older Europeans. Social Indicators Research, 105(2), 273-292. Buckley, J. (2008). Survey Context Effects in Anchoring Vignettes. New York University. Retrieved from: http://polmeth.wustl.edu/media/Paper/surveyartifacts.pdf Buckley, J., & Schneider, M. (2007). Charter schools: Hope or hype? United Kingdom: Princetown University Press. Glazerman, S., Goldhaber, D., Loeb, S., Staiger, D., Raudenbush, S., & Whitehurst, G. J. (2011). Passing muster: Evaluating teacher evaluation systems. Washington, DC: The Brookings Institution. Kane, T. J., & Staiger, D. O. (2005). Using imperfect information to identify effective teachers (pp. 61). Cambridge, MA: National Bureau of Economic Research. Kapteyn, A., Smith, J. P., & van Soest, A. (2007). Vignettes and self-reports of work disability in the US and the Netherlands. American Economic Review, 97(1), 461-473. Kapteyn, A., Smith, J. P., & van Soest, A. (2010). Life satisfaction. In E. Diener, J. E. Helliwell, & D. Kahneman (Eds.), International differences in well-being (pp. 70–104). Oxford: Oxford University Press. King, G., Murray, C., Salomon, J., & Tandon, A. (2004). Enhancing the validity and cross-cultural comparability of measurement in survey research. American Political Science Review, 98(1), 567-583. Kristensen, N., & Johansson, E. (2008). New evidence on cross-country differences in job satisfaction using anchoring vignettes. Labour Economics, 15, 96–117. Peracchi F., & Rossetti C. (2012). Heterogeneity in health responses and anchoring vignettes. Empirical Economics, 42(2), 513-538.

Author Information

Hana Vonkova (presenting / submitting)

Charles University in Prague

Faculty of Education

Prague

Gema Zamarro

University of Arkansas, USA

Vera DeBerg