Validity issues in assessing teacher competencies

Author(s):

Sandy TautVerónica Santelices

Conference:

ECER 2009

Network:

9. Assessment, Evaluation, Testing and Measurement

Format:

Paper

Session Information

09 SES 01 B, Assessment of Teacher Competencies and Attitudes

Paper Session

Time:

2009-09-28

09:15-10:45

Room:

HG, Marietta- Blau-Saal

Chair:

Nils Berkemeyer

Contribution

Teacher competencies are a key factor in ensuring student learning (Nye, Konstantopoulos & Hedges, 2004; Wenglinsky, 2000). A pioneering, standards-based assessment system of teacher competencies was developed in Chile in 2004 and has since evaluated teacher competencies of approx. 70,000 public school teachers in elementary and secondary schools (Ministry of Education, 2004). The assessment system includes a portfolio (determining 60% of the final score and including a written part and a video-taped lesson), a structured peer interview, a written supervisor assessment and a self-evaluation questionnaire. Teachers’ performance is categorized as either “unsatisfactory,” “basic,” “competent” or “outstanding.” This is a mandatory, high-stakes performance assessment where those teachers who are found to be high-performing are eligible for an increase in salary (merit pay), while low-performing teachers are subject to professional development, and – if evaluated “unsatisfactory” in three consecutive years – loss of employment (for more details see Avalos & Assael, 2007, as well as Manzi, Preiss, Gonzalez, Flotts & Sun, 2008). The task of our research team has been to develop a research agenda regarding the validity and reliability of this national assessment system. One of the most authoritative sources with regard to validation research is the Standards for Educational and Psychological Research (Joint Committee on Standards for Educational and Psychological Testing of AERA, APA, & NCME, 1999). These Standards define validity as “the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests” (p. 9). The Standards agree with Kane (2001), Messick (1994) and others that validation must focus on the proposed interpretation of test scores. The Standards differentiate types of validity evidence, based on content, response process, internal structure, relationships with other variables, and consequences. After four years of implementing the validity research agenda guided by the above-mentioned AERA, APA & NCME Standards, we are in a position to summarize our theoretical insights and empirical findings. The purpose of this paper is thus (a) to present our research agenda and its theoretical foundations, including current developments in validity research (e.g., Sireci, 2008), and (b) to give an overview of the empirical findings we have amassed regarding the Chilean teacher assessment system. The paper will be of interest for European researchers involved in teacher performance assessment (Tigelaar, Dolmans, Wolfhagen & van der Vleuten, 2004; Baumert & Kunter, 2006), as well as validation research of assessment systems in general (Atkinson et al., 2004).

Method

The empirical findings gleaned from our validity (and reliability) research are based on a wide range of both quantitative and qualitative methods: (a) investigating rater effect in the portfolio rating process by applying generalizability theory, as well as performing standard reliability calculations, e.g. internal consistency and inter-rater reliability, (b) examining the portfolio’s internal structure using factor analysis, (c) examining content coverage by the assessment instruments of the teaching performance standards, (d) construct validation based on an in-depth study of a sample of 60 previously assessed teachers, applying classroom observations, standardized teacher content knowledge testing, standardized student achievement testing (gain scores), and expert assessment of a teaching materials binder (see Bond, Smith, Baker & Hattie, 2000), and (e) investigating the consequences of the assessment system, again using a range of methods (see Fast & Hebbler, with ASR-CAS Joint Study Group on Validity in Accountability Systems, 2004; Lane & Stone, 2002).

Expected Outcomes

In a nutshell, the findings based on content- and construct-based evidence support the validity of the Chilean teacher assessment. Specifically, the in-depth study of teaching competencies of 60 teachers found statistically and practically significant differences between teachers pertaining to the two extreme performance categories (see Taut & Santelices, 2007). Regarding the generalizability of the portfolio ratings and the internal structure of the portfolio, our findings suggest points of improvement for this assessment system. Furthermore, preliminary evidence from the consequential validity study indicates that at the local (municipal) level the assessment results are used to inform professional development initiatives but do not inform incentive policies to the extent that was intended, and that there are a range of positive and negative unintended consequences (Santelices, Taut, Araya & Manzi, 2009).

References

Atkinson, A., Burgess, S., Croxson, B., Gregg, P., Propper, C., Slater, H., & Wilson, D. (2004). Evaluating the Impact of Performance-related pay for teachers in England. CMPO Working Paper Series No. 04/113. Centre for Market and Public Organisation, Department of Economics, University of Bristol. Avalos, B., & Assael, J. (2007). Moving from Resistance to Agreement: The Case of the Chilean Teacher Performance Evaluation. International Journal of Educational Research, 45, 254-266. Baumert, J., & Kunter, M. (2006). Stichwort: Professionelle Kompetenz von Lehrkräften. Zeitschrift für Erziehungswissenschaft, 9(4), 469-520. Bond, L., Smith, T., Baker, W., & Hattie, J. (2000). The Certification System of the National Board for Professional Teaching Standards: A construct and consequential validity study. Greensboro, NC: Center for Educational Research and Evaluation, University of North Carolina. Fast, E.F, & Hebbler, S., with ASR-CAS Joint Study Group on Validity in Accountability Systems (2004). A framework for examining validity in state accountability systems. Documento de la serie: Implementing the State Accountability System Requirements Under the No Child Left Behind Act of 2001 (NCLB). Washington, D.C.: Council of Chief State School Officers. Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, D.C.: American Educational Research Association. Kane, M. T. (2001). Current Concerns in Validity Theory. Journal of Educational Measurement, 38(4), 319-342. Lane, S., & Stone, C.A. (2002). Strategies for examining the consequences of assessment and accountability programs. Educational Measurement: Issues and Practice, 21(1), 23-30. Manzi, J., Preiss, D., Gonzalez, R., Flotts, P. & Sun, Y. (2008). Design and Implementation of a National Project of Teaching Assessment: The Chilean Experience. Paper presented at the annual meeting of the American Educational Research Association, March 24-28, 2008, New York City, USA. Messick, S. (1994). The Interplay of Evidence and Consequences in the Validation of Performance Assessments. Educational Researcher, 23(2), 13-23. Ministry of Education (2004). Marco Para La Buena Enseñanza [Guidelines for Good Teaching]. Santiago: Ministerio de Educación. Nye, B., Konstantopoulos, S., & Hedges, L. (2004). How Large Are Teacher Effects? Educational Evaluation and Policy Analysis, 26(3), 237-257. Santelices, V., Taut, S., Araya, C., & Manzi, J. (2009). Consequential Validity of Chile’s Teacher Evaluation System: Consequences at the Municipal (Local) Level. Paper accepted for presentation at the Annual Meeting of the American Educational Research Association, April 13-19, 2009, in San Diego, USA. Sireci, S. (2008). Packing and Unpacking Sources of Validity Evidence: History Repeats Itself. Presentation at the 9th Annual Maryland Assessment Conference, October 9-10, 2008, in Maryland, USA. Available as of January 22, 2009 under: http://www.education.umd.edu/EDMS/MARCES/conference/validity/agenda.htm Taut, S., & Santelices, V. (2007). Validating the Chilean National Teacher Evaluation System: A comprehensive research agenda. Paper presented at the Annual Meeting of the American Educational Research Association, April 9-13, 2007, in Chicago, USA. Tigelaar, D., Dolmans, D., Wolfhagen, I., & van der Vleuten, C. (2004). The development and validation of a framework for teaching competencies in higher education. Higher Education, 48(2), 253-268. Wenglinsky, H. (2000). How Teaching Matters. Bringing the Classroom Back Into Discussions of Teacher Quality. Princeton, NJ: Educational Testing Service and Milken Foundation.

Author Information

Sandy Taut

MIDE UC, Pontificia Universidad Católica de Chile

Department of Psychology, Measurement Center MIDE UC

Santiago

Verónica Santelices

MIDE UC, Pontificia Universidad Católica de Chile, Chile

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.