A Factor Analytic Study of the “Intern Keys” Pre-Service Teachers’ Performance Assessment Tool

Author(s):

Atakan Ata(presenting / submitting)Kadir Kozan(presenting)

Conference:

ECER 2017

Network:

10. Teacher Education Research

Format:

Paper

Session Information

10 SES 05 B, Programmes and Approaches: Assessments

Paper Session

Time:

2017-08-23

13:30-15:00

Room:

K5.20

Chair:

Bruno Leutwyler

Contribution

Introduction

In recent years, teacher education programs at higher education institutions have begun adopting a practice-based approach (McDonald, Kazemi, & Kavanagh, 2013). For instance, most states in the United States have adopted some teaching standards for teacher performance assessment recommended by the Interstate Teacher Assessment and Support Consortium, and these standards are also being used for pre-service teachers’ performance evaluation. In 2016, Turkish Department of Education also created some teacher observation criteria for teacher performance evaluation purposes, and teacher education programs at universities are closely following the developments, and adjusting their curricula accordingly.

Even though practice-based approach in teacher education is not a new idea, it has drawn more attention lately - again, mostly because some novice teachers feel that they are not ready to teach even after graduation, or first-year teachers face more problems compared to experienced teachers (Rust, 1994). As teacher education programs are aligning their regulations with teacher performance evaluation systems, and basing their courses on instructional performance, assessment of pre-service teachers’ performance has become an issue. Assessing pre-service teachers’ instructional performance is not easy, mostly because they are not teachers yet; but, most of the evaluation criteria are adapted from teacher evaluation forms which include such items as communication with students’ parents. It is also not easy because the number of observations a pre-service teacher would get might not reflect the reality. Teacher education programs must take equity into account considering different observers involved in the process as well. Therefore, adopting a comprehensive assessment tool that truly assess instructional performance, and that minimizes personal bias is essential for faculties of education.

Intern Keys is one of the assessment tools that is used for assessing pre-service teachers’ instructional performance. Intern Keys is described to be “not an observation checklist,” but a collection of standards that a pre-service teacher should meet (Elder, Ata, & Cramer, 2016, p. 2). The Intern Keys validation project is being conducted in the State of Georgia (USA) and was funded through a grant from the Georgia Network for Transforming Educator Preparation and the Council of Chief State School Officers.

Research Question and Objectives

The current research investigated the construct validity of the Intern Keys survey which was developed to be used by faculty supervisors to assess the teaching effectiveness or performance of their advisees. Therefore, this study addressed the following research question:

Does the teacher candidate effectiveness assessment tool have an interpretable factor structure that aligns with theoretical assumptions?

Method

Methods The research data were collected from 117 faculty members working at different universities across the US. All the faculty members were working as teacher educators at the time of data collection. Specifically, the participants were working as faculty supervisors of more than 500 different pre-service teacher candidates. MacCallum, Widaman, Zhang, and Hong (1999) stated that a sample of 100–200 participants is enough with communalities of ≥. 5 and well-determined factors, which held true in the current study. Both initial and extraction communalities were higher than .5 and they ranged from .527 to .738. The authors further stated that “common rules of thumb regarding sample size in factor analysis are not valid or useful” (p. 96). In addition, after checking the relevant assumptions ranging from normality to multivariate cases, exploratory factor analysis with oblimin rotation served the analyze the research data. Furthermore, the teacher candidate effectiveness assessment tool used to collect data included 10 main standards having 6-10 criteria to measure under each main standard. The instrument is very similar to Stronge Teacher Effectiveness Performance Evaluation System (2012) which is widely used by many school districts in the United States, and used in different effectiveness projects before. The instrument had a scale ranging from 4 (exemplary) to 1 (ineffective). The faculty supervisors were emailed to complete the survey after the candidates were finished with their school experiences. Originally, there were 296 numbers of surveys collected but due to the lack of names or other identifying information and a random elimination of duplicate cases (each supervisor completed the survey for more than one candidate), 117 of them were analyzed here.

Expected Outcomes

Expected outcomes The initial descriptive analyses showed that, on average, participants ratings were quite high ranging from a 2.90 (SD = .53) to 3.22 (SD = .54) given that the scale had a maximum point of 4. This overall descriptive insight suggests that the candidates were successful at their teaching practice. An initial exploratory factor analysis (EFA) using principal axis factoring was run to check sampling and data adequacy. The correlation matrix showed large correlation values bigger than .30 , which was suitable for factor analysis purposes. Bartlett's test of sphericity, χ2 (45.0) = 907.66, p < .001, revealed that correlations among the variables were large enough for an EFA. The Kaiser–Meyer–Olkin (KMO) indicated that sampling was suitable for the present analyses, KMO = .91 which is bigger than the suggested minimum values of .5 (Field, 2009) and .6 (Tabachnick & Fidell, 2013). There was only one eigenvalue higher than 1 thereby referring to a one-factor structure for the ten items. Similarly, factor matrix produced one single factor, and it was able to explain approximately 63 % of variance in the data. Further, the scree plot showed a leveling off at the second factor, which suggested a one factor. Consequently, a final EFA with one single factor extracted under the condition of a minimum factor loading of .40. was employed. All factors loaded positively and significantly on the factor ranging from .726 to .834. The second item showed the highest loading while the fourth item showed the smallest one. These findings suggest that the construct validity of the instrument is high measuring one factor that can be called “teaching effectiveness”.

References

References Elder, T., Ata, A., & Cramer, S. E. (2016). An Evaluation of the Validity and Reliability of the Intern Keys Assessment. Unpublished report. Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: SAGE Publications. MacCallum, R. C., Widaman, K. F., Zhang, S., & Hong, S. (1999). Sample size in factor analysis. Psychological Methods, 4(1), 84–99. McDonald, M., Kazemi, E., & Kavanagh, S. S. (2013). Core practices and pedagogies of teacher education a call for a common language and collective activity. Journal of Teacher Education, 64(5), 378-386. Rust, F.O. (1994). The first year of teaching: It’s not what they expected. Teaching & Teacher Education, 10(2), 205-217. Stronge, J. H. (2012). Teacher Effectiveness Performance Evaluation System. Stronge & Associates. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston: Pearson.

Author Information

Atakan Ata (presenting / submitting)

Bahcesehir University, Turkey

Kadir Kozan (presenting)