Development Of An Assessment-Center For Trainee Teachers: The PArcours
Author(s):
Martina Hechinger (presenting / submitting)
Conference:
ECER 2014
Format:
Paper

Session Information

09 SES 06 C, Competencies and Attitudes of Teachers (Part 2)

Paper Session: continued from 09 SES 05 C

Time:
2014-09-03
15:30-17:00
Room:
B009 Anfiteatro
Chair:
Åse Hansson

Contribution

Aptitude assessments for students, who intend to study teaching, are not common in Germany. In general, every student, who holds a university-entrance diploma, is permitted to enter teacher training programs. (However, there are examination tests for candidates who would like to study art, sport, and music education.) There are no qualification examinations that test significant competencies for studying teaching, and so it is arguably the case that some or even many students are not fully qualified to become teachers. Relating to the relevance of aptitude assessments, the question arises as to which competencies are significant for teaching.

Competencies can be defined as “(…) the specialized knowledge one has acquired that support cognitive performance (…)” (Mayer, 2003, as cited in Klieme & Hartig, 2007, p. 17). In several research studies, specific competencies have been identified as being essential for studying teaching and teaching. The Austrian researcher Johannes Mayr (2012), for instance, investigates several predictors of probation for studying and teaching. He finds out that students, who choose the profession because they are intrinsically motivated, are more successful than other students. Furthermore, their interests are important predictors; especially social interests and job-specific interests, these are essential for success at university and school. Besides that, the personality dimensions conscientiousness and extraversion predict the students´ success; moreover, mastery of speech is identified as being significant. Other scientists emphasize the importance of social competencies, such as communication skills, empathy, or the capacity of teamwork (e.g., Liekam, 2004; Sliwka & Klopsch, 2012). Aptitude assessments should measure these competencies and personality dimensions. It should be tested whether trainee teachers feature the competencies that are needed and to what extent the competencies exist.

Some aptitude procedures for trainee teachers were developed, but most of them use the self-evaluation method (e.g., FIBEL – Self-Assessment at the University of Münster/Germany; CCT – Career Counselling for Teachers/Austria, Germany, and Switzerland). Since self-evaluation procedures are influenced by the social desirability bias, the information gained cannot be used for decision making. Therefore, an aptitude procedure that uses external evaluation is needed.

When developing a procedure, the three test quality criteria are of focus: objectivity, reliability and validity. Objectivity refers to the extent to which the test or scale results are independent from the test or scale users´ feelings or beliefs (e.g., Bortz & Döring, 2006, p. 195). According to Christensen (2004, pp. 182-184), reliability refers to consistency, stability, or the repeatability of test or scale results. The scientist also states that reliability is directly related to the amount of measurement error which refers to the inaccuracy that is introduced into the measurement. Validity is the third and most important test quality criterion. A test or scale can be claimed valid if it measures what it purports to measure (e.g., Borsboom, Mellenbergh & Van Heerden, 2004, p 1061). Scale validity evidence comes from a multitude of sources. Every validation process should include an examination of content, construct, factorial, and criterion-related validity. Another important aspect is to analyze cognitive processes of the candidates when answering scale items. Furthermore, context analyses have to be conducted in order to test the influence of personal, situational and cultural characteristics on the scale scores (e.g., Hechinger, in press).

 

The presentation gives an introduction into a new aptitude procedure. The methods that are used to assess the trainee teachers´ competencies will be presented. Furthermore, I will discuss the results of examining objectivity, reliability and validity. It should be noted that there are some other European countries, such as Finland, which have established aptitude assessment-centers for trainee teacher candidates. It is an international topic, and sharing ideas promotes the research object.

Method

In Germany, at the University of Passau, an aptitude assessment-center for the evaluation of students who intend to study teaching has been developed. It is a one-day assessment that is offered several days, twice a year. The assessment-center is called PArcours. The purpose is to measure students’ competencies and characteristics. Competence differs from performance, as the former cannot be directly observed and arises through the accomplishment of tasks (Klieme & Hartig, 2007, pp. 13, 16), and so tasks are required in order to measure competencies. The assessment-center PArcours contains three tasks that are performed by trainee teacher candidates. The first task is self-presentation, where the candidates describe their qualifying characteristics, as well as their motives for choosing teacher education. In the second task, called group discussion, every candidate is requested to evaluate diverse teacher requirements alone, and afterwards, the evaluation results are discussed within a group. The third task is film analysis, where the candidates are shown a film of a conflict between a teacher and a pupil. Then the candidates depict the conflict situation and offer strategies to solve and prevent repetition of the conflict. The purposes of these tasks are, firstly to prompt the candidates to reflect on their personal characteristics and skills related to teaching requirements, secondly to face them with a complex school situation, and overall, to initiate activity in order to make the competencies observable. Scales for assessing competencies and personal characteristics have been developed, such as intrinsic motivation, presentation skills, mastery of speech, communication skills, and the ability to estimate and cooperate. These scales are used by experts who observe the candidates´ performance, so that external evaluation is used in PArcours. The scales are measured by between two and seven items, each having a 4-point Likert-type response format (“completely applicable” to “completely inapplicable”). It can be possible that the observer is briefly inattentive or an item does not appear, wherefore the additional category n.o. is used which means “not observed” or “not observable in the specific situation”. The scale scores are statistically evaluated and interpreted, using text modules. The candidates are given feedback based on the test results. They get advice whether the observed competencies and characteristics are compatible with the requirements for teacher education.

Expected Outcomes

The scales used in PArcours were examined for objectivity, reliability and validity, and the following results were obtained: Objectivity can be gained by item specifications and standardization of the assessment. Furthermore, objectivity is tested by comparing scores within a scale allocated by several scale users. In PArcours, every candidate is rated by at least two observers. Equivalence between the scores of one scale is tested, and some items are modified as a consequence of inconsistency. Cronbach´s Alpha coefficient estimates reliability, and the coefficients measured for the PArcours scales range between .64 und .83. Thus, internal consistency is satisfying for most scales, but it is essential to improve some scale items on the basis of the reliability analyses results. Another method, which can be used to determine the scales´ reliability, is the test-retest method. It compares the measures of the same people on two different occasions (e.g., Christensen, 2004, pp. 184f). Future research on the assessment-center PArcours will include the test-retest method as an addition method to prove reliability. In PArcours, some measurement errors and effects influence the scale results, for instance errors of central tendency, leniency-severity errors, the halo effect, and the primacy effect. It is important to mention that the error of central tendency and the halo effect are always statistically tested. Also, that the observers are being taught how to avoid measurement errors. The scale results are affected by some personal and situational characteristics. Previous knowledge about the procedure, nervousness of the candidates, as well as noise are important to monitor. The items have been developed on the basis of detailed definitions of the tested competencies in order to achieve content validity. In addition to scale validation, the whole assessment-center PArcours has to be proven as being valid, which will be a major aspect of future studies.

References

Borsboom, D., Mellenbergh, G. J. & Van Heerden, J. (2004). The Concept of Validity. Psychological Review, Vol. 111, No. 4, 1061-1071. Bortz, J. & Döring, N. (2006). Forschungsmethoden und Evaluation für Human- und Sozialwissenschaftler [Research method and evaluation for human and social scientists] (4th ed.). Heidelberg: Springer Medizin Verlag. CCT – Career Counselling for Teachers (Contact person: J. Mayr). Available on: http://www.cct-austria.at/ [Date of access: 25.01.14]. Christensen, L. B. (2004). Experimental Methodology (9th ed.). Boston i.a.: Pearson. FIBEL – Self-Assessment at the University of Münster (Project management: W. Böttcher & U. P. Kanning). Available on: http://egora.uni-muenster.de/ew/qe/sp_auto_63209.shtml [Date of access: 25.01.14]. Hechinger, M. (in press). Validierung in den Sozialwissenschaften. Die Konzeption eines Validierungsmodells zur Überprüfung der Validität von Skalen am Beispiel der Flow-Theorie [Validation in the social sciences. Conception of a validation model to examine scale validity by using the flow theory]. Bad Heilbrunn: Klinkhardt. Klieme, E. & Hartig, J. (2007). Kompetenzkonzepte in den Sozialwissenschaften und im erziehungswissenschaftlichen Diskurs [The concept of competence in social and educational sciences] [Electronic Version]. In M. Prenzel, I. Gogolin & H.-H. Krüger (Eds.), Kompetenzdiagnostik. Zeitschrift für Erziehungswissenschaft, Sonderheft 8, 11-29. Liekam, S. (2004). Empathie als Fundament pädagogischer Professionalität. Analysen zu einer vergessenen Schlüsselvariable der Pädagogik [Empathy as a fundament in the pedagogical professionalism. Analyses of a forgotten pedagogical key variable][WWW Document]. Unpublished dissertation. Ludwig-Maximilians-Universität München. Available on: http://edoc.ub.uni-muenchen.de/2514/1/Liekam_Stefan.pdf [Date of access: 31.01.14]. Mayr, J. (2012). Ein Lehramtsstudium beginnen? Ein Lehramtsstudium beginnen lassen? Laufbahnberatung und Bewerberauswahl konstruktiv gestalten [Starting teacher education? Let the teacher education start? Career counseling and candidate selection constructively organized]. In B. Weyand, M. Justus & M. Schratz (Eds.), Auf unsere Lehrerinnen und Lehrer kommt es an. Geeignete Lehrer/-innen gewinnen, (aus-)bilden und fördern (pp. 38-57). Essen: Stifterverband. Sliwka, A. & Klopsch, B. (2012). Auf den Lehrer/die Lehrerin kommt es an. Lehrerbildung und Lehrerprofessionalität in internationaler Perspektive [The teacher matters. Teacher education and teacher professionalism in an international perspective]. In B. Weyand, M. Justus & M. Schratz (Eds.), Auf unsere Lehrerinnen und Lehrer kommt es an. Geeignete Lehrer/-innen gewinnen, (aus-)bilden und fördern (pp. 14-35). Essen: Stifterverband.

Author Information

Martina Hechinger (presenting / submitting)
University of Passau
Passau

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.