Session Information
09 SES 14 B, School Evaluations
Paper Session
Contribution
Peer review is a partly widespread form of external evaluation of educational organizations in Europe. Several international projects focusing on the use of peer-review (e.g. four Leonardo da Vinci development projects: Peer Review in Initial VET, Peer Review Extended, Peer Review Extended II, Peer Review Impact). Similarly, in the years 2010-2011 at the national level in the Czech Republic, the project "Road to Quality Improvement" was supported by 32 schools with peer review. The practical implementation of the Peer Review has proved to be a determining competence for evaluating the participants. This article addresses the question: What competencies do peers apply? The paper provides a critical assessment of peer review competencies of peers based on data.
Theoretical framework
The peer review is framed by the requirements of the quality of education (e.g. Ross and Genevois, 2006), efficiency (see Scheerens, 2004), accountability (Elliott et al., 1981, Elmore, 2005). In this context, it was found that school pressure is the most appropriate combination of pressure on schools and the promotion of their efforts to improve work from within (Fullan, 2001; Leithwood, 2001, etc.). Schools respond to the above-mentioned requirements and use different tools, techniques, methods that allow them to cope with self-control and self-assessment through the process of self-evaluation (e.g. Ravnmark, 2003; Bírzea et al., 2005; MacBeath et al., 2000; McNamara and O'Hara, 2008; Leithwood et al., 2002, etc.), the concept of peer review (see Goldstein, 2004; Speer, 2007; Gutknecht-Gmeiner, 2008), when the school is heading to the ideal of a learning organization (Senge, 1990).
Evert Vedung (2009, p. 305) defines peer review as an assessment carried out by persons at the same level, but usually leading colleagues from the same profession. Maria Gutknecht-Gmeiner (2008, pp. 28-29) describes that peer review is "a form of external evaluation to support the institution that undergoes this process in its quality assurance and development efforts; the peer review process is carried out by a group of external experts and experts, the so-called peers, who are invited to assess the quality of the different areas and fields of the educational institution; the central element of peer review is usually on-site visit through peer-to-peer processes; such as the quality of teaching, and / or other specific questions in individual fields of expertise or throughout the organization; evaluation is usually done on the basis of an self-evaluation report.” "Peer" is then the person who: a) is equivalent to the person(s) whose performance is being reviewed; b) works in the same or similar field and / or in a similar institution; c) mostly comes from some other institution, is therefore an outsider; d) Has specific professional expertise and experience (professional competence, understanding, values, language ...), thus bringing the "inside knowledge" into the process by associating it with an external perspective ("external insider").
Method
This study has been conducted as part of a larger data collection. Through various research methods, an effort was made to identify the competences applied by peers through peer review. These were questionnaire survey for peers, content analysis of 27 evaluation reports, and 27 peer review facilitators' peer review sheets, peer review observation at 4 vocational schools, semi-structured interviews with 4 representatives of vocational schools, and focus group with peer review facilitators. The 32 schools (26 elementary schools, 2 grammar schools and 4 vocational schools) participated in the 28 peer review (24 was reciprocal - these schools have experienced both the role of evaluators and the role of evaluated). The 128 school staff took part in these events (43 peer review took part once and 85 twice in the role of evaluated school, but also as evaluators). 53% of the participants were members of the school management (the head, deputy head), 17% were teaching staff coordinating other activities in the school (e.g. the head of the youth home, the head of vocational training, the educational counselor, the school curriculum coordinator, coordinator of ICT) and 30% were other teachers. Evaluation reports from peer review have been the subject of content analyzes. Peer review participants were asked to complete anonymous questionnaire survey. All five facilitators submitted a written peer review report to the project management. These reports have also been the subject of content analyzes. After the end of all the events, a focus group was organized with the facilitators, attended by representatives of the Road to Quality Improvement project. The facilitator's role consisted of expert assistance in preparing the peer review, was available during the peer review, moderated the evaluation visit, and helped with the processing of the evaluation report.
Expected Outcomes
Overall, the findings have shown that when assessing peers’ competencies, it is necessary to keep in mind all the peer review phases, i.e. the planning of the evaluation visit, its implementation, the development of final report, the implementation of the suggestions at the evaluated school. All these phases demonstrated communication skills, team collaboration skills, organizational skills and evaluation competencies of the participants. The ability of teamwork has emerged during an evaluation visit. Here was how the team of evaluators together communicated how the individual evaluators were close to each other, as they had the position and power in the team. During the evaluation visit, evaluators had to have organizational skills (e.g. time management, solution of agreed topics) as well as communication competencies. The evaluation competencies of the evaluators is a separate topic: - On what basis did the evaluators establish the criteria for the area evaluated? - When data collection methods were not used appropriately, did this lead to inadequate or wrong interpretations and conclusions of the evaluators? - Did peers use triangulation? - Have the evaluators always had enough experts on different areas of evaluation? - Is the exchange of experience a valuation or should it be a critical assessment of the current situation? - The question is whether the evaluation reports are based on facts and records or on impressions and fictions? It is an unresolved question, whether the peers would in most cases meet the Standards for Evaluation of Professional Associations Standards for Educational Evaluation (Yarbrough et al., 2011).
References
Bírzea, C., Cecchini, M., Harrison, C., Krek, J., Spajič-Vrkaš, V. (2005) Tool for Quality Assurance of Education for Democratic Citizenship in Schools. Paris: UNESCO, Council of Europe, CEPS. Elliot, J., Bridges, D., Ebbutt, D., Gibson, R., Nias, J. (1981). School Accontability. London: Grant McIntyre Limited. Ellmore, R. F. (2005). Accontable Leadership. The Educational Forum, 69 (2), pp. 134-142. Fullan, M. (2001). The Meaning of Educational Change. New York – London: Teachers College Press – Routledge Falmer. Goldstein, J. (2004). Making Sense of Distributed Leadership: The Case of Peer Assistance and Review. Educational Evaluation and Policy Analysis, 26 (2), pp. 173-197. Gutknecht-Gmeiner, M. (2008) Externe Evaluierung durch Peer Review. Qualitätssicherung und entwicklung in der beruflichen Erstausbildung. Wiesbaden: VS Verlag. Leithwood, K. (2001). School leadership in the context of accountability policies. International Journal of Leadership in Education, 4 (3), pp. 217-235. Leithwood, K., Jantzi, D., Steinbach, R. (2002). Leadership Practices for Accountable Schools. In K. Leithwood, P. Hallinger (Eds.), Second International Handbook of Educational Leadership and Administration. Dordrecht: Kluwer Academic Publishers, pp. 849 – 879. MacBeath, J., Schratz, M., Meuret, D. and Jakobsen, L. (2000) Self-Evaluation in European Schools: A Story of Change. New York– London: Routledge. McNamara, G., O´Hara, J. (2008). The importance of the concept of self-evaluation in the changing landscape of education policy. Studies in Educational Evaluation, 34 (3), pp. 173-179. Ravnmark, L. L. (2003). An European Guide on Self-assessment for VET-providers. Thessaloniki: CEDEFOP. Ross, K. N., Genevois, I. J. (2006). Cross national studies of the quality of education: planning their design and managing their impact. Paris: UNESCO. Scheerens, J. (2004). Review of school and instructional effectiveness research. In Paper commissioned for the EFA Global Monitoring Report 2005, The Quality Imperative. UNESCO. Senge, P. M. (1990), The Fifth Discipline, Doubleday/Currency. Speer, S. (2007). Evaluation report of the Leonardo da Vinci Project Peer Review in Initial Vocational Education and Training. Cologne: Institut für Evaluation Dr. Beywl & Associates GmbH. Vedung, E. (2009). Public Policy and Program Evaluation. Fourth paperback printing. New Brunswick, NJ: Transaction Publishers. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.