The Quality of Self-Evaluation Process in Italian Schools
Conference:
ECER 2015
Format:
Paper

Session Information

11 SES 06 B, Factors Impacting on the Quality of Education

Paper Session

Time:
2015-09-09
15:30-17:00
Room:
IX. Előadó [C]
Chair:
Andra Fernate

Contribution

School self-evaluation has become more and more important for the quality assurance of educational systems of many OECD countries. Since the eighties a variety of self-evaluation practices have emerged. Also, several goals have been ascribed to school self-evaluation: on the one hand accountability to central authorities or local stakeholders, on the other hand stress on the potential for school improvement. Recent recommendation by OECD indicates the need to align external evaluation with school self-evaluation in order to improve school quality.

Despite the growing attention toward school self-evaluation, research on this topic is still lacking and far from being exhaustive. Little attention has been given to assess the impact of self-evaluation on school improvement. Research evidences suggest that school self-evaluation is perceived as positive from school leaders and teachers; its usefulness concerns mainly the potential for improving teachers attitudes toward  professional development and toward school improvement.  On the other hand, there are unsufficient findings that school self-evaluation has a positive effect on students’ outcomes. Some studies evaluated the use of feedback in self-evaluation that aims at  improving school performance. These findings show a limited use of such feedback by schools. Other studies have pointed out the problem of validity and reliability of the self-evaluation process carried out by schools. Blok et al. (2008) analyzed school self-evaluation reports  finding significant shortcoming. The self-reports did not seem to be useful as a basis for accountability or for school improvement. A way of improving school self-evaluation validity is to connect it with external evaluation procedures. According to Nevo (2002), external evaluation can stimulate the internal evaluation process and improve its scope and validity. However, it has been pointed out that externally initiated self-evaluation may decrease school compliance and thus many of its positive impact on schools. Also, school self-evaluation that mainly focus on the measures of productivity or output can have paradoxical effects on teachers and schools: it might lead to resistance, destroying strategic compliance.

Literature reserach shows that, despite positive views of school self-evaluation, more research is needed on the quality of the process of self-evaluation carried out by schools in order to assess its validity and its use in the context of schools’ and students’ outcome improvement. Given the increasing importance attributed to school self-evaluation by national policies and by schools themselves, assuring validity is the basic requirement for expecting a positive impact on improvement.

Starting form such concerns, our study intended to assess the quality of the self-evaluation process carried out in 290 Italian schools involved in the Vales project. The Vales project, promoted by the Ministry of Education and by the National Institute for the Educational Evaluation of Instruction and Training (INVALSI) is a pilot study aimed at testing the validity of instruments and procedures for school evaluations, it is a preliminary step in the implementation of the National Evaluation System (enacted in 2013 which extend school evaluation to all Italian schools). Vales schools implemented self-evaluation and then received a visit by a team of external evaluators. Self-evaluation and external evaluation were strongly connected: first of all, they shared the same theoretical framework. Schools were asked to analyze the same areas evaluated by external teams, using the same set of indicators. School self-evaluation was supported by INVALSI; the schools received data about their performance in several areas (students’ achievement, educational and management processes) and information about national average data to compare their position; they also received a self-evaluation report format and a sourcebook. Self-evaluation reports were then used by external evaluators as one of their sources of information during school visits.

 

Method

The study involved 290 primary and secondary schools located throughout Italy; such schools volunteered to participate to the project. The study was carried out using both qualitative and quantitative methods. A content analysis of 290 school self-reports was carried out using QDA Miner. Data on coding occurrences were later exported in Spss in order to perform more sophisticated analysis. Also, data from NGT and focus groups with 70 head teachers participating to the project were analyzed in order to investigate their views about the self-evaluation tools and process. The aim of the content analysis was to assess the quality of the self-evaluation process. Quality was operationalized as: 1) the utilization by schools of data (either data given by INVALSI and other data collected by schools) and of benchmarking as a basis for the analysis of their performance in several areas; 2) the ability to find out strengths and weaknesses and to plan actions for improvement. Schools carried out internal evaluation on several areas ranging from students’ outcomes to educational processes, school leadership and management. In each area, schools were asked to evaluate their performance using a numeric score (from 1 = inadequate to 4 = excellent) and to justify the choice of the score with a detailed analysis based on empirical evidence. Schools then could indicate up to four improvement objectives in order to face the major weaknesses. The NGT and focus group were aimed at measuring and discussing participants’ opinions about the self-evaluation method and process. Within groups the agreement/disagreement with some items was first assessed (using a Likert scale) and then discussed. Items concerned: the utility and validity of specific instruments (self-report format, sourcebook, indicators), strengths and weaknesses of the self-evaluation process; degree of acceptance of self-evaluation in the school; the use of self-evaluation results for the management of the school. Data analysis focused on the description of the quality and validity of the self-evaluation process. The scores self-attributed by schools were compared with those given by external evaluators in order to enhance the analysis of validity. Results about the content analysis were then compared with participants’ views expressed during focus groups.

Expected Outcomes

Results show that although schools, in most cases, are able to identify their strengths and weaknesses (comparison of self-attributed scores and externally-attributed scores show a good degree of correlation), they nevertheless are not always able to identify the factors and conditions that contributed to the low quality of their performance in some areas, as well as of their good results in other areas. Self-report analysis showed a general poor quality of analysis carried out by schools in order to justify the attribution of the scores. Some schools had problems with the interpretation of the data; seldom used benchmarking to gather information about the school’s position and were more willing to describe the activities they were doing than to rely on data about such activities (e.g. number of participants, students’ outcome, etc.) in order to evaluate efficacy and outcomes. Also, the ability to link self-evaluation results and improvement objectives need to be ameliorated. Many schools were not able to define improvement objectives in an operative way. In some cases, objectives were too wide or too vague to be translated into measurable indicators and expected results; in others, indicators and expected results were not consistent with the scope of the improvement objective. NGT and focus groups results confirmed the positive view of schools about self-evaluation. Consistently with the content analysis results, main difficulties were founded in the identification and operationalization of improvement objectives. As highlighted by other studies on this topic, the results of our research confirm the need to enhance the quality of self-evaluation processes through developing school competences in doing self-evaluation, especially on data analysis and the use of feedback for planning school improvement.

References

Blok H., Sleegers P., Karsten S. (2008) Looking for a balance between internal and external evaluation of school quality: evaluation of the SVI model, Journal of Education Policy, 23:4, 379-395. Chapman C., Sammons P. (2013) School Self-evaluation for School Improvement: What works and why? Project Report. CfBT Education Trust. Durrant J., Dunnill R., Clements S. (2004) Helping schools to know themselves: exploring partnerships between schools and Higher Education Institutions to generate trustful, critical dialogue for review and development. Improving Schools, 7(2), 151–170. Ehren M.C.M., Altrichter H., McNamara G., O’Hara J. (2013). Impact of school inspections on improvement of schools—describing assumptions on causal mechanisms in six European countries. Educational Assessment, Evaluation and Accountability, 25(1), 1-41. Janssens F.J.G., Amelsvoort G.H.W.C.H. van (2008) School self-evaluations and school inspections in Europe: an exploratory study. Studies in educational evaluation, 34 (1). pp. 15-23. Ladd H.F. (2012). School Accountability: To what ends and with what effects? Keynote address for Conference on Improving Education through Accountability and Evaluation: Lessons from Around the World, Rome, Italy. Retrieved January 29, 2015. http://fds.duke.edu/db/attachment/2050. MacBeath J. (2008) Leading learning in the self-evaluating school, School Leadership & Management: Formerly School Organisation, 28:4, 385-399. MacBeath J. (2002) Scotland: Schools speaking for themselves, School-Based Evaluation: International Perspective, vol. 8, 243-259. Nevo D. (2002) Dialogue Evaluation: combining internal and external evaluation, School-Based Evaluation: International Perspective, vol. 8, 3-16. OECD (2013) Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris. Ozga J. (2009) Governing education through data in England: from regulation to self-evaluation. Journal of Education Policy, 24(2), 149–162. Scheerens, J., Van Amelsfoort H.W.C.G., Donoughue C. (1999) Aspects of the organizational and political context of school evaluation in four European countries. Studies in Educational Evaluation 25: 79–108. Schildkamp K., Vanhoof , Petegem van P., Visscher A. (2012) The use of school self-evaluation results in the Netherlands and Flanders, British Educational Research Journal, 38:1, 125-152. Schildkamp K., Visscher A. (2010) The utilisation of a school self‐evaluation instrument, Educational Studies, 36:4, 371-389. Vanhoof J., Petegem Van P. (2011) Designing and evaluating the process of school self-evaluations. Improving Schools, 14 (2), 200–212. Vanhoof J. Petegem Van P. [2006]How to Match Internal and External Evaluation? Conceptual Reflections from a Flemish Perspective. Antwerp University, Belgium, 20th International Congress for Effectiveness and Improvement. http://www.fm-kp.si/zalozba/ISBN/978-961-6573-65-8/261-277.pdf.

Author Information

Donatella Poliandri (presenting / submitting)
INVALSI - National Institute for the Educational Evaluation of Instruction and Training, Italy
Isabella Quadrelli (presenting)
INVALSI
Frascati
INVALSI - National Institute for the Educational Evaluation of Instruction and Training, Italy
INVALSI - National Institute for the Educational Evaluation of Instruction and Training, Italy
INVALSI - National Institute for the Educational Evaluation of Instruction and Training, Italy

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.