A model for evaluating the impact of Intervention in Educational Organizations
Conference:
ECER 2015
Format:
Paper

Session Information

WERA SES 02 B, World-Wide Perspectives on Management and Leadership in Education

Paper Session

Time:
2015-09-08
15:15-16:45
Room:
303. [Main]
Chair:
Teresa Bracho

Contribution

The progressive application of intervention plans and programs in a very diverse range of organizations over the last several decades has brought about an increasingly frequent presence of publications making reference to the evaluation of objectives, effects, achievements, or impacts as outcomes of such intervention (Fernández Díaz, Rodríguez Mantilla & Fontana, 2014). Clearly, professionals have a great interest in determining just how efficient and effective those plans or programs are so that they can then assess the extent to which the effort spent actually produces the expected objectives or results, their magnitude, and even the factors determining their attainment.

For that reason, research is needed to find out the true impact, in which ‘impact’ is understood as being the set of relevant substantial changes produced over time following a specific action or intervention, and consolidated as sustainable changes in organizations, whether that be in the attitude of the personnel, in the way of working, in the climate, in the culture of the institution, in the planning, evaluation and in general in the development of the organizations (Fernández Díaz, 2013).

However, despite the benefits and improvements that certain intervention programs may have, there is little evidence of their impact on the organization (Rodríguez Ponce et al., 2011; De Vries, 2005; Stensaker et al., 2011). It therefore becomes necessary to introduce the culture of impact evaluation, i.e., of assessing the mid- and long-term effects that crystallize as sustainable changes, thereby modifying or enhancing their operation and their results. We estimate that there are three main conditions for evaluating impact in organizations understood in that way (Fernández Díaz, 2013; Valdéz, 2008):

  1. At least two measurements must be taken to be able to compare them and assess the change accordingly.
  2. Evaluation of impact involves measuring the changes observed in the application of an intervention.
  3. A minimum number of years of implementing the program at the organization is required to be able to analyze its impact.

Starting from those three premises, this paper proposes a model of evaluation that can be defined as:

  1. A multidimensional model: the dimensions that configure an educational organization must be identified and defined for an intervention to have an effect on it.
  2. A model with its own methodology of evaluative processes, defined as:
    • Longitudinal study, although not always feasible, since processes and outcomes must be assessed over time at different moments.
    • Measuring and collecting information that involves obtaining data from the very start of the intervention and across time to be able to compare the results. The information gathering techniques must be both quantitative and qualitative (Lambert, G & Ouedraogo, 2008).
    • Analytic techniques: the information must be analyzed (with quantitative and qualitative techniques) to get a deep understanding of the impact, the magnitude of the change, and its associated factors as an outcome of implementing the program.
    • Decision-making: along with knowledge of the achievement of the objectives of the processes and results, the analysis must help make decisions for improving the system and let these programs be applied to other organizations in the conditions most suited to their effectiveness.

The paper will justify the pertinence and consistency of the model and will present its application to a specific field, namely, the Quality Management System (QMS) at educational institutions.

Method

The following methodological process was used to define the model and then apply it: 1) Bibliographic review on educational organizations, and detailed analysis of them to define the dimensions, sub-dimensions and indicators that encompass the entire educational organization and on which the impact may be perceived. 2) Justification and definition of a suitable evaluative methodology that can be used to assess the impact on an organization. This methodology was defined in aspects such as design, measurement, data collection, and decision-making. 3) Application of the model to the evaluation of the impact of the Quality Management System in educational organizations: a) From the bibliographic study, the following dimensions were then defined: Information and Communication Systems, Management System, School Climate, Learning-Teaching Processes, Satisfaction of the Education Community and External Relations and Links to Society. b) Definition of the research design: given the fact that there were no initial assessments done before the QMS was implemented, it was necessary to use an indirect evaluation through how the members of the educational community who were at the institution from the start perceived the change. c) Selection of information gathering techniques, both quantitative and qualitative. Drawing up a scale consistent with the dimensions and indicators as defined, and whose technical study showed excellent levels of validity and reliability. Complementary to that, there were discussion groups, interviews, etc. with different members of the educational community. d) Assessment of the operability of the model in its empirical application by analyzing the process and the results obtained. The results gave a positive assessment of the model.

Expected Outcomes

The most relevant conclusions from the study are: - There is a significant lack of impact evaluation studies, which makes it difficult to assess the mid-to-long-term effect of the many intervention programs much beyond the most immediate results. This may indicate a lack of efficiency in using resources in the area of education. - The design of the intervention plans needs to incorporate an impact evaluation design that contemplates all the elements needed for a true, in-depth evaluation of the impact on the organization. This necessitates measurement-taking prior to the start of the intervention and throughout the entire process, over a period of time long enough to reveal consolidated and integrated changes in the organization. - There is the requirement of taking on evaluative methodological procedures and adapting them to suit the particulars of each organization and the plan for intervention to be evaluated. - Although they are not always feasible, longitudinal design are needed to evaluate the change by considering measurements taken at the start and at different moments in the process. - Different instruments are requiered to evaluate a wide range of aspects with the required methodological rigor. - Application of the model to the study of the impact of the QMS at educational institutions shows the operability of the model and has yielded results of great interest. QMSs are shown to have significant impact on most of the dimensions in the model, to varying magnitudes in some cases. This sheds a positive light on their application and generalization to other contexts and systems of organization.

References

De Vries, W. (2005). Calidad, eficiencia y evaluación de la educación superior. Madrid: Netbliblo. Fernández Díaz, M. J. (2013). Evaluación del impacto para un cambio sostenible en las organizaciones educativas. Revista Española de Pedagogía, 254, 45-65. Fernández Díaz, M. J., Rodríguez Mantilla, J. M. & Fontana, M. (2014). Impact of implementation of quality management systems on internal communications and external relations at schools. Total Quality Management & Business Excellence, DOI: 10.1080/14783363.2014.954365. See http://www.tandfonline.com/doi/abs/10.1080/14783363.2014.954365#.VMfIX0B0zDc (accessed on 13-1-2015) Lambert, G. y Ouedraogo, N.(2008) Empirical investigation of ISO 9001 quality management systems impact on organizational learning and process performance. Total Quality Management, 19:10, pp. 1071-1085. Rodríguez-Ponce, E., Pedraja-Rejas, L., Araneda-Guirriman, C., González-Plitt, M & Rodríguez-Ponce, J. (2011). El impacto del sistema de aseguramiento de la calidad en el servicio entregado por las universidades privadas en Chile. Ingeniare. Revista chilena de ingeniería, 19 (3), 409-419. Stensaker, B., Langfeldt, L., Harvey, L, Huisman, J, & Westerheijden, D. F. (2011). An in-depth study on the impact of external quality assurance. Assessment & Evaluation in Higher Education, 36 (4), 465- 478. Valdés, M. (2008) La evaluación de impacto de proyectos sociales. See http://scholar.google.es/scholar?q=La+evaluaci%C3%B3n+de+impacto+de+proyectos+sociales+de+Valdes&btnG=&hl=es&as_sdt=0%2C5 (accessed on 18-1-2015)

Author Information

Gonzalo Jover (presenting / submitting)
Universidad Complutense
Educational Theory and History of Education
Villaviciosa de Odon
Universidad Complutense
Madrid
Universidad Complutense
Madrid

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.