Effects of Large Scale Assessments in Schools: How Standard-Based School Reform Works
Author(s):
Tobias Diemer Harm Kuper
Conference:
ECER 2008
Network:
11. Educational Effectiveness and Quality Assurance
Format:
Paper

Session Information

11 SES 04A, School Assessment to Improve Performance

Paper Session

Time:
2008-09-10
16:00-17:30
Room:
B1 133
Chair:
Ineta Luka

Contribution

Comparative large scale assessments form a centerpiece of recent standard-based reforms of the educational systems of the Federal Republic of Germany. Together with reform measures such as the implementation of external evaluation and inspection of schools on the basis of statewide quality frameworks, the formulation of national as well as state-specific standards in education and the implementation of according large scale standard tests in many States (Länder) of Germany mark a considerable shift away from an input-oriented towards an output-oriented account of governance. By providing and feeding back standardized and comparative data about pupils’ achievement, standard tests intend to make teachers and schools accountable for their outcomes. At the same time, they are supposed to help to control and improve the outcomes of the pupils’ work by means of evidence-based decision-making processes concerning the profession of teaching as well as the task of organizing schooling. The proposed paper deals with the question of whether and how this really works, that is whether and how the results from comparative large scale assessments are utilized by teachers as professionals and schools as organizations. It therefore will examine the profession- and organization-related processes and effects that are produced in line with large scale standardized testing and the feeding back of comparative results to teachers and schools. The paper will present typological descriptions of effects and process-related patterns of individual as well as collective data-based decision-making that is based on large scale assessment test results in schools. Special attention will be drawn on noticeable consequences and changes regarding the conceptualisation and the design of teaching and learning processes by teachers. Exploration and analysis of the outlined effects and processes are carried out on two levels of abstraction. On a first, comparatively concrete level, the observable phenomena are described within the framework of a heuristic model suggested by Helmke (2004). According to this model. the process of utilization of test results conceptually subdivides into four cyclically iterative stages: (1) the reception of reported information about results, (2) the reflection of possible and actual reasons and causes, (3) the discussion, planning, and realization of suitable actions, and (4) the evaluation of achieved and missing effects. Subsequently, the findings described within these categories are further aggregated as well as reaggregated on a more abstract level. On this level theories of professions (PARSONS, RÜSCHEMEYER) and organizational theories, particularly new institutionalism (MINTZBERG, MEYER/ROWAN, SCOTT), sensemaking theory (WEICK) and system theory (LUHMANN) are analysed in reference to the results found on the more concrete level.

Method

To capture the processes of decision-making, it is used a longitudinal case study approach based on semi-structured qualitative problem-centered interviews with headmasters and teachers. The material is analyzed according to procedures of qualitative content analyses and grounded theory. The data basis consists of about 120 interviews and 8 observations, that are accomplished in 4 schools in the space of 4 data collecting phases spanning over a period of 2 years. Due to its longitudinal design, the study gives information on the process-related conditions and effects of standard-based reforms in schools. As the schools are located in two different Federal States of Germany, it is possible to observe influences of partially differing test designs and institutional environments.

Expected Outcomes

Within the outlined conceptual frameworks, empirical evidences will be presented that provide systematic as well as exemplary insight into the ways standard-based school reform works in schools. Furthermore, by reason of the integration of profession- and organization-related models, the study contributes to the development of a general theory of the functioning of school development in the context of the present standard-based and outcome-orientated paradigm of governance within the educational system.

References

Carnoy, M., & Loeb, S. (2002). Does External Accountability Affect Student Outcomes? A Cross-State Analysis. Educational Evaluation and Policy Analysis, 24(4), 305–331 Coburn, C. E. (2001). Collective Sensemaking about Reading: How Teachers Mediate Reading Policy in Their Professional Communities. Educational Evaluation and Policy Analysis, 23(2), 145–170 Darling Hammond, L. (1994). National Standards and Assessments: Will They Improve Education? American Journal of Education, 102(4), 478–510 Darling Hammond, L. (2004). Standards, Accountability, and School Reform. Teachers College Record, 106(6), 1047–1085 Darling Hammond, L., & Ball, D. Loewenberg. (1994). Teaching for High Standards: What Policymakers Need To Know and Be Able To Do. CPRE Joint Report Series. American Journal of Education, 102(4), 478–510 Darling Hammond, L., & Wise, A. E. (1985). Beyond Standardization: State Standards and School Improvement. Elementary School Journal, 85(3), 315–336 Denzin, N. K. (2005). The SAGE handbook of qualitative research. Thousand Oaks, CA: Sage. Dey, I. (1999). Grounding grounded theory. San Diego: Academic Press. DiMaggio, P. J., & Powell, W. W. (1983). The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields. American Sociological Review, 48(2), 147–160 Dingwall, R., & Lewis, P. (Eds.) (1983). The Sociology of the Professions: Laywers, Doctors, and Others. New York: St. Martin's Press. Douglas Orton, J., & Weick, K. E. (1990). Loosely Coupled Systems: A Reconceptualization. The Academy of Management Review, 15(2), 203–223 Fidler, B. (2001). A Structural Critique of School Effectiveness and School Improvement. In A. Harris & B. Nigel (Eds.), School Effectiveness and School Improvement. Alternative Perspectives. New York/London: Continuum. 47-74 Glaser, B. G. (1992). Basics of grounded theory analysis. Mill Valley, CA: Sociology Press. Glaser, B. G. (1998). Doing grounded theory. Mill Valley, CA: Sociology Press. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory. New York: Aldine de Gruyter. Goldstein, H. (2001). Using Pupil Performance Data for Judging Schools and Teachers: Scope and Limitations. British Educational Research Journal, 27(4), 433–442 Hamilton, L. S., Stecher, B. M., & Klein, S. P. (Eds.) (2002). Making sense of test-based accountability in education. Santa Monica, CA: Rand. Hanushek, E.A., & Raymond, M.E. (2004). The Effect of School Accountability Systems on the Level and Distribution of Student Achievement. Journal of the European Economic Association, 2(2-3), 406-415 Harris, S. G. (1994). Organizational Culture and Individual Sensemaking: A Schema-Based Perspective. Organization Science, 5(3), 309–321 Helmke, A. (2004). Von der Evaluation zur Innovation. Pädagogische Nutzbarmachung von Vergleichsarbeiten in der Grundschule. Das Seminar, 2, 90-112 Ingram, D., Louis, K. Seashore, & Schroeder, R. G. (2004). Accountability Policies and Teacher Decision Making: Barriers to the Use of Data to Improve Practice. Teachers College Record, 106(6), 1258–1287 Kane, T.J. & Staiger, D.O. (2002b). Validity in School Test Scores: Implications for Test-Based Accountability Systems, Brookings Papers on Education Policy. Washington DC: The Brookings Institution. Klieme, E. et al. (2004). The development of national educational standards. An expertise. Education reform, 1. Bonn: Federal Ministry of Education and Research. Kuper, H. (2001). Organisationen im Erziehungssystem. Vorschläge zu einer systemtheoretischen Revision des erziehungswissenschaftlichen Diskurses über Organisation. Zeitschrift für Erziehungswissenschaft 4(1), 83-106 Kuper, H. (2004). Das Thema ‚Organisation’ in den Arbeiten Luhmanns über das Erziehungssystem. In D. Lenzen (Ed.). Irritationen des Erziehungssystems, Frankfurt am Main: Suhrkamp, 122-151 Kuper, H. (2005). Evaluation im Bildungssystem, eine Einführung. Stuttgart: Kohlhammer. Kuper, H./Hartung, V. (2007): Überzeugungen zur Verwendung des Wissens aus Lernstandserhebungen. Eine professionstheoretische Analyse. Zeitschrift für Erziehungswissenschaft, 10(2), 214-229 Louis, K. Seashore, Febey, K., & Schroeder, R. (2005). State-Mandated Accountability in High Schools: Teachers' Interpretations of a New Era. Educational Evaluation & Policy Analysis, 27(2), 177-204 Luhmann, N. (2000). Organisation und Entscheidung. Opladen: Westdeutscher Verlag. Luhmann, N. & Schorr, K.E. (1979). Das Technologiedefizit der Erziehungswissenschaften und die Pädagogik. Zeitschrift für Pädagogik, 85, 345-365 Luyten, H., Visscher, A., & Witziers, B. (2005). School Effectiveness Research: From a Review of the Criticism to Recommendations for Further Development. School Effectiveness and School Improvement, 16(3), 249-279 Meyer, H.-D., & Rowan, B. (Eds.). (2006). The new institutionalism in education. Albany: State University of New York Press. Meyer, J. W., Meyer, J. W., & Rowan, B. (1977). Institutionalized Organizations: Formal Structure as Myth and Ceremony. American Journal of Sociology, 83(2), 340–363 Mintzberg, H. (1979). The Structuring of Organizations. Englewood Cliffs (NJ): Prentice Hall. Parsons, T. (1939). The Professions and Social Structure. Social Forces 17, 457-467 Parsons, T. (1968). Professions. International Encyclopedia of the Social Science 12, 536-547 Parsons, T. (1978). Research with Human Subjects and the ‘Professional Complex”. In T. Parsons (1978). Action Theory and the Human Condition, New York: Free Press, 35-65 Strauss, Anselm L., Professions, Work and Careers, New Brunswick: Transactio Rowan, B. (2006). The New Institutionalism and the Study of Educational Organizations: Changing Ideas for Changing Times. In: Meyer, H.-D. & Rowan, B. (Eds.) (2006). The New Institutionalism in Education. New York: State University of New York Press. 15-32 Rueschemeyer, D. (1964). Doctors and Lawyers: A Comment on the Theory of the Professions. Canadian Review of Sociology & Anthropology, I(1), 17–30 Rueschemeyer, D. (1983). Professional Autonomy and the Control of Expert Services. In R. Dingwall, & P. Lewis (Eds.), The Sociology of the Professions: Laywers, Doctors, and Others (pp. 38–58). New York: St. Martin's Press. Spillane, J. P. (2000). Cognition and Policy Implementation: District Policymakers and the Reform of Mathematics Education. Cognition and Instruction, 18(2), 141–179 Spillane, J. P., Reiser, B. J., & Reimer, T. (2002). Policy Implementation and Cognition: Reframing and Refocusing Implementation Research. Review of Educational Research, 72(3), 387–431 Stake, R. E. (1995). The art of case study research. Thousand Oaks, CA: Sage Stake, R.E (2005). Qualitative Case Studies. In N.K. Denzin & Y.S. Lincoln (Eds.), Qualitative Research (pp. 443-466). Thousand Oaks, CA: Sage. Strauss, A. L. (1997). Grounded theory in practice. Thousand Oaks, CA: Sage. Strauss, A. L., & Corbin, J. (1990). Basics of qualitative research. Newbury Park, CA: Sage. Visscher, A. J., & Coe, R. (Eds.) (2002). School improvement through performance feedback. Lisse: Swets & Zeitlinger. Visscher, A. J., & Coe, R. (2003). School Performance Feedback Systems: Conceptualization, Analysis, and Reflection. School Effectiveness and School Improvement, 14(3), 321-349 Weick, K. E. (1976). Educational Organizations as Loosely Coupled Systems. Administrative Science Quarterly, 21(1), 1–19 Weick, K. E. (1982). Administering Education in Loosely Coupled Schools. Phi Delta Kappan, 63(10), 673–676 Weick, K. E. (1993). The Collapse of Sensemaking in Organizations: The Mann Gulch Disaster. Administrative Science Quarterly, 38(4), 628–652 Weick, K. E. (1995). Sensemaking in organizations. Foundations for organizational science. Thousand Oaks, CA: Sage. Weick, K. E. (2001). Making Sense of the Organization. Malden, MA: Blackwell Publishers. Yin, R. K. (2005). Case study research: Design and Methods. Thousand Oaks, CA: Sage. Zucker, L. G. (1980). Institutional Structure and Organizational Processes: The Role of Evaluation Units in Schools. Los Angeles, CA: California University, Center for the Study of Evaluation.

Author Information

Tobias Diemer
Freie Universität Berlin
Education and Psychology, Adult Education
Berlin
54
Harm Kuper
Freie Universität Berlin, Germany

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.