A Longitudinal Study of Randomized Trial: The Impact Of a School Effects Intervention on an Urban School District

Author(s):

Eugene Schaffer(presenting / submitting)Samuel Stringfield(presenting)

Conference:

ECER 2011

Network:

11. Educational Effectiveness and Quality Assurance

Format:

Paper

Session Information

11 SES 10 A, Searching Educational Quality on Rural, Urban and Suburban Contexts

Paper Session

Time:

2011-09-15

15:00-16:30

Room:

JK 26/101,G, 36

Chair:

Salvador Peiró-i-Gregòri

Contribution

Our objective is to present the design, implementation, outcomes and possible policy implications of a three-year randomized control trial (RCT) attempting to implement “effective schools correlates” in 34 elementary schools, followed by a two-year follow up in four urban elementary schools serving relatively high concentrations of high poverty students. The essential questions were:

1. Will an intervention based on the School Effectiveness literature improve the performance of the schools in the experimental group at a significantly higher level during the initial three-year intervention?

2. Within a subset of the intervention schools, do student outcomes improve in the two years post-intervention?

3. What level and duration of implementation appear to be needed to improve school performance?

The Effective Schools for the 21^st Century (ES-21) project

Following a study developed in the United Kingdom(Stringfield, Reynolds and Schaffer, 2008) ES-21 was to test the ability of seven school effectiveness correlates (Taylor & Bullard, 1995, Teddlie & Reynolds, 2000) to improve the academic achievement of students attending relatively high poverty schools in the United States. A secondary purpose was to examine the two-year follow up results.

In this paper we will describe the research methods, the intervention, the study’s 3-year results, results from a 2-year follow up in a sub-sample of schools, and possible conclusions that can be drawn from this effort.

Method

The overarching design for ES-21 was a “gold standard,” random control trial (RCT) effectiveness study. As described by Shadish, Cook & Campbell (2002), proactive random assignment “reduced the plausibility of alternative explanations for observed effects” (p. 247). The unit of assignment for this RCT was the school. Power analyses indicated that the project would need at least 30 schools (15 experimental and 15 control) to have a reasonable probability of finding significant differences where significant differences existed, so the design called for random assignment of four sets of four experimental and four controls in a total of four states (with apologies to T.S. Elliot, “The Four Quartets”). Four of the schools were from one urban district and these are followed in the longitudinal follow-up phase. The follow-up intervention included tracking of the four experimental schools and their control counterparts’ performance on standardized tests, as well as interviews and observations from the schools.

Expected Outcomes

Qualitative data indicated that implementation was uneven across the districts and schools. Four of five districts changed leadership, the majority of schools changed principals at least once during the three years of intervention, and schools’ leadership teams of teachers often suffered 100+% turnover. Still, many changes were incorporated in classrooms, particularly regarding use of data as a decision-making tool in the schools. The 3-year analyses revealed no advantages for experimental schools in any of the four contexts. The four schools in the follow-up study reflected the no-3-year advantage for the intervention pattern described above. However, two-year follow-up data indicated that in both reading and mathematics, three of four experimental schools made substantially greater gains than did their controls. Results from the initial 3-year intervention replicated findings from previous studies in the U.S. and Europe causing skepticism that the “effective schools correlates” were correlates and not valuable as levers to change. However, Fullan and Miles (1992) have observed initial “implementation dips” and that meaningful change takes years. Follow up analyses in one district more nearly track Stringfield, Reynolds, and Schaffer’s (2008) follow-up of an intervention in Wales, indicating that improvement in the “out years” may be more likely than during the initial implementation phase.

References

Borman, G., Hewes, G., Overman, L., & Brown, S. (2003). Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research, 73 (2), 125-230. Datnow, A., Borman, G., Stringfield, S., Rachuba, L., & Castellano, M. (2003). Comprehensive school reform in culturally and linguistically diverse contexts: Implementation and outcomes from a four-year study. Educational Evaluation and Policy Analysis, 25(2), 143-170. Datnow, A., Lasky, S., Stringfield, S., & Teddlie, C. (2005). Systemic integration for educational reform in racially and linguistically diverse contexts: A summary of evidence. Journal of Education for Students Placed At Risk, 10 (4), 441-453. Edmonds, R. (1979). Effective schools for the urban poor. Educational Leadership, 37, 15-27. Fullan, M.G., & Miles, M.M. (1992). Getting reform right: What works and what doesn’t. Phi Delta Kappan, 73 (10), 744-752. Hargreaves, A., & Fink, D., (2006). Sustainable Leadership. San Francisco: JosseyBass/Wiley. Miles, M B., & Huberman, A. M. (1994). Qualitative data analysis. 2nd Edition. Thousand Oaks, CA: Sage. Nunnery, J. A. (1998). Reform ideology and the locus of development problem in educational restructuring. Education and Urban Society, 30 (3), 277-295. Stringfield, S., Ross, S., & Smith, L. (Eds.). (1996). Bold plans for school restructuring: The New American Schools designs. Mahwah, NJ: Lawrence Erlbaum Associates. Supovitz, J. & Weinbaum, E. (2008). The implementation gap. New York: NY: Teachers College. Taylor, B., & Bullard (1995). The revolution revisited. Bloomington, IN: Phi Delta Kappa. Teddlie, C. & Reynolds, D. (2000). Handbook of research on school effectiveness and improvement. London: Falmer

Author Information

Eugene Schaffer (presenting / submitting)

University of Maryland Baltimore County

Education

Baltimore

Samuel Stringfield (presenting)

University of Louisville, United States of America

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.