Session Information
09 SES 14 C, Education-Based Program Modeling, Implementation and Modification – Defining Systematic Methods and Strategies
Symposium
Contribution
This presentation will first provide a brief overview of the state of implementation science in the field of education evaluation and will then walk through a methodology for systematically collecting and reporting implementation data – developed and utilized throughout the United States Department of Education’s Investing in Innovation (i3) initiative. i3 has funded 143 school-based evaluations conducted throughout the US for over $1,000,000,000 since 2010. The presenters will also introduce challenges and approaches in developing logic models of developing programs and means to account for adaptations to the model as implementation occurs. As the national evaluator, Abt Associates has been contracted to assess the quality implementation studies conducted for each funded evaluation and report findings associated with both sets of studies. For this presentation, the authors focus on Abt’s developed guidance and an assessment system which specifies characteristics of high-quality fidelity of implementation studies. Although the field largely agrees that measurement of fidelity is an important process in program evaluation and recent US-based publications have provided a series of frameworks from which to ground implementation research in academic interventions (Century, Rudnick, and Freeman, 2010) and social, health, and other educational programs (Dusenbury, Brannigan, Falco, & Hansen, 2003; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005), additional guidance for developers and researchers on creating valid fidelity measurement systems is needed. This paper will present an approach to developing implementation measures and combining data through a four step process which includes: 1. Using logic models to identify key components necessary to support program implementation 2. Identifying the activities or elements of full implementation and operationalizing them as measurable indicators 3. Creating a fidelity scoring system across indicators and a summary score for each key component at the sample-level (i.e., across all units implementing the program) 4. Establishing a definition of what constitutes “adequate” implementation for each key component at the sample-level Along with the steps embedded in this system, the paper will present the ways in which Abt, as the national evaluator, addresses issues of program modification and adaptation as interventions are implemented across multiple years of funding. This presentation will highlight universal issues in implementation measurement and program adaptation and will lead into the subsequent papers which present specific cases of implementation challenges that arose in particular contexts – 200 public institutions across Hungary and over 25 middle schools across six states in the United States.
References
Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199-218. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 28, 237-256. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, Florida: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.