Pre-Post-Test- and Control-Group-Designs have usually been used to test the effectiveness of educational interventions (e.g., Astleitner, in print). These designs are driven by scientific concepts of internal or/and external validity (e.g., Lissitz & Samuelsen, 2007). In the field of educational effectiveness and quality assurance, these designs are important, but have to be expanded to a more holistic approach of testing in order to get a more complete picture of effectiveness patterns. It is the objective of this paper, to develop and show a methodological model that can stimulate holistic ways of effectiveness testing. The model considers and integrates parameters of time, of contexts, and of different-intervention-functioning. This model is called MUEMO (Multiple Evidence Model) and it combines different lines of effectiveness concepts which are given in social research (e.g., Campbell & Russo, 1999). The meta-theoretical resp. methodological background concerning time-bound effects of educational interventions can be found in single case, process-orientated, and also longitudinal research concepts. Background in respect to context-related effectiveness can be found in fields and concepts like, for example, “action research”, “ecological validity”, “technological theories”, or “design research” (e.g., Middleton, Gorard, Taylor, & Bannan-Ritland, 2008). The methodological background for “different-intervention-functioning” lies in “utility research”, “medicine outcome research”, and “Aptitude-Treatment-Interaction(ATI)-research” (e.g., Caspi & Bell, 2004). According to MUEMO, “effectiveness” of educational interventions is given, when goal competencies are persistently increased by interventions (e.g., instructions, trainings, or counselling) in comparison to a non-intervention situation. Such non-intervention situations can be found in contexts given before the intervention (“Pre-Condition”) or in contexts without interventions (“No-Conditions”). In MUEMO, it is assumed that educational effectiveness has to be tested three-times in eight resulting tests: concerning the time aspect, the context aspect, and the different-intervention-functioning aspect. Each of these tests is based on comparison between an intervention and a non-intervention situation. “Effectiveness in respect to time” is given, when there are increasing goal competencies during (1), at the end (2), and after the intervention (follow up) (3). “Effectiveness in contexts” is given, if there are increasing goal competencies within similar (4) and nested contexts (5), as well as within daily life situations (6). “Different-intervention-functioning” concerns side-effects (7) and interaction-effects (8) of educational interventions.

The MUEMO is built by using qualitative methods of theory construction, i.e. “tactics for generating meaning” by Miles & Huberman (1984). This method consists of subsuming particulars into general, factoring, noting relations between concepts, building a logical chain of concepts, and finally creating conceptual/theoretical coherence. In order to show how MUEMO can be used in educational effectiveness studies, a quasi-experimental study was undertaken testing an instructional method (“learning cycles” as intervention) in Science courses. About 50 8th-grade students took part in a pre-posttest-study with a control group. The intervention was applied in more than 30 lecture units within Chemistry and Physics courses. For measuring dependent and control variables, Science Reasoning Tasks (Wylam & Shayer, 1980) und Culture Fair Intelligence Tests (Weiß, 1987), subscales on Interest (Hoffmann, Häußler, & Lehrke, 1998), and grades as indicators of overall achievement were used. Data has been analysed by computing ANOVA- and ANCOVA-statistics.

The major results of this piece of research are 1. a methodological model for improving effectiveness research, i.e. MUEMO, and 2. a MUEMO-based test of effectiveness concerning a certain instructional method in Science education (learning cycles). MUEMO has a function for hypothesis testing (about expected effectiveness), but also represents stimulation for exploring hypothesis about so far unknown or unexpected effectiveness as discovered by using MUEMO. MUEMO can be used in combination with approaches dealing with ways how scientifically confirmed educational knowledge can be applied to educational practice, especially, in case of data-based interventions. Within a next step in the development of MUEMO, decision mechanisms have to be established that help to make modifications of educational interventions based on different patterns of effectiveness.