The Natural sciences and Technology for All (NTA) programme is a teacher support programme built on inquiry-based learning principles (James, 2006; Cakir, 2008). It is a Swedish version of the Science and Technology for Children (STC) programme in the US. The NTA is, by far, the largest teacher support program in Sweden. At the beginning of our period of study, in 2011, it was used in 1/3 of Sweden’s 290 municipalities, by around 7 000 teachers, teaching about 100 000 students. Our study includes around 1 500 NTA teachers and almost 25 000 NTA students. This makes it the largest quantitative effect evaluation of a STC-like programme that has been undertaken hitherto (Slavin et al., 2014; Zoblotsky et al., 2016). For example, the number of participating students is 25 times larger than in an earlier analysis of the NTA programme (Mellander & Svärdh, 2018a-b). Together with data on high-stake scholastic achievements and very rich background information for both the NTA participants and students that have not attended the programme, this allows the programme effects to be much more precisely than in previous studies and, moreover, to be broken down by different categories of students.

Our empirical analyses are based on records on student and teacher participation in the NTA programme, collected by the NTA administrative organization, that have been matched with individual register data (longitudinal population data), and information about school and municipality characteristics. The register data contain information about student results on national, standardised tests in the Science subjects and in Mathematics, grades in these subjects plus grades in Technology, extensive information about the students and their parents, and detailed records of the NTA teachers’ education. The fact that participation in the NTA programme is not determined by means of random assignment is accounted for by propensity score analysis, which identifies “synthetical twins” to the NTA participants among the students that did not participate in the programme. Specifically, the NTA and non-NTA students are compared along 24 different dimensions, statistically weighted together to a scalar in the [0,1] interval. Twin couples are formed by choosing NTA and non-NTA students with (almost) the same scalar values. The effect of the NTA programme is estimated as the mean differences in scholastic achievements over the twin couples.

Participation in the NTA programme during grades 4–6 is shown to have positive and statistically significant effects on the results in national, standardised, tests in Biology, Chemistry, Physics, and Mathematics as well as on grades in these subjects, and in the subject Technology, in school year 6. The effects on the two outcome variables are not statistically different from one another. The magnitude of the effects varies between 0.4 and 1.0 merit points, which, in turn, corresponds to 15 % and 40 %, respectively, of the difference between two consecutive grades on the “Pass or higher” part of the grading scale. No effect differences are found between girls and boys, but for students with foreign background the effects are 40 % lower than for students born in Sweden. The NTA effects are larger for students that have participated to a greater extent in the programme, compared to students that have participated less. The education of the NTA teachers matters, but not in all respects. NTA students that have been taught by NTA teachers that have a formal pedagogical qualification benefit twice as much as students taught by NTA teachers lacking such a qualification. However, if a pedagogically qualified teacher also has formal qualifications with respect to Science, Technology or Mathematics that does not increase the effect further.

During the period 2011–2014, all schools employing the NTA programme in grades 4–6 were asked to provide the NTA administrative organisation with information about the names and personal IDs of the participating teachers and students, by class, NTA “theme”, and semester. Accordingly, teacher-student links were ascertained. Moreover, the personal IDs made it possible to match the collected information with register data – longitudinal population records – containing outcome variables and as well as large sets of background variables (Mellander, 2017). From this information, data were extracted on two cohorts of students, born in 2000 or 2001. For these, there is outcome information in the form of results on national, standardised, tests in the Science subjects and in Mathematics, as well as grades in these subjects and in Technology, in school year 6. Further, data on the educational background of the NTA teacher’s education were added. Finally, potential control students were extracted from the register data, i.e., students born in the years 2000 or 2001 that had not participated in the NTA programme. Altogether, the resulting data set encompassed over 125 000 students – approximately 25 000 NTA participants and 100 000 potential control students, and 1 500 NTA teachers. With respect to the effect evaluation, the study’s methodological challenge is that participation in the NTA is not determined by random assignment, but by self-selection. Accordingly, merely comparing the scholastic achievements of participants and non-participants will yield biased estimates. To account for the self-selection, we employ propensity score analysis (Guo & Fraser, 2010). This method estimates the likelihood of treatment (NTA participation), as a function of a multitude of observable personal characteristics. The estimation includes all students, i.e., students that actually did participate as well as for students that did not participate. Our data allowed us to include 24 student characteristics. Couples of participants and non-participants with (almost) equal likelihoods of participation (propensity scores) are then formed. The effect of the NTA programme can be estimated as the mean difference in scholastic achievements over these couples. To examine if the effects differ across groups of students or vary with the extent of participation in the NTA programme, we also apply multivariate linear regression analysis to the matched sample. The matched sample is made up of the NTA participants and their synthetical twins, as determined by the propensity score analysis. All analyses were conducted by means of the Stata program (StataCorp., 2019).

Participation in the NTA programme during grades 4–6 have positive and statistically significant effects at the end of grade 6, on the results in national, standardised tests in Biology, Chemistry, Physics, and Mathematics, and on grades in the same subjects, plus Technology. The effects on the tests and on the grades are not statistically significantly different. The effects can be expressed in terms of merit points, which are associated with letter grades, according to: F (0), E (10), D (12.5), C (15), B (17.5) A (20) where F, denoting Fail, is the lowest grade and A the highest. On average, the NTA effect on the science subjects is 0.8 merit points, i.e., about 1/3 of the differences between two consecutive grades in the E – A range. For Mathematics, the effect is slightly higher, about 1 merit point. The smallest effect is recorded for Technology, 0.4 merit points. Regarding potentially heterogeneous effects across students, no differences were found between girls and boys. However, the scholastic achievements of NTA students with foreign background were 40 % lower than the achievements of native NTA students. With respect to the dose-response relation, the results show that the NTA effects are increasing in the amount of programme participation, as expected. The fact the NTA teachers differ by educational background makes it possible to examine the importance of different forms of treatment. We find that NTA classes taught by persons with pedagogical qualifications yield effects that are twice as high as the effects generated by persons without such qualifications. For persons with pedagogical qualifications it does not matter, however, whether they also have formal subject matter qualifications. Apparently, the extensive teaching instructions, guidance, and preparation provided by the NTA organisation makes up for deficiencies with respect to the latter.

Cakir, M. (2008) Constructivist Approaches to Learning in Science and Their Implications for Science Pedagogy, International Journal of Environmental & Science Education, 3(4). 193-206. Gou, S. & M.W. Fraser (2010) Propensity Score Analysis: Statistical methods and applications Advanced Quantitative Techniques in the Social Sciences 11. Los Angeles: Sage Publications. James, M. (2006) Assessment, Teaching and Theories of Learning, in J. Gardner (Ed.) Assessment and Learning (47-60) (London: Sage). Mellander, E. (2017) On the use of register data in educational science research, Nordic Journal of Studies in Educational Policy (NordSTEP), 3(1), 106–118. Mellander, E. & J. Svärdh (2018a) Inquiry-based learning put to the test: Medium-term effects of Science and Technology for Children programme, Review of Education, 6(2), 103–141. Mellander, E. & J. Svärdh (2018b) Context and Implications Document for: Inquiry-based learning put to the test: Medium-term effects of Science and Technology for Children programme, Review of Education, 6(2), 142–145. Slavin, R. E., C. Lake, P. Hanley, & A. Thurston (2014) Experimental evaluations of elementary science programs: A best evidence synthesis, Journal of Research in Science Teaching, 51(7), 870–901. StataCorp. (2019) Stata Statistical Software: Release 16. College Station, TX: StataCorp LLC. Zoblotsky, T., Bertz, C., Gallagher, B. & Ahlberg, M (2016) The LASER Model: A Systemic and Sustainable Approach for Achieving High Standards in Science Education, SSEC i3 Validation Final Report of Confirmatory and Exploratory Analyses (Memphis, USA: University of Memphis, The Center for Research in Educational Policy).