This paper presents the unexpected lifecycle of an evaluation.

The plan was to evaluate an initiative to enhance the subject-specialist pedagogy of trainee vocational teachers in English Further Education (FE) Colleges. The intended participants were already teaching in the vocational science, engineering and technology (SET) sector whilst undertaking a 2-year part-time teacher training course. The short intervention, which was entirely voluntary and not assessed, augmented their generic studies by informing decision-making in relation to their own subject specialism.

Like most European countries, England has insufficient vocational teachers and many are approaching retirement. Finland is highly unusual in reporting that VET teaching is over-subscribed, with only a third of teacher training applicants getting a place, although even here over half the workforce is 50+ (Finnish National Board of Education, 2016). This situation feeds into a skills shortage: a recovering economy is hungry for recruits, the talent is there but there is a lack of teachers to enable them to achieve the required qualifications. Could subject-specialist input help improve teaching and support trainees to stay in the teaching system?

The intervention comprised two full-day workshops several weeks apart, supplemented by online materials. It was supported by two web-based seminars. It majored on four key concepts: pedagogy; pedagogical content knowledge (PCK; Shulman, 1986); recontextualisation (helping the learner move between educational and working contexts); and occupational identity (linked to Bourdieu’s concepts of habitus and field; Bourdieu, 1984). Pedagogy was conceptualised as content (the knowledge, skills and attributes students are given the opportunity to learn), process (how learning happens), teacher knowledge (of pedagogy and context), and decision-making (about what happens in the classroom and why).

The intention was to build the evaluation around how these concepts developed among trainees by analysing the language they used to describe their teaching, with an emphasis on their decision-making in session planning and delivery.

A literature review of the empirical research into generic and subject-specialist pedagogy (including PCK) was conducted to inform the design of the evaluation. We found few studies relating to vocational education. Most took place in secondary/high schools with an especially rich seam exploring PCK in science and mathematics. The interest in measuring PCK was international, and European studies included those from Germany (Kirschner, Borowski, Fischer, Gess-Newsome, & von Aufschnaiter, 2016; Kuhn, Alonzo, & Zlatkin-Troitschanskaia, 2016), the Czech Republic (Janik, Najvar, Slavík, & Trna, 2009), and Scandinavia (Lakkala, & Ilomäki, 2015; Nilsson & Vikström, 2015). Several of the 52 studies found had very low sample sizes.

A number of researchers had designed tests of PCK, and one interesting feature was the use of vignettes (Kirschner et al, 2016; Kuhn et al, 2016). These took the form of presenting the participant with a classroom-based scenario and asking them to make a decision about what should happen next.

Another interesting approach, developed by Loughran and colleagues in Australia, was to use content representations (CoRes) as a framework for teachers to focus on a “big idea” in their subject and how they would teach it. CoRes have been used to develop and to assess teachers’ PCK, thus deserving consideration as a potentially useful feature in both the intervention and evaluation.

Our evaluation was designed to assess the effectiveness of the intervention in the development of good teaching, with particular reference to the concept of subject-specific pedagogy. The key research questions were:

• What impact has the intervention had on the development of trainees’ teaching practice, including lesson planning and decision-making in the classroom?
• How successful has it been in enabling trainees to identify key elements of subject-specialist pedagogy and key principles in teaching and learning in SET subject areas?

For pragmatic reasons, it was decided to base the evaluation on participant self-report, using questionnaires and interviews. To gain a longitudinal perspective, these would be administered at different points from before implementation to long after the intervention finished, including after completion of their teacher training course. Discourse analysis would be used to examine the language employed by participants at different stages. We would also collect data on variables that influence teaching but are independent of the intervention, such as support in the workplace, previous experience, educational level. These would be used as factors in a Qualitative Comparative Analysis (Ragin, 2014) which aims to bring together the worlds of qualitative and quantitative analysis. It was anticipated that there would be difficulties isolating the effects of the intervention from those of the generic training that the participants were receiving, and from their day-to-day classroom experience. Consequently, a control group of SET trainees who not undergoing the intervention would be recruited to compare expressions of concepts and language and by comparing their course assignments with those of the intervention trainees. Nonetheless, we recognised the need to be wary of ascribing a causal relationship to any findings. We aimed to recruit about 100 trainees to the intervention. In the event, just seven participated. This necessitated a radical rethink. By this time, a cohesive package of resources (online documents, annotated clips of classroom practice, interviews with vocational teachers and so on) had been put together. It was deemed important to continue dissemination to a wider audience without sacrificing collection of an evidence base for its effectiveness. After discussion, the intervention has continued along two routes. One is to work with teacher educators, representing a shift in the theory of change such that the teacher educators are provided with information and training around the resource package and can then convey the ideas and concepts to their trainees. Although this removes the developers from direct contact with the trainees, it is a much more sustainable model with greater potential reach. In terms of the evaluation, the teacher educators form the research sample. The second route is serendipitous since it concerns a recently-conceived 2-year part-time training course for vocational SET teachers. A cut-down version of the subject-specialist intervention has been incorporated within the first year of this course: one full day with a half-day follow-up five months later, again supported by online materials. 23 participants are involved.

So far, and unexpectedly, the findings of this research have concerned issues around studying the vocational SET teaching population rather than the evaluation itself. Why did so few trainees sign up to the project when we were recruiting from the four institutions responsible for most FE teacher training? Further exploration showed that this was not because colleges were failing to send their new and unqualified staff to acquire teaching qualifications (not mandatory in English FE), nor primarily because the trainees were not attracted to the course (although workload did contribute to some drop-outs between signing up and actually attending). The key factor came back to the lack of SET teachers, including trainees, on the VET workforce. The evaluation of the two new intervention routes will proceed, although there are now extra timing and budgetary constraints. Methodologically, it is now highly unlikely that a control group can be recruited. The research methods will be adjusted to take into account the smaller and more diffuse samples than originally anticipated. Precautionary measures will also be taken. Response rate among the first cohort was poor, particularly after the intervention had finished (2 out of 7 completing a post-interview). To maximise the data gathered from the second trainee cohort, we are intending to collect data during the final workshop. We plan to design a limited number of vignettes from which participants can choose and complete one appropriate to their SET area. This will later be evaluated by the researchers using an agreed protocol. Discussion of the task (which will be audio-recorded) will act as a learning as well as an evaluative opportunity. Self-report data collected from the three cohorts so far suggests that the intervention is proving helpful and engaging. The concepts of PCK and big ideas/CoRes have proved particularly attractive.

Bourdieu, Pierre (1984). Distinction: a social critique of the judgement of taste. London: Routledge. Finnish National Board of Education (2016). Supporting teachers and trainers for successful reforms and quality of vocational education and training: mapping their professional development in the EU – Finland. Cedefop ReferNet thematic perspectives series. http://libserver.cedefop.europa.eu/vetelib/2016/ReferNet_FI_TT.pdf Janík, T., Najvar, P., Slavík, J., & Trna, J. (2009). On the dynamic nature of physics teachers' pedagogical content knowledge. Orbis scholae, 3(2), 47-62. Kirschner, S., Borowski, A., Fischer, H. E., Gess-Newsome, J., & von Aufschnaiter, C. (2016). Developing and evaluating a paper-and-pencil test to assess components of physics teachers’ pedagogical content knowledge. International Journal of Science Education, 38(8), 1343-1372, DOI: 10.1080/09500693.2016.1190479. Kuhn, C., Alonzo, A. C., & Zlatkin-Troitschanskaia, O. (2016). Evaluating the pedagogical content knowledge of pre-and in-service teachers of business and economics to ensure quality of classroom practice in vocational education and training. Empirical Research in Vocational Education and Training, 8(1), 1-18. Lakkala, M. M., & Ilomäki, L. L. (2015). A case study of developing ICT-supported pedagogy through a collegial practice transfer process. Computers & Education, 90, 1-12. Loughran, J., Berry, A., & Mulhall, P. (2012) Understanding and Developing ScienceTeachers’ Pedagogical Content Knowledge. Rotterdam: Sense Publishers. Nilsson, P., & Vikström, A. (2015). Making PCK Explicit—Capturing Science Teachers’ Pedagogical Content Knowledge (PCK) in the Science Classroom. International Journal of Science Education, 37(17), 2836-2857. Ragin, C. (2014). The comparative method. Oakland: University of California Press. Shulman, L. (1986) Those who understand: knowledge growth in teaching. Educational Researcher, 15(2), 4-14.