Student engagement and quality pedagogy This research has its origin with a concern to better understand how those aspects of teaching and learning over which educators have most control – school environments, curriculum and pedagogy – impact on engagement and learning outcomes. This work flows from wider research into the effectiveness of school based reform in the State of South Australia. These were to focus on ‘measuring what matters’ to avoid the likelihood of goal displacement (Goldspink and Winter 2007). There is currently a widespread concern within educational circles about student ‘engagement’(Zyngier 2004). Engagement was defined for our purposes as including the behaviour of involvement, the experience of wellbeing and the use of higher order thinking. The instrument set is intended to provide methods suited to the collection of data from all age groups. The approach is distinctive in three ways: 1) it avoids the use of proxies such as attendance which link only weakly to learning outcomes preferring direct measures collected in particular learning contexts; 2). It is designed to be sensitive to the effect of learning environments and quality of pedagogy on engagement; and 3) it does not assume that the learning environment involves conventional schooling. Our aim was ultimately be able to measure the effect of alternative pedagogy on engagement. We do know that constructivist pedagogy has a significant impact on learning outcomes from prior research such as that undertaken in association with the authentic pedagogy movement in the US (Newmann et al. 1995). Related research undertaken in Australia, particularly the Queensland School Reform Longitudinal Study (QSRLS) (Queensland School Reform Longitudinal Study 2001; Allan 2003; Lingard et al. 2003; Gore et al. 2004) and more recently the New South Wales Quality Teaching model (QT) (State Government NSW 2003; Ladwig 2005) has shown a similar effect. Each of these approaches has, however, focused on measuring the effect of alternative pedagogy directly on learning outcomes. Furthermore the learning outcomes considered have varied (the Australian approaches cited above have excluded social outcomes for example). All have noted that achieving and sustaining change in pedagogic practice is difficult. The South Australian approach has proceeded along distinctive lines. Firstly, within the South Australian curriculum framework are Essential Learning’s. These relate to social outcomes of education namely: futures, identity, interdependence, thinking and communication. Secondly, the approach to reform has combined theory and research input with a bottom up approach to transformation – what Foster refers to as ‘learning our way forward’ (Foster 2001). This has involved teachers in developing local responses to change and in participating in the research to find change strategies that work and can be sustained. It brings to the fore the fourth difference – recognition that in order to change teachers, parents and school leaders as well as policy advocates, need short cycle evidence of the effect of alternative strategies or pedagogic choices on learners. It was recognition of this final point that led us to place significant emphasis on measuring engagement.

The trial sample was chosen using purposive sampling. Fourteen sites in urban and rural South Australia were chosen, being selected to include a diverse student base and a mix of learning situations (both conventional schooling and special needs and a variety of subjects) between years 7 and 13. Several sites were included from the Innovative Community Action Networks program (ICAN). ICAN is an extra schooling program designed to meet the needs of students who, for a variety of reasons, have found it difficult to engage with conventional schooling. This ensured that a diverse student base was included, with representation from different socio-economic backgrounds, ethnicity and learning abilities. Observation data was collected by experienced educators who were also trained in the use of the observational instruments. Training included an introduction to the concepts and exercises to converge different observers’ judgments with respect to the scales. Past experience (Winter 2003; Ure 2004) has shown a very good level of inter-rater reliability with the use of these instruments. Observers were drawn from a pool and in general observers were not staff at the school being observed so can be expected to be free of site related bias.Data was collected by observation, by student self-report using the trial questionnaires and from teachers. Observational data was collected from a sub-sample of students during the learning, while self-report data was collected from all participants immediately after the students had experienced the selected learning.

The analysis used both exploratory and confirmatory approaches. With the exception of the thinking scale, the instruments proved sensitive to student difference and site/class differences as per their design. The detailed analysis revealed that the involvement and wellbeing instruments for both methods (without the dispositions component) produce statistically comparable results when used to study the same respondents in the same learning situation. This indicates that each is measuring what it is designed to measure and hence is good evidence as to the instrument validity. All scales used were measured as producing internal reliability scores in the satisfactory to good range. The instruments, again with the exception of the dispositions component and the thinking scale, were judged fit for use in a mixed methodology or independently. The observation method is judged as most suitable for establishing absolute scores benchmarked against professional judgment where observers with an educational background are used and provided they have been trained to ensure familiarity with concepts and convergence of ratings. The self-report method is most suitable where a relative difference between student groups is important. The analysis has also suggested the need to improve some questions and the next stage will involve the testing of revised instruments with a larger sample. Incorporation of thinking statements in the self-report instrument failed to support a separate thinking scale. Attention needs to be given to the development of an effective means for measuring quality of thinking.