Validating An Observation Instrument To Measure Pedagogical Didactic Behaviour Of University Teachers

Author(s):

Ine Noben(presenting / submitting)Adriaan Hofman Jan Folkert Deinum

Conference:

ECER 2017

Network:

22. Research in Higher Education

Format:

Paper

Session Information

22 SES 09 E,

Paper Session

Time:

2017-08-24

13:30-15:00

Room:

K5.14

Chair:

Maja Jankowska

Contribution

This study is inspired by the increasing attention to teaching quality by higher education institutions and policy makers. Notable authors (Kyriakides, Christoforou & Charalambous, 2013) in the field of teacher effectiveness research further remarked that tertiary education appears to be a less well examined field when it comes to teaching factors and their impact on students’ cognitive learning outcomes.

There is little agreement in the literature on the definition of teaching quality in higher education. Teaching quality is generally broadly defined as teaching that is oriented towards high quality learning outcomes (e.g. Biggs & Tang, 2011; Devlin & Samarawickrema, 2010; Hénard & Roseveare, 2012; Kember & Leung, 2011; Skelton, 2004). Furthermore, higher education institutions do not systematically collect data on the instructional practices of their staff (Wieman & Gilbert, 2014). In order to provide university teachers with detailed information about how they can improve their teaching, more insight into the complexity of teaching quality is needed.

The central research question of the study focuses on how teaching quality of university teachers can be measured and evaluated. Since previous research in other educational levels has established that the instruction and behaviour of teachers impact student learning (e.g. Kyriakides et al., 2013; Muijs et al., 2014), this study will focus on visible pedagogical didactic behaviour of university teachers in the classroom as an aspect of teaching quality. Furthermore, classroom observations are perceived as a valuable method to generate feedback for professional development purposes (Van der Lans, Van de Grift & Van Veen, 2017). The objective is to validate an observation instrument that can be used for professional development purposes.

We hypothesize that (1) effective teaching behaviour of university teachers can be measured and evaluated by an observation instrument, (2) a stagewise development of the teaching domains can be found and, (3) teachers with more teaching experience will score higher on the observation instrument than novice teachers.

Method

Instrument Items of good teaching behaviour in higher education were collected through a literature review. Most of the items found can be identified in the teaching domains of the International Comparative Analysis of Learning and Teaching (ICALT) observation instrument. Van de Grift (2007) developed the ICALT instrument to evaluate the pedagogical didactic behaviour of teachers in primary and secondary education. The reliability of the instrument has been established (Van de Grift, Helms-Lorenz, & Maulana, 2014; Van der Lans et al., 2017). The instrument consists of 32 items grouped into six quality teaching characteristics - safe and stimulating learning climate, efficient organisation, clarity of instruction, intensive and activating teaching, adaptive teaching, and teaching learning strategies - that are observable in every teaching and learning activity (Van de Grift, 2007). The instrument shows a stagewise development ordering of the teaching domains. The cumulative ordering of the items makes it possible to determine the teacher’s zone of proximal development (Van de Grift et al., 2014). To adapt the instrument to the context of higher education items specific for higher education were added based on the literature and feedback of higher education experts. Respondents 200 university teachers at the University of Groningen will be observed with the observation instrument between February and June 2017. All observers followed an observation training to safe-guard interrater-reliability. Sampling consisted of several stages. Depending on the faculty, teachers were asked to participate on voluntary basis (group A) or had the possibility to object against the observation after a letter was sent to inform them about the upcoming observations (group B). Teachers (in group A) who initially did not respond to the question to provide consent were sent a reminder in order to make the sample as representative of the teaching population as possible. In the sample we took several factors - gender, class size, discipline, level of the course, and moment when consent was given - into consideration. Additional personal and contextual characteristics will be collected by means of a questionnaire. Data analysis To examine and verify if the 6 hypothesised teaching domains hold for teachers in higher education, confirmatory and exploratory factor analysis will be carried out. If our hypothesis of 6 teaching domains will be confirmed, descriptive analyses will be carried out to study the general level of teaching behaviour. To examine if a cumulative, stagewise development can be found in the teaching domains, a Rasch analysis will be done.

Expected Outcomes

We expect the adapted ICALT observation instrument to be a valid method to measure and evaluate pedagogical didactic behaviour of university teachers. This implies that the 6 quality teaching characteristics are found to be sufficient in describing quality pedagogical behaviour in higher education. As well as our expectation that the level of effective pedagogical didactic behaviour of more experienced teachers will be higher than that of beginning teachers. Finally, since “effective teaching behaviour should show a cumulative, stagewise development” (Van der Lans et al., 2017, p.5) we expect our observations to fit the assumptions of the Rasch model and therefore to confirm such a development. However, due to the specific nature of higher education, the order of the items might differ from the one identified in primary and secondary education. We expect the outcomes of our study to be relevant for other higher education institutions. More insight in the complexity of teaching quality can help universities to enhance their professional development activities. Comparative research in other educational levels shows that teaching behaviour can be compared in a reliable and valid way regardless of the differences between countries (Van de Grift, 2007, 2014).

References

Biggs, J., & Tang, C. (2011). Teaching for quality learning at university (4th ed.). Berkshire, England: Open University Press. Devlin, M., & Samarawickrema, G. (2010). The criteria of effective teaching in a changing higher education context. Higher Education Research & Development, 29(2), 111-124, doi:10.1080/07294360903244398 Hénard, F., & Roseveare, D. (2012). Fostering Quality Teaching in Higher Education : Policies and Practices. OECD, (September), 54. Kember, D., & Leung, D.Y.P. (2011). Disciplinary differences in student rating of teaching quality. Research in Higher Education, 52(3), 279-299. doi:10.1007/s11162-010-9194-z Kyriakides, L., Christoforou, C., & Charalambous, C.Y. (2013). What matters for student learning outcomes: A meta-analysis of studies exploring factors of effective teaching. Teaching and Teacher Education, 36, 143-152. Maulana, R., Helms-Lorenz, M., & Van de Grift, W. (2016): Validating a model of effective teaching behaviour of pre-service teachers. Teachers and Teaching, 1-23. doi:10.1080/13540602.2016.1211102 Muijs, D., Kyriakides, L., van der Werf, G., Creemers, B., Timperley, H., & Earl, L. (2014). State of the art – teacher effectiveness and professional learning. School Effectiveness and School Improvement, 25(2), 231–256. doi:10.1080/09243453.2014.885451 Skelton, A. (2004). Understanding “teaching excellence” in higher education: a critical evaluation of the National Teaching Fellowships Scheme. Studies in Higher Education, 29(4), 451–468. doi:10.1080/0307507042000236362 Van de Grift, W. (2007). Quality of teaching in four European countries: a review of the literature and application of an assessment instrument. Educational Research, 49(2), 127–152. doi:10.1080/00131880701369651 Van de Grift, W. (2014). Measuring teaching quality in several European countries. School Effectiveness and School Improvement, 25(3), 295-311. doi:10.1080/09243453.2013.794845 Van de Grift, W., Helms-Lorenz, M., & Maulana, R. (2014). Teaching Skills of Student Teachers: Calibration of an evaluation instrument and its value in predicting student academic engagement. Studies in Educational Evaluation, 43, 150-159. Van der Lans, R., van de Grift, W. J. C. M., & van Veen, K. (2017). Developing an instrument for teacher feedback: using the rasch model to explore teachers' development of effective teaching strategies and behaviors. The Journal of Experimental Education, doi:10.1080/00220973.2016.1268086 Wieman, C., & Gilbert, S. (2014). The teaching practices inventory: A new tool for characterizing college and university teaching in mathematics and science. CBE Life Sciences Education, 13(3), 552–569. doi:10.1187/cbe.14-02-0023

Author Information

Ine Noben (presenting / submitting)

University of Groningen

Teacher education

Groningen

Adriaan Hofman

University of Groningen, Netherlands, The

Jan Folkert Deinum