Student Evaluation of Teaching (SET) is the procedure by which students evaluate and rate teaching performance. Usually, during a SET procedure students complete rating forms or questionnaires about different aspects related to their teachers, but mostly about their teaching practices. Universities or higher education institutions from all over the world implement SET procedures to achieve 3 main purposes. Generally, and from a practical point of view, the main purpose of implementing this type of procedure in most universities is the necessity of reporting SET results to quality assurance agencies. The other main goal of SET procedures, and surely the most important one from an educational perspective, is to provide feedback to academics about their teaching practices and/or to design teacher training programs focused on developing teaching skills. Another important use of SET results is related to evaluating evidence of teaching performance to use the results for academic career advancements or other ways of rewarding teaching effectiveness.

The topic of Student Evaluations of Teaching (SET) is one of the most researched ones in the domain of educational research, with over 2000 studies published in peer-reviewed journals over a period of a little more than 100 years (Spooren et al., 2017). One of the earliest debates in this field of research is about the validity of the SET scales and procedures. The main question was whether the measurement instruments applied to students during these procedures can accurately measure teaching effectiveness. Even if this debate was most active in the 1970s and the evidence was inclining more towards the affirmative answer to the question in case (see reviews from Richardson, 2005 and Marsh, 2007), a recently published meta-analysis (Uttl et al., 2017) presented some evidence which seriously threatens the validity of SET results. The results of the mentioned study strongly suggest that there is no relationship between the SET results of a teacher and the level of their students’ achievement/learning.

Analyzing the SET scales utilized in the studies included in the last-mentioned meta-analysis, we observed that lots of them contain items that are either too vague, either unclear, or even completely unrelated to observable teaching actions and behaviors. Moreover, many SET instruments had only 1 or 2 overall questions about teaching quality.

Some of the main reasons why we chose to develop a new SET scale are the following: (1) most SET scales are not rigorously developed from the theoretical and psychometrical point of view, and those that are, were developed in older times when digital environments were not so common; (2) the majority of SET instruments lack available and published psychometrical evidence for the validity of the results; (3) there are very few instruments that contain questions strictly related to the observable behaviors and actions of teachers; and (4) there are not many SET instruments than can be adapted to different teaching contexts by adapting the selection and number of dimensions to be measured.

 Thus, we propose the development and validation of a new multi-dimensional instrument that is based on a systematic mixed-methods process of defining effective teaching in higher education, consists of questions related to the observable actions and behaviors of teachers, and contains a large number of dimensions suitable for several teaching contexts.

Given that the purpose of the study is to develop and validate a SET scale, the research objectives are the following:

1) To perform the theoretical substantiation of the construct and related dimensions.

2) To develop and refine items for each dimension of the scale.

3) To gather evidence in favor of validity (content, construct and criterion) and fidelity (test-retest, internal consistency) of the scale.

For start, we will carry out an extensive and systematic literature review on the topic of teaching effectiveness and SET measures, to outline a comprehensive definition and dimensionality of the construct of interest. Also to define the construct and identify the dimensions, we will conduct interviews with effective award-winning teachers, with students, and with experts in education, asking them about what an effective teacher does, in terms of actions and behaviors, to facilitate and stimulate learning in higher education. Once the dimensions are identified, we will define and operationalize them in terms of teaching actions and behaviors that are observable from the student’s perspective. Afterward, we will generate an expanded set of items of 10-15 items for each dimension (considering the dimensions’ operational definitions) and we will also choose an answer scale that reflects the nature of the dimensions and the purpose of the measurement. Regarding the process of gathering evidence in favor of the new SET scale’s reliability and validity, we will follow the following steps: 1) Assessment of content validity by a panel of 3 experts in teacher training and evaluation, who will evaluate the dimension definitions and the items, for assessing their relevance and coverage of the construct. 2) Pre-piloting the instrument with a small number of participants from the intended population to observe whether the items and answer options are properly interpreted by the respondents and, if necessary, to revise the items according to what was observed. 3) Piloting the instrument on a sample closely representative of the target population and performing descriptive analysis of data, the analysis of the fidelity of the sample (test-retest, internal consistency), and confirmatory factor analysis, to revise and reduce the scale and to refine the items that need refinement. 4) In terms of checking relationships with other constructs, we will use 3 tools that are theoretically related to our construct. To check the convergent validity we will use the ATI (Approaches to Teaching Inventory) (Trigwell, Prosser, and Ginns, 2005) which is a teaching approach evaluation tool. At the same time, we will also use the SEEQ (Students Evaluation of Educational Quality) (Marsh, 1982) which is maybe the most rigorously developed SET scale. In terms of criterion validity, we will analyze the links between our instrument and the R-SPQ-2F (Revised-Study Process Questionnaire-2 Factors) (Biggs, Kember, & Leung, 2001), which refers to the ways in which students approach learning (deep or surface).

The expected outcome is a new rigorously developed and evidence-based multi-dimensional SET scale for evaluating teaching effectiveness in higher education. We expect to obtain positive evidence of content, construct, and criterion validity and also evidence for the usefulness of the results in improving teaching actions and behaviors. Also, considering the large number of teaching dimensions we want to assess through the new SET scale, we expect that it could be used flexibly in different settings or disciplines, by teachers or SET administrators. In other words, it will offer teachers or SET administrators the opportunity to select and evaluate only the dimensions that are relevant for that specific setting or discipline. From the perspective of the impact on educational practice and teacher training, the SET scale resulting from this research project will allow the following: (1) giving feedback and support to teachers based on the scores obtained on each dimension; (2) identifying the training needs of teaching staff with the aim of developing training programs aimed at those needs; (3) measuring teaching effectiveness as a result of various initiatives to increase the quality of teaching and reviewing these initiatives based on their effects; and last but not least (4) the use of the results in promotion decisions or for awarding teaching activity, aspects that may lead to greater involvement in teaching improvement initiatives from teachers.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Biggs, J., Kember, D., & Leung, D. Y. (2001). The revised two‐factor study process questionnaire: R‐SPQ‐2F. British Journal of Educational Psychology, 71(1), 133-149. Marsh, H. W. 1982. “SEEQ: A Reliable, Valid, and Useful Instrument for Collecting Students’ Evaluations of University Teaching.” British Journal of Educational Psychology 52 (1): 77–95. Marsh, H. W. (2007). Students’ evaluations of university teaching: Dimensionality, reliability, validity, potential biases and usefulness. In P.R., Pintrich & A. Zusho (Coord.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 319-383). Springer, Dordrecht. Richardson, J. T. (2005). Instruments for obtaining student feedback: A review of the literature. Assessment & evaluation in higher education, 30(4), 387-415. Spooren, P., Vandermoere, F., Vanderstraeten, R., & Pepermans, K. (2017). Exploring high impact scholarship in research on student's evaluation of teaching (SET). Educational Research Review, 22, 129-141. Trigwell, K., Prosser, M., & Ginns, P. (2005). Phenomenographic pedagogy and a revised approaches to teaching inventory. Higher Education Research & Development, 24(4), 349-360. Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22-42. Willis, G. B. (2004). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.