Academic Progress Under Various Assessment Systems

Author(s):

Rob Kickert(presenting / submitting)Marieke Meeuwisse Karen Stegers-Jager Peter Prinzie Lidia Arends

Conference:

ECER 2017

Network:

22. Research in Higher Education

Format:

Paper

Session Information

22 SES 05 D,

Paper Session

Time:

2017-08-23

13:30-15:00

Room:

K5.19

Chair:

Diana Aguiar Vieira

Contribution

“The quickest way to change student learning is to change the assessment system” (Elton & Laurillard, 1979, p.100)

Improving academic progress in higher education is an important goal for educational stakeholders, since academic progress rates in Europe are disappointing (Vossensteyn et al., 2015). Several studies have shown that assessment drives learning (Al-Kadri, Al-Moamary, Roberts, & Van der Vleuten, 2012; Heeneman, Oudkerk Pool, Schuwirth, van der Vleuten, & Driessen, 2015). However, less is known about how assessment drives academic progress. For instance, what happens to students’ academic progress when the stakes are raised, or when compensation between grades is allowed? The fact that the assessment systems of a large university in the Netherlands recently changed, offers an opportunity to examine whether changes in assessment systems are related to changes in academic progress.

In this study, we define an assessment system as the way in which multiple assessments within a course programme are combined in order to decide whether students are allowed to continue their course programme, or are forced to drop out (i.e. academic dismissal). We differentiate the assessment systems on the basis of three factors: i) the height of the stakes (Cole & Osterlind, 2008), i.e. the number of credits that is required for students to continue their course programme, ii) the degree of compensation among grades for separate examinations (Chester, 2003), and iii) the number of resits students are allowed to take (Proud, 2015).

In case of changes in academic progress following alterations in factors of the assessment system, it is important to know whether this merely happened because the assessment system has become easier (i.e. altered selection), or because students are actually doing a better job (i.e. altered student performance). Therefore, in addition to academic progress, we will take Grade Point Average (GPA) into account. Combining academic progress with GPA will inform us about the relative contribution of altered selection and altered student performance. For instance, when significantly more students show increased academic progress under a new assessment system, but the overall GPA remains the same, this is a clear indication that only the selection has changed (i.e. it has gotten easier to pass), and not the student performance (i.e. students do not perform better). When both progress and GPA show the same trend, it is an indication that the changes are mainly due to different student performance. In addition to comparing GPA with academic progress, we will also simulate how many students would have progressed, if they had studied under a different assessment system. In this case, the assessment system that affects students’ performance, for instance through their motivation, is different from the assessment system that makes the simulated selection. This should clarify the relative contribution of altered selection and altered student performance to possible differences in academic progress as well.

In this study, we will compare academic progress and performance of students in the same university, before and after changes in the assessment systems were made. In general, the old assessment systems were conjunctive, thus students needed to pass each individual assessment with a satisfactory grade. In the new assessment systems, the stakes were raised by requiring a higher number of credits for students to continue their course programme, and compensation between grades was introduced. The first research question is: Does academic progress change under higher-stakes compensatory assessment systems compared to lower-stakes conjunctive assessment systems? The second research question is: To what extent are possible differences in academic progress attributable to changes in selection or changes in student performance under the renewed assessment system?

Method

Educational Context Data were gathered at three large schools of the Erasmus University Rotterdam, the Netherlands. The school of management and the school of economics changed their assessment systems in 2012, the medical school did so in 2014. Although the schools have different assessment systems, they all raised the stakes and introduced some form of compensation between grades. Therefore, we will call the new assessment systems compensatory and the old systems conjunctive (i.e. no compensation is allowed). Participants Participants were first-year students of the last two cohorts under the conjunctive assessment systems and first-year students of the first two cohorts under the compensatory systems, in all three schools. Outcome measures To determine academic progress, we used the actual pass/fail decisions made about students at the end of each academic year. To answer our first research question we compared the number of students who were allowed to continue their course programme under both assessment systems. In the conjunctive assessment systems, students could obtain a so-called provisional decision at the end of their first year, and the final decisions were only made after two years. On the contrary, in the compensatory assessment systems, at the end of the first year only positive and negative decisions are possible. To deal with this disparity, we will compare the decision in the conjunctive systems both after the first year and the second year, with the decision after one year in the compensatory systems. To answer our second research question, we first compared the GPA of the first-year course exams under the conjunctive systems to the GPA of the first-year course exams under the compensatory systems. Furthermore, we compared academic progress in four scenarios: i) for students under the conjunctive assessment system, with the rules of the conjunctive system, ii) for students under the conjunctive system, with the rules of the compensatory system, iii) for students under the compensatory system, with the rules of the compensatory system and iv) for students under the compensatory system, with the rules of the conjunctive system. Statistical analyses In order to compare academic progress we performed chi-square tests on the numbers of students who showed academic progress under both systems. To compare the GPA’s of the first-year course exams under both systems, we performed ANOVA’s. In order to compare academic progress under the four scenarios, we performed chi-square tests on the number of students who progressed.

Expected Outcomes

For our first research question on academic progress, preliminary results (from only one school) showed that significantly more students progressed in the compensatory assessment system (71.0%), compared with the conjunctive system after one year (50.9%). However, a significantly lower number of students progressed in the compensatory system (71%), compared with the conjunctive system after two years (85.4%). This seems to indicate that the compensatory assessment system is more difficult but speeds up the selection process. For our second aim, regarding the possible sources of differences in academic progress, preliminary results for GPA showed that students exhibited enhanced performance under the compensatory assessment system. Although there was no statistically significant difference in GPA when comparing the compensatory system to the conjunctive system after one year, this picture changed when comparing the compensatory system (M=6.69) to the conjunctive system after two years (M=6.40): Students under the compensatory system obtained significantly higher GPA’s. At this time, we do not have any results for the comparison of academic progress under the four different scenarios. However, we expect academic progress to be highest for the scenarios in which there is congruence between the assessment system that causes student performance and selection, and when students need to perform under higher stakes (Wolf & Smith, 1995). Thus, we expect the highest academic progress for compensatory students with compensatory rules, followed by conjunctive students with conjunctive rules, compensatory students with conjunctive rules, and lastly conjunctive students with compensatory rules. In conclusion, this study shows that under higher-stakes compensatory assessment systems, progressing is more difficult, but students who do progress show superior performance and do this faster. Hence, our conclusion is that assessment drives academic progress by raising academic performance. Therefore, optimizing the assessment systems may be an efficient way to boost academic progress in higher education across Europe.

References

Al-Kadri, H. M., Al-Moamary, M. S., Roberts, C., & Van der Vleuten, C. P. M. (2012). Exploring assessment factors contributing to students’ study strategies: literature review. Medical Teacher, 34 Suppl 1, S42-50. https://doi.org/10.3109/0142159X.2012.656756 Chester, M. D. (2003). Multiple Measures and High-Stakes Decisions: A Framework for Combining Measures. Educational Measurement: Issues and Practice, 22, 32–41. https://doi.org/10.1111/j.1745-3992.2003.tb00126.x Cole, J. S., & Osterlind, S. J. (2008). Investigating differences between low-and high-stakes test performance on a general education exam. The Journal of General Education, 57, 119–130. Elton, L. R. B., & Laurillard, D. M. (1979). Trends in research on student learning. Studies in Higher Education, 4, 87–102. https://doi.org/10.1080/03075077912331377131 Heeneman, S., Oudkerk Pool, A., Schuwirth, L. W. T., van der Vleuten, C. P. M., & Driessen, E. W. (2015). The impact of programmatic assessment on student learning: theory versus practice. Medical Education, 49, 487–498. https://doi.org/10.1111/medu.12645 Proud, S. (2015). Resits in Higher Education: Merely a Bar to Jump Over, or Do They Give a Pedagogical “Leg Up”? Assessment & Evaluation in Higher Education, 40, 681–697. https://doi.org/10.1080/02602938.2014.947241 Vossensteyn, H., Kottmann, A., Jongbloed, B., Kaiser, F., Cremonini, L., Stensaker, B., … Wollscheid, S. (2015). Dropout and completion in higher education in Europe: main report. Retrieved from http://doc.utwente.nl/98513/ Wolf, L. F., & Smith, J. K. (1995). The Consequence of Consequence: Motivation, Anxiety, and Test Performance. Applied Measurement in Education, 8, 227–242. https://doi.org/10.1207/s15324818ame0803_3

Author Information

Rob Kickert (presenting / submitting)

Erasmus University Rotterdam

Department of Psychology, Education and Child Studies (DPECS)

Rotterdam

Marieke Meeuwisse

Department of Psychology, Education & Child Studies, Faculty of Social Sciences, Erasmus University Rotterdam, Netherlands, The

Karen Stegers-Jager

Institute of Medical Education Research Rotterdam, Erasmus University Rotterdam, Rotterdam, The Netherlands

Peter Prinzie

Department of Psychology, Education & Child Studies, Faculty of Social Sciences, Erasmus University Rotterdam, Netherlands, The

Lidia Arends

Department of Psychology, Education & Child Studies, Faculty of Social Sciences, Erasmus University Rotterdam, Netherlands, The; Department of Biostatistics, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands

Search the ECER Programme

Search for keywords and phrases in "Text Search"
Restrict in which part of the abstracts to search in "Where to search"
Search for authors and in the respective field.
For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.