The Rhetoric Of Online Testing in Higher Education: A Mixed Methods Longitudinal Case Study
Author(s):
Ken Brown (presenting / submitting) Victor Lally Jarkko Hurme
Conference:
ECER 2017
Format:
Paper

Session Information

Paper Session

Time:
2017-08-22
15:15-16:45
Room:
K6.04
Chair:
Man Ching Esther Chan

Contribution

The application of computer aided assessment (CAA) is widely employed across many domains in Higher Education. It is suggested that computer aided methods of assessment offer several benefits in Higher Education such as reduction in lecturer workload for large class groups and the possibility of a higher degree of consistent objectivity in the assessment process. The increasing sophistication offered by CAA systems makes the use of such technologies attractive for assessment purposes. The assessment develops a communication with the student where feedback and estimation of performance are the main components. Research focusing on the assessment technologies (Jordan, 2013; Sangwin, 2012; Narciss et al, 2014) suggests growing maturity in the domain of mathematics and the promise of greater availability of such technologies to lecturing staff. The benefits and positive aspects of online assessment have been outlined in the literature for, mathematics applications (Passmore & Brookshaw, 2011), online quizzing (Jordan, 2013) and formative assessment and feedback (Tempelaar et al, 2014; Cox 2012). The growth of Computer Algebra Systems (CAS) (Sangwin, 2012) extends the mechanics of computer aided assessment systems by introducing greater capability to consider concepts that are too abstract for lower order testing methods; for example, Multiple Choice Questioning. The importance of the role of assessment technologies cannot be understated and their encapsulation within Higher Education curricula is becoming an accepted element of programme design.

 

In Ireland, the mathematics curriculum was redesigned (Prendergast, 2016) in an attempt to address reported deficiencies in the mathematical skillsets of students at second level. A review of the mathematics provision (McCraith, 2015) raised further issues about the skillset with particular reference to the transition from second level to third level, the use of Information and Communications Technologies, and international comparison. Discussion with lecturing colleagues in a similar Higher Education institution in Finland revealed similar concerns (Rinneheimo, 2010; Cole et al, 2014).

 

This paper presents the findings from a two year longitudinal study conducted by the partner Higher Education institutes in Finland and Ireland using a mixed methods approach beginning in academic year 2015/2016. The research gathered information from students and lecturers on their respective expectations and perceptions about online assessment of first year engineering mathematics. The basis for the research was anecdotal observation of negative behavioural attributes, prior to, and shortly after, online assessment. Evidence within the literature suggests these attributes may be deeply embedded in students as they make the transition to third level (Gallimore & Stewart, 2014), leading to greater support requirements in third level. The research outputs are forming a baseline for the design of improved programmes utilizing online assessment methods. The interaction of students and staff with the technologies is vital to the success of these programmes, in conjunction with an understanding of the perceptions held by students (Tempel & Newmann, 2014) in relation to barriers, and how students engage online.

 

The research was designed within the boundaries of a social-cognitive approach, to self-efficacy theory, within engineering mathematics (Bandura, 1977; Martchand, 2012; Artino & Jones, 2012), as the core concept, to help the researchers gain a better understanding of the online assessment experiences, prior to and post assessment. The premise is that the actions and reactions of the students are influenced by their own observations and experiences; these pre-existing attributes are determining factors of self-efficacy and awareness of learning. The study of lecturers as a parallel activity was conducted to determine if any mismatch exists between the expectations and perceptions of lecturers and those of students. 

Method

The longitudinal study commenced in April 2015 with a short pilot questionnaire and the main study commenced in October 2015. The research has been conducted within the domain of engineering mathematics for students in their first year of study of a Bachelor of Engineering programme. A mixed methods approach was designed to validate the findings from the study. Data was collected from the students in both countries simultaneously at the end of October in each year by means of a questionnaire. The questionnaire contained dichotomous questions, six-point Likert scale questions and two open questions. The cohort for this questionnaire was n = 186 in academic year 2015/2016, and n = 158 in 2016/2017, respectively. An Interpretative Phenomenological Analysis of the response to the open questions within the questionnaires formed the basis of qualitative group interviews with students (n = 50). The primary focus of the student interviews was students’ self-efficacy, confidence, and barriers, whilst engaging with online assessment. All student interviewees were asked the same questions to ensure comparability between responses. Utterances by the students were analysed and a schema developed for coding. The code was altered once a thematic change was detected. The topics were revisited for the second year of the study and found to be representative and relevant. A further group interview of students in Finland is planned for early 2017 to ensure balance within the data. In parallel with the student group interviews, a series of individual interviews were conducted with lecturers from Ireland (n = 6) and Finland (n = 2). The questions for this qualitative process were developed from the thematic outputs, derived from the student responses to questions, in the group interviews, and open questions from the questionnaire. The interview questions were formed around themes of confidence, barriers, and optimal assessment. A rigorous process of ethical scrutiny was engaged in between all Higher Education institutions involved in the research prior to the commencement of the research.

Expected Outcomes

Analysis of the responses, to the quantitative questionnaire, and qualitative interviews of students, reinforces the initial findings reported at ECER 2016; they were still pertinent in the second year within the longitudinal study. At the commencement of study, many first year engineering mathematics students reported feelings of low confidence and high levels of anxiety in relation to online assessment. The experience of online assessment for mathematics was new for the vast majority of students in both countries and this is reflected in the comments “I was a lot better than I thought I would be.” Evidence of anxiety came through in the interviews revealing that anxiousness was carried through from second level affecting self-confidence in the initial stages of study. Confidence levels improved as the students progressed to the second year of study however, the negative anxiety attributes transferred to other forms of assessment outside of the scope of this study. “I am frustrated and panicked because I feel that I am all over the place.” Comments relating to negative experiences, linked to online assessment, reduced considerably, from 28% in year 1 of study to 8% in year 2, as students became more familiar with techniques. Issues raised, relating to the current online assessment system, continue to demonstrate that students consider the system to be deficient in providing a fully objective analysis of students’ performance, and ability, because partial credits are not provided for (Ashton et al, 2006). The challenge of abstract questioning is an area to be addressed in the second stage of this research project in both institutions.

References

Artino, A.R. & Jones, K.D., 2012, Exploring the complex relations between achievement emotions and self-regulated learning behaviors in online learning, The Internet and Higher Education, 15(3), pp. 170-5 Ashton, H.S., Beevers, C.E., Korabinski, A.A. & Youngson, M.A., 2006, Incorporating partial credit in computer-aided assessment of mathematics in secondary education, Br J Educ Technol, 37 (1), pp. 93 – 119. Bandura, A., 1977, Self-efficacy: Toward a Unifying Theory of Behavioral Change, Psychological Review, 84(2), pp. 191 – 215. Cole, J.S., McCartan, C.D., Tuohi, R. & Steinby, P., 2014, Mathematics background of engineering students in Northern Ireland and Finland, Proceedings of 10th International CDIO Conference, Universitat Politecnica de Catalunya, Barcelona, Spain, June 16 – 19. Gallimore, M. & Stewart, J., 2014, Increasing the impact of mathematics support on aiding student transition in higher education, Teaching Mathematics and its Applications, 33(2), pp. 98-109. Jordan, S., 2013, E-assessment: Past, present and future, New Directions, 9(1), pp. 87-106. Marchand, G.C. & Gutierrez, A.P., 2012, The role of emotion in the learning process: Comparisons between online and face-to-face learning settings, The Internet and Higher Education, 15(3), pp. 150-60 McCraith, B., 2015, Average is no longer good enough – it’s time for a step change in STEM education in Ireland, in Education Matters, Yearbook 2016 – 2016, pp. 13-18. Narciss, S., Sosnovsky, S., Schnaubert, L., Andrès, E., Eichelmann, A., Goguadze, G. & Melis, E., 2014, Exploring feedback and student characteristics relevant for personalizing feedback strategies, Computers & Education, 71, pp. 56-76 Passmore, T. & Brookshaw, L., 2011, A flexible extensible online testing system for mathematics, Australasian Journal of educational Technology, 27(6), pp. 896 – 906 Prendergast, M., Breen, C., Carr, C. & Faulkner, F., 2016, Investigating Third Level Lecturers' Awareness of Second level Curriculum Reform, 13th International Congress on Mathematical Education. Rinneheimo, K., 2010, Methods for teaching mathematics case Tampere University of Applied Sciences, 1st intl workshop on maths and ICT: Education, Research and Applications, Bucharest, pp. 48-55. Sangwin, C., 2012, Computer Aided Assessment Of mathematics Using STACK, 12th International Congress on Mathematical Education, South Korea. Tempel, T. & Neumann, R., 2014, Stereotype threat, test anxiety, and mathematics performance, Social Psychology of Education, 17(3), pp. 491-501. Tempelaar, D.T., Niculescu, A., Rienties, B., Gijselaers, W.H. & Giesbers, B., 2012, How achievement emotions impact students' decisions for online learning, and what precedes those emotions, The Internet and Higher Education, 15(3), pp. 161-9.

Author Information

Ken Brown (presenting / submitting)
Letterkenny Institute of Technology
Letterkenny
University of Glasgow
Education
Glasgow
Oulu University of Applied Sciences, Finland

Update Modus of this Database

The current conference programme can be browsed in the conference management system (conftool) and, closer to the conference, in the conference app.
This database will be updated with the conference data after ECER. 

Search the ECER Programme

  • Search for keywords and phrases in "Text Search"
  • Restrict in which part of the abstracts to search in "Where to search"
  • Search for authors and in the respective field.
  • For planning your conference attendance, please use the conference app, which will be issued some weeks before the conference and the conference agenda provided in conftool.
  • If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.