Session Information
16 SES 09 B, Artificial Intelligence in Education
Paper Session
Contribution
This review aims to synthesize empirical research evidence on student’s use of ChatGPT in higher education, emphasizing pedagogical possibilities and addressing emerging threats and challenges. Chat Generative Pre-Trained Transformer (ChatGPT) swiftly gained prominence as an open-access tool in higher education since its introduction in November 2022. It has rapidly become widely used across various domains, including higher education. The use of ChatGPT is still an emerging area, with a surge in studies reflecting its widespread adoption. As higher education institutions grapple with the integration of ChatGPT, concerns and opportunities abound. This review focuses on understanding the impact dimensions on students' use of ChatGPT, particularly considering the evolving landscape of their learning processes.
While ChatGPT use is a relatively new practice, research into it is an emerging area for researchers. However, there are still several studies that have been published in such a short time, because of the substantial use of it across the world and in every domain of life including higher education institutions by teachers, students, and administrators. In our systematic review, we examine the impact dimensions on students’ use mainly because of the increasing concerns about how they use it and how this might influence their learning. Initial studies have also explored potential benefits of ChatGPT in language learning within higher education contexts (Baskara,2023). While educational technologies driven by artificial intelligence (AI) are progressively used to automate and provide support for various learning activities (Cavalcanti et al., 2021;), recent research has focused on the impact of ChatGPT, identifying challenges and opportunities in learning, but they have not examined this within the higher education sector (Lo, 2023).
The ongoing debate surrounding ChatGPT's use in higher education presents varying perspectives. These concerns and benefits create different perspectives where some argue for its use freely and suggest that graders need to create more critical assigned tasks that require personalized and contextualized examples and justifications which may not directly be generated by ChatGPT, while others argue against its use or its use with caution by students (Tlili et al., 2023). Also, many higher education institutions have started to apply restrictions or ban ChatGPT’s use by students in their updated policy documents. On the other hand, a review of media news articles on how ChatGPT use can disrupt students’ learning and teaching in universities also revealed that the sentiment in media news is on more into the negative discourse than a positive one, hence highlighting the public discussions and university responses on such controversies about academic integrity (Sullivan et al., 2023). There are also those who believe we need to add new components in the process of assessment including verbal exams where students demonstrate their verbal ability to present the assignment that they generate (Rudolph et al., 2023).
There are several issues that emerge in the first year of the use of ChatGPT reported and discussed in the published research. However, despite the increasing body of research on ChatGPT in higher education, there is no systematic review that provides a comprehensive overview of what research has found. Therefore, it is timely to present a consolidated overview of the impact dimensions of the ChatGPT’s use and the potential implications for higher education. More specifically, in this review, we sought answers to the following research questions:
RQ 1: What are the defining characteristics of empirical research on ChatGPT in higher education?
RQ 2: What pedagogical possibilities and insights can we gain from the students’ use of ChatGPT in the context of higher education?
Method
To address our research questions, we employed a systematic review approach, following guidelines by Page et al., (2022). The methodological framework guided our process, involving literature search, study identification, data extraction/study coding, study quality appraisal, and thematic analysis. The literature search, conducted on November 10th, 2023, targeted three databases—ERIC, Scopus, and Web of Science—chosen for their extensive coverage of educational studies. The search string, incorporated terms such as "chat generative pre-trained transform*" OR "gpt*" AND "higher education*" OR "universit*" OR "college*." The following inclusion and exclusion criteria were used to identify relevant studies: • Population: Students in higher education. • Concept: Students' use of ChatGPT. • Context: Higher education settings. • Types of studies: Primary research with data. • Publication language: Studies presented in full text in English. • Time of publication: Studies published after the introduction of ChatGPT in November 2022. Studies addressing other aspects, like performance testing or comparisons between teacher and ChatGPT feedback, were excluded. After eliminating duplicates, a two-stage screening process involved reviewing titles and abstracts, followed by full-text examination, with disagreements resolved through discussion. Using EPPI-Reviewer Web, the second author extracted information about each study, including characteristics such as country, research question, study design, research method, study informants, field of study, and study purpose. Findings were also extracted to identify common themes, and the third author reviewed and updated the extracted data for accuracy. Thematic analysis facilitated data synthesis and theme derivation. The analysis team, consisting of three authors, undertook a stepwise process, beginning with data extraction, followed by inductive coding, and subsequent theme generation through co-author discussions. Rigor was maintained through continuous challenge and validation of assumptions and potential biases by the third author. The Mixed Methods Appraisal Tool (MMAT; Hong et al., 2018) in EPPI Reviewer assessed the methodological quality of each included article. This tool, designed for various study types, involved screening questions and additional criteria for assessing quantitative, qualitative, and mixed-method studies. Ratings ('yes,' 'no,' or 'can't tell') were independently assigned by the second and third authors, with disagreements resolved through discussion. Studies with quantitative (randomized control trial), quantitative (non-randomized), and mixed-method designs were omitted from the MMAT's checklist as they were not present in the reviewed studies.
Expected Outcomes
Eight studies were identified through a comprehensive literature search in three databases in October 2023, employing various research designs. The analysis revealed four overarching themes: 1) promoting students' learning and skill development; 2) providing content and immediate feedback; 3) activating motivation and engagement; and 4) addressing ethical aspects of ChatGPT use. The results in our review show that ChatGPT might function as an effective tool to provide timely scaffolding by offering precisely enough assistance to empower students to eventually complete their tasks autonomously. Consistently, studies highlight positive impacts, acknowledging ChatGPT for improving writing skills, promoting personalized learning, and facilitating self-directed learning. ChatGPT's role in providing feedback is essential, offering real-time assistance to enhance writing and deepen understanding. This feedback enriches the teaching and learning experience, fostering connection. Findings indicate students view ChatGPT as a motivational tool, recognizing its role in minimizing affective barriers, reducing stress during assignments. Positive perceptions encourage usage, emphasizing teachers' role in enhancing perceived usefulness. However, concerns include potential ethical issues, plagiarism, unauthorized information ownership, and the risk of impeding creativity and critical thinking. Some studies express concerns about blind reliance, potentially slowing actual learning progress. The systematic review suggests practical implications. Clear guidelines, workshops, and ethical ChatGPT use promotion in higher education institutions are recommended. Essential training programs for students and teachers, emphasizing responsible use, are crucial. Redefining assessment policies, aligning with the assessment for learning approach and incorporating multiple evaluation points throughout the course, is advised. In conclusion, the systematic review recognizes the evolving landscape of ChatGPT's integration into higher education and aims to provide a consolidated overview of its impact dimensions and potential implications. By addressing critical research questions, the review endeavors to contribute valuable insights for higher education decision-makers and policymakers navigating the complex terrain of AI-driven tools in the educational landscape.
References
Baskara, R. (2023). Exploring the implications of ChatGPT for language learning in higher education. Indonesian Journal of English Language Teaching and Applied Linguistics, 7(2), 343-358. Cavalcanti, A. P., Barbosa, A., Carvalho, R., Freitas, F., Tsai, Y.-S., Gašević, D., & Mello, R. F. (2021). Automatic feedback in online learning environments: A systematic literature review. Computers and Education: Artificial Intelligence, 2, 100027. Hong, Q. N., Pluye, P., Fàbregues, S., Bartlett, G., Boardman, F., Cargo, M., Dagenais, P., Gagnon, M.-P., Griffiths, F., & Nicolau, B. (2018). Mixed methods appraisal tool (MMAT), version 2018. Registration of copyright, 1148552(10). Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. Page, M. J., Moher, D., & McKenzie, J. E. (2022). Introduction to PRISMA 2020 and implications for research synthesis methodologists. Research synthesis methods, 13(2), 156-163. Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342-363. Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15.
Search the ECER Programme
- Search for keywords and phrases in "Text Search"
- Restrict in which part of the abstracts to search in "Where to search"
- Search for authors and in the respective field.
- For planning your conference attendance you may want to use the conference app, which will be issued some weeks before the conference
- If you are a session chair, best look up your chairing duties in the conference system (Conftool) or the app.