1. Introduction

The study explores student engagement in collaborative learning with AI agents in a large language model (LLM)-empowered learning environment. The rapid advancement of LLMs has significantly reshaped the landscape of online education, particularly in the design and implementation of learning environments. Traditional Massive Open Online Courses (MOOCs) have long provided learners with accessible and self-paced educational resources, yet they often fall short in delivering the dynamic interactions and personalized guidance necessary for deep learning and skill development. The integration of LLMs into these environments presents a transformative opportunity to address these limitations. As a result, student learning behaviors and characteristics are evolving, becoming more visible through real-time dialogue interactions rather than traditional log-based data. Dialogue not only captures learners’ actions but also reveals their cognitive processes, decision-making strategies, and overall learning approaches. Analyzing these interactions offers deeper insights into different learner types and engagement patterns, enabling a reclassification of learning modes and providing a more nuanced understanding of educational experiences in AI-enhanced environments.

2. Research Questions

This study aims to investigate the impact of learner characteristics on engagement and outcomes in AI-empowered collaborative learning environments. Specifically, we explore how students interact with AI agents in multi-agent settings, identifying patterns in learner behaviors and their effects on cognitive and social dimensions of learning. By leveraging data-driven approaches, we seek to refine existing learner typologies in the context of AI-enhanced education. The study further aims to provide insights into optimizing student-AI collaboration for improved self-regulated learning, creativity, and knowledge construction.

The following research questions will be addressed by this study:

1. How can clustering and classification methods help in identifying distinct learner types in the student-AI collaborative learning environment?
2. What behavioral patterns emerge in dialogue-based interactions between different types of students and AI agents, and how do they relate to learning outcomes?

3. Conceptual or Theoretical Framework 

The study is grounded in theories of collaborative learning, self-regulated learning, and AI-mediated interaction. Drawing from constructivist perspectives (Vygotsky, 1978; Piaget, 1950), we view learning as an active process, where students co-construct knowledge through interaction with AI agents, extending traditional human-human collaboration frameworks. The research is also informed by social constructivism theories (Bruffee, 1999; Dillenbourg, 1999), which emphasizes the role of dialogue and peer interaction in knowledge acquisition, now adapted to include AI as a collaborative partner. Given recent empirical findings on the role of AI in fostering metacognition and creativity (Yan et al., 2024; Kim & Lee, 2022), we examine how different learner characteristics interact with AI affordances to shape cognitive and behavioral regulation from a self-regulated learning (SRL) perspective (Zimmerman, 2002).

Furthermore, we draw upon learning analytics and clustering methodologies (Wang et al., 2021; Liu et al., 2023) to identify distinct learner profiles, learner types or learning styles based on interaction data, behavioral transitions, and engagement patterns. Researchers have examined the relationship between different learner types and learning outcomes, revealing how distinct behavioral patterns influence academic performance and engagement (Li et al., 2022; Yang et al., 2022). These findings provide a foundation for tailoring instructional strategies to different learner profiles, enabling more personalized and effective learning experiences.

By integrating these theoretical perspectives, the study aims to provide a holistic understanding of student-AI collaborative learning, bridging the gap between traditional collaborative learning models and emerging AI-driven educational paradigms.

1. Settings and participants This study utilizes the Massive AI-empowered Course (MAIC) system, an online learning infrastructure underpinned by LLM-based multi-agent architecture (Yu et al., 2024). This system is comprised of six integral interaction agents – the AI Teacher, the AI Teaching Assistant, and four AI Student roles. The system also has backend agents who undertake enormous responsibilities such as managing the class progression and facilitating seamless interaction between AI students and the AI teacher. The study involved 305 participants (34.75% of whom were female, with a mean age of 20.16 years, SD=1.26), from a prestigious university in China. These participants were enrolled in an introductory course of General Artificial Intelligence provided on the MAIC platform. The course was organized into six modules, which the students had to complete within a span of two months. Out of the initial population, only 110 students, constituting roughly 30.91% females and a mean age of 19.96, SD=1.18, completed the entire course. 2. Data collection The study collected pre-course and post-course questionnaires, pre- and post-tests, and student-AI interaction dialogues. Questionnaires gathered demographic information, personality traits (using the Big Five Inventory), non-cognitive skills (academic self-efficacy and self-regulated learning), and attitudes toward AI. All measures used established, reliable scales with high Cronbach's α values. Pre- and post-tests assessed students’ prior knowledge and course content understanding. The learning was self-paced, with quizzes at the end of each module and a final exam at the end of the course. Interaction-dialogues were recorded by the MAIC system and coded using a well-established framework on the cognitive, behavioural and emotional nature of each data entry. 3. Data analysis An auto-encoding framework that categorizes messages into behavior, cognition, and emotion dimensions is utilized to analyze interaction dialogues. This framework builds on the Flanders Interaction Analysis System, Bloom's Taxonomy, and Russell's circumplex model of affect. With the help of GPT-4 Turbo and few-shot prompting techniques, the study achieved high inter-rater reliability in coding dialogues. Descriptive analysis was conducted to extract interaction metrics, and clustering analysis, using hierarchical clustering, identified distinct student engagement patterns. Variables included interaction metrics, Big Five personality traits, non-cognitive skills, attitudes towards AI, and pre-test scores. Epistemic Network Analysis (ENA) was applied to explore differences among student clusters and examine their social-cognitive engagement. The study utilized Python for variable aggregation and R studio for clustering and ENA.

A hierarchical cluster analysis identified three student clusters (N = 110) based on votes from 23 metrics, with significant differences in message count and length (χ²(2, N = 110) = 38.52, p < .001; χ²(2, N = 110) = 46.09, p < .001). Cluster 3 (n = 14) did not engage in interactions, while Clusters 1 (n = 63) and Cluster 2 (n = 33) differed in message length (t(94) = 2.29, p = .012) but not message count (t(94) = 0.49, p = .313). Further analysis of the nature of message revealed that Cluster 1 engaged more in cognitive activities, including Remember & Understand (z = 5.81, p < .001), Apply (z = 2.57, p = .010), and Analyze, Evaluate & Create (z = 2.63, p = .008), while Cluster 2 exhibited higher proportions of regulatory behaviors (e.g., process regulation: z = 3.54, p < .001) and off-topic messages (z = 6.64, p < .001). Based on the results of the cluster analysis, we classified the students into three categories: Cluster 1: Active Questioner; Cluster 2: Responsive Navigator; and Cluster 3: Silent Observer. ENA results indicated distinct interaction patterns, with Cluster 1 associating more with questioning and knowledge comprehension, while Cluster 2 showed a stronger connection to emotion-sharing and process-regulating. A two-sample t-test on the ENA model confirmed significant differences along the primary axis (t(42.88) = 9.08, p < .001, Cohen’s d = 2.30). The ANOVA on learning outcomes showed no significant differences in final exam scores (F(2,72) = 0.10, p = .91) among three clusters. Compared to Cluster 2, students in Cluster 1 experienced a significant decrease in self-regulated learning (z = -2.33, p = .01) and an increase in attitudes toward AI (z = 2.16, p = .03).

Bloom, B.S. (Ed.), Engelhart, M.D., Furst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy of educational objectives: Handbook I: Cognitive domain. New York: David McKay. Bruffee, K. A. (1999). Collaborative learning: Higher education, interdependence, and the authority of knowledge. Johns Hopkins University Press, 2715 North Charles Street, Baltimore, MD 21218-4363. Dillenbourg, P. (1999). What do you mean by collaborative learning?. Collaborative-learning: Cognitive and computational approaches., 1-19. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of educational research, 74(1), 59-109. Kim, J., & Lee, S.-S. (2022). Are Two Heads Better Than One?: The Effect of Student-AI Collaboration on Students’ Learning Task Performance. TechTrends : For Leaders in Education & Training, 67, 1–11. https://doi.org/10.1007/s11528-022-00788-9 Le Quy, T., Friege, G., Ntoutsi, E. (2023). A Review of Clustering Models in Educational Data Science Toward Fairness-Aware Learning. In: Peña-Ayala, A. (eds) Educational Data Science: Essentials, Approaches, and Tendencies. Big Data Management. Springer, Singapore. https://doi.org/10.1007/978-981-99-0026-8_2 Lee, G.-G., Mun, S., Shin, M.-K., & Zhai, X. (2024). Collaborative Learning with Artificial Intelligence Speakers. Science & Education. https://doi.org/10.1007/s11191-024-00526-y Liu, S., Kang, L., Liu, Z., Fang, J., Yang, Z., Sun, J., Wang, M., & Hu, M. (2023). Computer-supported collaborative concept mapping: The impact of students’ perceptions of collaboration on their knowledge understanding and behavioral patterns. Interactive Learning Environments, 31(6), 3340–3359. https://doi.org/10.1080/10494820.2021.1927115 Tan, S. C., Chen, W., & Chua, B. L. (2023). Leveraging generative artificial intelligence based on large language models for collaborative learning. Learning: Research and Practice, 9(2), 125–134. https://doi.org/10.1080/23735082.2023.2258895 Tan, S. C., Lee, A. V. Y., & Lee, M. (2022). A systematic review of artificial intelligence techniques for collaborative learning over the past two decades. Computers and Education: Artificial Intelligence, 3, 100097. https://doi.org/10.1016/j.caeai.2022.100097 Venkatesh, Viswanath & Thong, James & Xu, Xin. (2012). Consumer Acceptance and Use of Information Technology: Extending the Unified Theory of Acceptance and Use of Technology. MIS Quarterly. 36. 157-178. 10.2307/41410412. Vygotsky, L. (2019). Collaborative learning. Collaboration, communications, and critical thinking: A STEM-inspired path across the curriculum, 43. Yang, H., Alozie, N., & Rachmatullah, A. (2022). Collaboration at Scale: Exploring Member Role Changing Patterns in Collaborative Science Problem-solving Tasks. Proceedings of the Ninth ACM Conference on Learning @ Scale, 309–312. https://doi.org/10.1145/3491140.3528319 Zimmerman, B. J., & Schunk, D. H. (Eds.). (2011). Handbook of self-regulation of learning and performance. Routledge/Taylor & Francis Group.