Chatbots have come a long way in very little time. As few as four years ago, one of the most-often quoted reasons for low acceptance was that users perceived the conversational agents as unpleasant because they were too rigid in their conversation and, overall, just not human enough (cf. D’Onofrio & Stucki, 2020; Gnewuch et al., 2020). With the advent of large language models (LLMs) and generative artificial intelligence (AI), this has changed quite drastically: the new generation of chatbots is capable of human-like dialogue on any given topic. Although the potential of conversational agents as tools for digital teaching and learning has been discussed for decades (e.g., Kuhail et al., 2022; Hobert & Berens, 2020), this has given a new sense of urgency to the debate on chatbots in higher education.

While the possible advantages and pitfalls of chatbot use have sparked a vivid discussion among researchers, teachers, and policy-makers alike (c.f. Ilieva et al., 2023; Kuhail et al., 2022), one important question usually remains unasked until development is already well under way: What could—and should—a chatbot for higher education look like? Not with regard to its functionalities or the underlying algorithm, but considering the way it presents itself? Should a chatbot meant for higher education have a clear “identity”? And if so, how should it be designed and communicated in order to influence users’ attitudes and behavior in a positive way? If a more human-like chatbot appearance is the key to higher acceptance ratings, is Uncanny Valley the only effect to steer clear of? Or might too much humanness prove just as problematic as too little?

Our article aims to provide answers to these questions. In order to help educators and policy-makers evaluate design choices and make informed decisions, we have developed an analysis grid for chatbot identities that focuses on three main factors: anthropomorphism, gender, and overall coherence of the represented chatbot identity. To this end, we combine educational research with empirical findings from a variety of fields such as marketing, tourism, and the service industry, in which conversational agents have already been studied for many years. We use these findings to add a fourth step to the chatbot design framework presented by Kaiser et al. (2019): using the analysis grid to identify and evaluate possible consequences of particular design choices.

As a case study for the application of this analysis grid, we use the LLM-based chatbot “Hansi”. This particular chatbot was created as part of the “HAnS” project (Schmohl et al., 2023), the aim of which is to develop a digital assistance system that uses artificial intelligence to support German students in higher education. As one might suspect, “Hansi” was eventually named after the system in which it is embedded—but this was neither the developers’ first choice nor, as we would argue, an ideal fit. In our article, we show how this chatbot persona was analyzed using our grid and how options for a redesign were evaluated against the backdrop of the HAnS project’s brand and educational goals.

This article is based on the chatbot design framework presented by Kaiser et al. (2019), which we have developed into an analysis grid for the critical evaluation of AI-based chatbots meant to be used in higher education. The main focus of this analysis is the chatbots’ “identities”, which are not only conveyed by the manner in which the applications interact with its users, but also via additional markers, such as names, voices, and avatars. Kaiser et al. (2019) suggest that before developers start to work on the more technical aspects of a chatbot, they should follow a three-step process to create an identity for the application. For this, the chatbot should first be understood as a brand ambassador for the institution that will develop and/or use it. This means that the chatbot should act in accordance with the core values of this institution. In a second step, developers should then decide which human attributes they want users to associate with the institution, and develop a matching persona. In a third step, they should then consider the context in which the conversation between the chatbot and its users will take place: Who is going to interact with this program and under which circumstances? Once these parameters have been set, the chatbot’s signature style of communication can be designed accordingly. We now aim to adapt this method for higher education by adding another step: Critical reflection on the resulting chatbot identity. Drawing upon educational research, as well as studies from fields such as marketing, tourism, and the service industry, in which virtual agents’ identities have already been discussed and evaluated for years, we have created an analysis grid. This grid, presented as an easy-to-use flowchart, illustrates the advantages and drawbacks that different chatbot identities may entail, especially with regard to anthropomorphism and gender. Once a chatbot identity has been created as proposed by Kaiser et al. (2019), this flowchart can be used to evaluate the possible impact of individual design aspects on the way users perceive the chatbot and interact with it. Using the chatbot “Hansi” – which was developed as part of the abovementioned HAnS project – as a case study, we show how educators and policy-makers in higher education can use this analysis grid to gain insight into how a chatbot’s design may influence its users.

Our article presents two main findings. A) Chatbot identities can make a great difference. The studies we collated suggest that factors such as anthropomorphism and gender determine how useful, trustworthy, and approachable an application appears to users. At the same time, however, these factors may prove exceedingly problematic. If an anthropomorphic identity is perceived as especially trustworthy, does that also mean that users will be less likely to critically evaluate the chatbot’s output? And although a female chatbot may be perceived as approachable—would that not also be affirming gender stereotypes? If institutions of higher education are to successfully integrate chatbots into their processes, design solutions need to be evaluated as thoroughly as possible. Regarding our case study, for example, we come to the conclusion that the chatbot “Hansi” should be re-designed to compensate for both a lack of identity and a surplus of perceived trustworthiness. B) Further research is of great importance. With the analysis grid presented in this article we aim to provide both developers and decision-makers in higher education an overview of the aspects of a chatbot’s identity that may influence users’ learning processes. Studies on these factors, however, are few and far between, especially those which focus on higher education. To guide decision-making processes, these scarce findings must therefore first be collated and compared. The analysis grid and corresponding flowchart are our solution to this problem–albeit a temporary one. Most of the studies on chatbot identities we included are exploratory and/or focus on the non-AI-based chatbots that made up the majority of conversational agents until the release of GPT-3. Therefore, our flowchart will soon need to be adapted to findings regarding generative AI. In the meanwhile, however, it may provide a starting point for institutions of higher education that are currently deliberating chatbot use.

Bastiansen, M. H. A., Kroon, A. C. & Araujo, T. (2022). Female chatbots are helpful, male chatbots are competent? Publizistik, 67, 601–623. Borau, S., Otterbring, T., Laporte, S., & Fosso Wamba, S. (2021). The most human bot: Female gendering increases humanness perceptions of bots and acceptance of AI. Psychology & Marketing, 38(7), 1052–1068. Cai, D., Li, H. & Law, R. (2022). Anthropomorphism and OTA chatbot adoption: a mixed methods study. Journal of Travel & Tourism Marketing, 39(2), 228–255. Chae, S. W., Lee, K. C. & Seo, Y. W. (2016). Exploring the Effect of Avatar Trust on Learners’ Perceived Participation Intentions in an e-Learning Environment. International Journal of Human–Computer Interaction, 32(5), 373–393. D’Onofrio, S. & Stucki, T. (2023). Chatbots – Grundlagen, Funktionsweise und Klassifikations- und Gestaltungsmerkmale. In S. D'Onofrio & S. Meinhardt (eds.), Robotik in der Wirtschaftsinformatik (pp. 25–39). Springer Vieweg. Gnewuch, U., Feine, J., Morana, S. & Maedche, A. (2020). Soziotechnische Gestaltung von Chatbots. In E. Portmann & S. D'Onofrio, (eds.), Cognitive Computing (pp. 169–189). Springer Vieweg. Hobert, S., & Berens, F. (2020). Chatbot-basierte Lernsysteme als künstliche Tutoren in der Lehre. Datenschutz und Datensicherheit, 44, 594–599. Ilieva, G., Yankova, T., Klisarova-Belcheva, S., Dimitrov, A., Bratkov, M., & Angellov, D. (2023). Effects of Generative Chatbots in Higher Education. Information, 14(9), 492. Kaiser, M., Buttkereit, A. F. & Hagenauer, J. (2019). Journalistische Praxis: Chatbots. Automatisierte Kommunikation im Journalismus und in der Public Relation. Springer VS. Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2022). Interacting with educational chatbots: A systematic review. Education and Information Technologies, 28, 973–1018. Schmohl, T., Schelling, K., Go, S., Freier, C., Hunger, M., Hoffmann, F., Helten, A.-K., & Richter, F. (2023). Combining NLP, speech recognition, and indexing: an AI-based learning assistant for higher education. The Future of Education, 13. Yang, Y., Liu, Y., Lv X, Ai J. & Li, Y. (2022). Anthropomorphism and customers’ willingness to use artificial intelligence service agents. Journal of Hospitality Marketing & Management, 31(1), 1–23. Zheng, T., Duan, X., Zhang, K., Yang, X., Jiang, Y. (2023). How Chatbots’ Anthropomorphism Affects User Satisfaction: The Mediating Role of Perceived Warmth and Competence. In Y. Tu & M. Chi (eds.), E-Business. Digital Empowerment for an Intelligent Future. Lecture Notes in Business Information Processing 481 (pp. 96–107). Springer.