In November of 2022, OpenAI released ChatGPT forcing academic communities to grapple with the use of generative AI in academic settings and especially on human interaction with these systems in writing and research. Generative AI is a category of artificial intelligence that is "programmed to emulate the creative and generative qualities of human thought" (Yusuf et al., 2024). It has also been described as "deep-learning models that can generate high-quality text, images, and other content based on the data they were trained on” (IBM, 2023). This includes, but is not limited to, what is commonly referred to as a large-language model (LLM).   

Artificial intelligence (AI) had been slow to impact higher education on a large scale in the same way that it had in the private sector, but with the release of ChatGPT and other LLMs this quickly changed. Generative AI systems have shaken the academic world by forcing us to wrestle with the ways we create content and view original work. Generative AI has demonstrated the opportunity to help educational researchers generate new research ideas (Arowosegbe et al., 2024; Chan & Hu, 2023), summarize key research articles, and synthesize ideas across publications (Schryen et al., 2024).  The capabilities of the LLMs that underlie Generative AI tools have evolved quickly, potentially changing the way that educational researchers conduct their work. Current literature on Generative AI for educational research is overwhelming theoretical, demonstrating a need for more empirical work in the field (Morande, 2023). This proposed presentation will share the results of an exploratory qualitative comparison of the processes and results from three different Generative AI tools (i.e., Avidnote, Scopus AI, and Elicit), compared to a human-written and peer-reviewed literature review as a control case. 

This study is investigating the following research questions: 

1. How does the ease of search compare between GenAI research tools?  

2. Is there consistency of cited sources across GenAI research tools?  

3. What is the quality of cited sources? 

4. How does each GenAI tool summarize information? How does each synthesize information? 

5. What are some potential pitfalls for educational researchers utilizing GenAI tools in their research? 

Scholarly Significance and Limitations 

Due to the fast-paced changes occurring within generative AI technologies, research in this area is not only timely but necessary to ensure both PhD students, higher educational faculty, and academic researchers are in the best position to navigate these technological changes and integrate them ethically into their research practices. This qualitative study is exploratory. It was designed to offer comparisons between existing GenAI research tools and offer guidance for the development of more formal comparative research. 

Methods This comparison study was designed to be exploratory. An exploratory study "consists of an attempt to discover something new and interesting, by working your way through a research topic" (Swedberg, 2020, p.17). Swedberg suggested exploratory studies are conducted to either investigate a topic not researched before or to further explore an already existent topic to create new lines of inquiry (p.18). While there has been an explosion of work related to Generative AI for research, it is still an emerging area and one that has heavily focused on theoretical work. Data Source Three current Generative AI research tools were investigated for this exploratory study. AvidNote (https://avidnote.com/), Scopus AI (https://www.elsevier.com/products/scopus/scopus-ai) , and Elicit (https://elicit.com/). AvidNote markets itself as a tool for reading papers, writing research, and analyzing data. Scopus AI was created by Elsevier. They highlight Scopus AI as a “responsible AI” tool that summarizes research and creates concept maps to discover emerging themes connected to the searched topic. Finally, Elicit pulls articles from Semantic Scholar. The product markets the tool’s ability to synthesize abstracts into a cohesive summary and creates a reference grid with information about the sources used. Data Analysis In December 2024, the researcher conducted a systematic literature review. The review is currently under final revision based on peer-review feedback and is due for publication this spring. The topic of this literature review, sources, and conclusions will be used as a control case for this exploratory research. A database has been created to track prompts, sources, and responses across the sample of three Generative AI tools. A descriptive comparative analysis will be finalized that details similarities and differences between the tools and their outputs. Finally, the researcher will compare the AI outputs to the soon-to-be-published literature review and write a narrative reflecting on potential benefits and drawbacks of using these Generative AI tools for research.

Preliminary results are thus far in line with past findings (Al-Zahrani, 2023; Morande, 2023). Overall, the GenAI tools investigated here offer great potential but still fall short of a magic bullet for research. All three tools were easy to use; however, all three did not have equal access. Scopus AI does not have a free or trial version, limiting the audience who has access to the tool. AvidNote and Elicit both offer at least basic free versions in which to allow researchers to explore. All three provided relevant sources – including a key foundational text – but did not offer as robust of citations as in the human written literature review. All three platforms did, however, offer helpful summaries of the literature cited and were easy to use as a new user. The current Generative AI research tools offer tremendous potential in inspiring new research ideas, linking disparate literature bases, and increasing efficiency in shifting through the large amount of published work. As GenAI technologies and platforms for research continue to rapidly evolve, increased empirical research to both understand these tools and prepare researchers to engage with them will be needed. More detailed findings will be presented if accepted.

Arowosegbe, A., Alqahtani, J. S., & Oyelade, T. (2024). Perception of generative AI use in UK higher education. Frontiers in Education, 9. https://doi.org/10.3389/feduc.2024.1463208 Al-Zahrani, A. M. (2023). The impact of generative AI tools on researchers and research: Implications for academia in higher education. Innovations in Education and Teaching International, 61(5), 1029–1043. https://doi.org/10.1080/14703297.2023.2271445 Chan, C.K.Y., & Hu, W. (2023). Students’ voices on generative AI: perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(43). https://doi.org/10.1186/s41239-023-00411-8 IMB. (2023). What is Generative AI? https://research.ibm.com/blog/what-is-generative-AI Jin, W., Cheng, Y., Shen, Y., Chen, W., & Ren, X. (2021). A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-language models. ArXiv Preprint ArXiv:2110.08484. Morande, S. (2023). Benchmarking generative AI: A comparative evaluation and practical guidelines for responsible integration into academic research. SSRN. http://dx.doi.org/10.2139/ssrn.4571867 Schryen, G., Marrone, M., & Yang, J. (2024). Adopting generative AI for literature reviews: An epistemological perspective. In HICSS 2024: Proceedings of the 57th Hawaii International Conference on System Science. https://hdl.handle.net/10125/106870 Swedberg, R. (2020). Exploratory Research. In C. Elman, J. Gerring, & J. Mahoney (Eds.), The Production of Knowledge: Enhancing Progress in Social Science (pp. 17–41). chapter, Cambridge: Cambridge University Press. Yusuf, A., Pervin, N., & Roman-Gonzalez, M. (2024). Generative AI and the future of higher education: A threat to academic integrity or reformation? Evidence from multicultural perspectives. International Journal of Educational Technology in Higher Education, (21).