Do chatbots have a role in higher education?
It's a question Joshua Lambert, an associate professor and biostatistician in the University of Cincinnati College of Nursing, is pondering. He's turned to a group of his students to find out their thoughts about the helpfulness and satisfaction of a custom AI education chatbot .
Lambert piloted his custom chatbot by examining how a small group of Doctor of Nursing Practice (DNP) students evaluated answers to a set of questions from three different sources: a professor, a graduate assistant and a chatbot.
The results of the study have been published in the Journal of Nursing Education . This pilot project used a randomized, blinded, within-subjects comparison study with survey-based evaluation.
Seven doctoral students in the study submitted statistical questions related to their capstone projects and received blind responses from the professor, graduate assistant and chatbot. They rated each response on helpfulness, satisfaction and likelihood of use on a scale of one to five with the best being five. They then guessed which response came from the chatbot.
"Students first gave us their questions and then we gave them three responses back in a blinded and randomized fashion so students were unaware which response came from either the professor, graduate assistant or chatbot," explains Lambert. "The students ranked each response in terms of helpfulness, overall satisfaction and guessed which of the three responses came from the chatbot."
"The students rated the chatbot's response the highest in terms of overall satisfaction and helpfulness," adds Lambert.
The chatbot's responses were preferred by the students, but Lambert thought the data offered a more nuanced story than originally thought. He found that when students were asked to guess what response came from the chatbot, the lowest rated responses in helpfulness and satisfaction were guessed as coming from the chatbot.
"Students preferred the large language model (LLM) chatbot's responses when blinded yet demonstrated a bias against it when the source was suspected," explains Lambert. "This bias is likely rooted in a lack of trust, and trust may influence AI adoption by both students and professors.
"The students rated the chatbot's responses the highest yet consistently guessed the lowest rated response as the chatbot's was very interesting and somewhat unexpected. Yet when we read the current academic literature on this topic, we found that user trust is an important component in almost all AI research right now," says Lambert.
Other researchers in the study from the UC College of Nursing include Robyn Stamm, DNP, associate professor of clinical nursing; Shannon White, DNP, assistant professor in the doctor of nursing practice program; and Melanie Kroger-Jarvis, DNP, associate dean for graduate clinical learning programs. Bailey Martin, PhD, a postdoctoral research fellow at the University of Colorado Anschutz Medical Campus, is also a co-author of the study.
Researchers in the study acknowledge that while the small sample size is appropriate for a pilot study, it is insufficient for determining adequate effectiveness. They suggest that larger studies, replicated in multiple sites with additional qualitative and quantitative data are needed to thoroughly evaluate AI chatbot tools in nursing education and advising.
"For this reason, the descriptive results should be considered an initial 'first step' toward understanding how such a tool may assist in student learning and consultation," the researchers wrote in their study.
Lambert says he considered using the chatbot because students, like others, are sometimes hesitant to ask questions to another individual, particularly a professor, that might seem silly or make them appear as not so smart.
However, the chatbot won't judge you based on your questions, he adds.
"Sometimes the topics we cover are challenging or intimidating," says Lambert. "Educators want something that will lower the barrier so students can ask any questions they like."
Funding for the study came from an internal grant from the University of Cincinnati College of Nursing to support conference attendance, participant reimbursement and software fees. The authors of the study report no potential conflicts of interest.
Read the story on the UC website .
