When a person interacts with an artificial intelligence (AI)-powered chatbot, the bot can store details about the user and the interaction. During long conversations, this personalization feature can make the chatbot more likely to agree with users or mirror their views, according to researchers from the Penn State College of Information Sciences and Technology (IST) and the Massachusetts Institute of Technology (MIT).
This behavior, which often occurs among humans and is called sycophancy, can stop a chatbot from correcting a user when they are wrong, reducing the accuracy of its responses. When bots reflect a user's political beliefs or worldview, they can also spread misinformation and create an echo chamber that distorts the user's perception of reality.
For this study, the researchers observed real conversations between people and chatbots. Over two weeks, 38 participants used a chatbot in their daily lives, allowing the researchers to observe how five large language models (LLMs) - a type of AI used to build chatbots - behaved over extended interactions.
"We are using these models through extended interactions, and they have a lot of context and memory, but our evaluation methods are lagging behind," said Dana Calacci, assistant professor in the College of IST Department of Informatics and Intelligent Systems. "We wanted to evaluate LLMs in the ways people are actually using them to understand how they are behaving in the wild."
The researchers examined two situations: how LLMs respond to requests for personal advice and how they explain political topics. They found that conversation context made four of the five models more agreeable, even when doing so could reduce the accuracy of the bot's response. The biggest increase in agreeableness happened when the LLM stored a short user profile in its memory.
The researchers also found that mirroring a user's political views increased when the LLM could accurately infer those views from the conversation context - a behavior that may reinforce misinformation or create echo chambers.
"There is a lot we know about the benefits of having social connections with people who have similar or different viewpoints," Calacci said. "But we don't yet know about the benefits or risks of extended interactions with AI models that have similar attributes."
Although reducing sycophancy was not the main goal of the study, the researchers said potential solutions could include designing LLMs to better identify which details from a conversation or user memory matter. They could also be built to detect when they are agreeing too much and flag those responses. Another option is to give users more control over how models use personal context.
Calacci, who earned a doctoral degree in information technology from MIT in 2023, is also a faculty member of Penn State's Institute for Computational and Data Sciences. Her fellow researchers on this study included Matt Viana, a graduate student pursuing a doctoral degree in informatics from the College of IST, and MIT's Shomik Jain, Charlotte Park and Ashia Wilson.
The researchers will present their findings at the Association for Computing Machinery Conference on Human Factors in Computing Systems (ACM CHI), which will be held April 13-17 in Barcelona.
