Anyone who has used ChatGPT or another artificial intelligence chatbot has likely seen a disclaimer that its answers may be incorrect.
Large language models are built to generate the most probable response to a prompt. Even when they lack reliable information, they may produce an answer that sounds convincing rather than acknowledge uncertainty.
Researchers at the University of Miami College of Engineering are developing a way to identify when those systems may be operating beyond what they reliably know. The work could become particularly important as artificial intelligence is integrated into high-stakes fields.
"The importance of this is that in certain applications like medicine, health care and defense, you don't know whether the answer that the AI system is giving is correct or not," said Kamal Premaratne, professor of electrical and computer engineering. "Even a slight oversight in the output could be severe and very costly."
Measuring uncertainty
Earlier this year, Premaratne and doctoral candidate Pragatheeswaran Vipulanandan presented the research at the International Conference on Learning Representations in Rio de Janeiro. The work, conducted with Dilip Sarkar, associate professor of computer science, offers a new way to measure how much confidence users should place in an AI-generated response.
One current approach to measuring uncertainty is to ask a model the same question several times and compare its answers. When the responses vary substantially, it could mean that the model is unsure about its output.
The University team wanted to look beyond the answers themselves and understand how confident the model was in producing them.
Every response from a large language model is built from probabilities that guide how the system selects words and phrases. The researchers developed a mathematical framework that tests how stable those probabilities remain when even small changes are introduced. Large fluctuations may indicate that its response deserves a closer look.
"Right now, as soon as you ask an LLM a question, it gives you an answer rather than saying, 'I'm not sure,'" Vipulanandan said. "The ultimate goal is for the LLM to tell you, 'I'm not sure, but these are the top answers I have,' or even ask for more context before responding."
Accounting for what the model does not show
The researchers are also examining possible answers that an AI model does not produce.
They use a statistical technique known as the missing-species model, which considers possibilities that may exist beyond what has already been observed. Premaratne compares the idea to visiting a wildlife reserve.
"If you go to a park to see animals, you come out knowing how many giraffes, how many hippos and how many rhinos there are," he said. "But the fact that you didn't see a leopard doesn't mean there are no leopards in the park. It's just that you didn't see one."
In the same way, a large language model may contain other possible answers that never appear in its output. Accounting for those unseen possibilities may provide a more complete picture of the model's uncertainty.
Borrowing tools from quantum physics
To make those measurements possible, the researchers drew inspiration from quantum physics.
While the team is not building a quantum computer, they've adapted mathematical tools developed to measure uncertainty in complex systems. Those tools can help estimate how many outcomes may be possible, even when researchers can observe only some of them.
Human judgement still reins
The approach is not intended to eliminate AI hallucinations. Its purpose is to provide a clearer signal when an AI-generated response may be unreliable and should receive a second look.
As large language models become part of more decisions and everyday activities, Premaratne said human judgment must remain alongside their answers.
"The logical thinking comes when you actually know how to do the work on your own," he said, "not by simply parroting whatever ChatGPT is saying."
