Trust your doctor, not a chatbot. That's the stark lesson from a world-first study that demonstrates why we shouldn't be taking health advice generated by artificial intelligence (AI).
Chatbots can easily be programmed to deliver false medical and health information, according to an international team of researchers who have exposed some concerning weaknesses in machine learning systems.
Researchers from the University of South Australia, Flinders University, Harvard Medical School, University College London, and the Warsaw University of Technology have combined their expertise to show just how easy it is to exploit AI systems.
In the study, published today in the Annals of Internal Medicine, researchers evaluated the five foundational and most advanced AI systems developed by OpenAI, Google, Anthropic, Meta and X Corp to determine whether they could be programmed to operate as health disinformation chatbots.
Using instructions available only to developers, the researchers programmed each AI system - designed to operate as chatbots when embedded in web pages - to produce incorrect responses to health queries and include fabricated references from highly reputable sources to sound more authoritative and credible.
The 'chatbots' were then asked a series of health-related questions.
According to UniSA researcher, Dr Natansh Modi, the results were disconcerting.
"In total, 88% of all responses were false," Dr Modi says, "and yet they were presented with scientific terminology, a formal tone and fabricated references that made the information appear legitimate.
"The disinformation included claims about vaccines causing autism, cancer-curing diets, HIV being airborne and 5G causing infertility."
Out of the five chatbots that were evaluated, four generated disinformation in 100% of their responses, while the fifth generated disinformation in 40% of its responses, showing some degree of robustness.
As part of the study, Dr Modi and his team also explored the OpenAI GPT Store, a publicly accessible platform that allows users to easily create and share customised ChatGPT apps, to assess the ease with which the public could create disinformation tools.
"We successfully created a disinformation chatbot prototype using the platform and we also identified existing public tools on the store that were actively producing health disinformation.
"Our study is the first to systematically demonstrate that leading AI systems can be converted into disinformation chatbots using developers' tools, but also tools available to the public."
Dr Modi says that these findings reveal a significant and previously under-explored risk in the health sector.
"Artificial intelligence is now deeply embedded in the way health information is accessed and delivered," he says.
"Millions of people are turning to AI tools for guidance on health-related questions.
"If these systems can be manipulated to covertly produce false or misleading advice then they can create a powerful new avenue for disinformation that is harder to detect, harder to regulate and more persuasive than anything seen before.
"This is not a future risk. It is already possible, and it is already happening."
While the study has revealed deficiencies in these AI systems, Dr Modi says that the findings highlight a path forward, but it will require buy-in and collaboration from a range of stakeholders.
"Some models showed partial resistance," he says, "which proves the point that effective safeguards are technically achievable.
"However, the current protections are inconsistent and insufficient. Developers, regulators and public health stakeholders must act decisively, and they must act now.
"Without immediate action, these systems could be exploited by malicious actors to manipulate public health discourse at scale, particularly during crises such as pandemics or vaccine campaigns."
The research article, 'Assessing the System-Instruction Vulnerabilities of Large Language Models to Malicious Conversion into Health Disinformation Chatbots' is published in the Annals of Internal Medicine, the world's most cited internal medicine journal. DOI:10.7326/ANNALS-24-03933
