Large language models (LLMs), such as ChatGPT and Claude, are increasingly being used to provide health information to the public. Although this is beneficial for improving access to treatment information, their limitations were revealed by a new study that found LLMs were biased in their treatment choices and using these outputs in practice could drive up treatment costs.
A research team led by Professor Daiju Ueda at Osaka Metropolitan University's Graduate School of Medicine in collaboration with The University of Texas' MD Anderson Cancer Center asked five LLMs (GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Flash, Llama 4, and MedGemma) to recommend up to three FDA-approved drug products for a range of simulated scenarios.
Although an average of 5.5 FDA-approved products were available for each specified active ingredient, a single product accounted for an average of 97.5% of first-choice recommendations across the tested scenarios and models.
"The recommendations were mostly for brand-name drugs, those highly visible in Google searches, and treatments with dedicated Wikipedia pages," Professor Ueda said.
The LLMs also tended not to recommend products with later FDA approval and those more commonly prescribed in actual physician practice. The study also identified recommendations that were inconsistent with regulatory information, including discontinued products.
The researchers then evaluated how much this could potentially cost patients using US data. They looked at a scenario where physicians' prescribing behavior was performed in accordance with the LLMs' recommendations.
"We found that annual acquisition costs would increase by approximately US$55.6 million," Professor Ueda summarized. "This represents a 1,019% increase in the cost of purchasing these prescriptions."
When instructed to prioritize lower-cost or generic products, the models shifted their recommendations accordingly; however, recommendations remained concentrated on a limited set of products.
"Our findings highlight the need to evaluate healthcare AI for biases in product selection and the handling of cost and regulatory information," Professor Ueda said. "Combining AI systems with trusted drug information sources and human oversight may help support more appropriate medication choices."
