PISCATAWAY, NJ – When trained correctly on medically accurate information, ChatGPT can provide trustworthy information for pregnant women seeking medical advice for treating opioid use disorder, according to new research in the Journal of Studies on Alcohol and Drugs, based at Rutgers University.
The research might one day lead to apps and websites for people seeking help for conditions that they may not want to discuss face to face with a health care provider.
"Seeking health advice online is a common practice that seems to be accelerating with the use of generative AI chatbots," said the study's lead author, Drew Herbert, of the Sinclair School of Nursing at the University of Missouri. "For a condition as complex and time sensitive as opioid use disorder in pregnancy, this creates a real sense of urgency, as inaccurate information or inaction can be immediately harmful and could have long-term consequences."
For their study, researchers led by Drew Herbert, of the Sinclair School of Nursing at the University of Missouri, provided ChatGPT with prompts telling the large language model to act like an experienced clinician who provides medication-assisted treatment for opioid use disorder in pregnancy. The researchers told the large language model to base its answers off of resources and guidelines from well-established medical organizations, including the American Society of Addiction Medicine, the Substance Abuse and Mental Health Services Administration, and the American College of Obstetrics and Gynecology.
The authors then developed a persona of a pregnant woman named "Jade," who had an opioid use disorder. For their purposes, Jade would be seeking information about getting treatment. The researchers then used this persona to frame a conversation with ChatGPT, writing things such as, "I am 6 weeks pregnant, and I cannot get off of heroin," and "How can I find a doctor? I live outside of Denver."
Herbert and colleagues had 30 separate conversations in that same vein with ChatGPT. They then took the conversations and scored them on a pre-defined rubric.
They found that nearly 97% of the responses were safe, accurate, and relevant, including descriptions of what medications might be prescribed and details about finding a local doctor. Only three responses were scored as irrelevant or inaccurate.
"The most surprising aspect was the consistency with which it provided information aligned with accepted clinical practice," said Herbert. "Its level of accuracy far exceeded our initial expectations."
The authors caution, however, that ChatGPT needed to be provided with parameters beforehand and that general requests for similar information may not provide advice that's as medically accepted.
"Our goal is not necessarily to build something entirely new, but to determine how we can better and more safely leverage this powerful emerging technology," Herbert said. "Further prompt engineering and fine-tuning are certainly needed, as is additional testing, including, eventually, field-based testing."
-----
Herbert, D., Westendorf, J., Farmer, M., & Reeder, B. (2025). Generative AI-derived information about opioid use disorder treatment during pregnancy: An exploratory evaluation of GPT-4's steerability for provision of trustworthy person-centered information. Journal of Studies on Alcohol and Drugs, 86(6), 894–905. https://doi.org/10.15288/jsad.24-00319
