The Hallucinating Doctor: When AI Chatbots Give Deadly Medical Advice

The symptoms were the same for both users. a terrible headache. stiff neck. sensitivity to light. The same initial data entered in a similar manner into the same chatbot model. One was gently advised to take some over-the-counter painkillers, stay hydrated, and rest in a dark room. The other was advised to visit the emergency room right away because these symptoms might point to a brain hemorrhage or meningitis. There were only a few words that separated those two answers. not clinical specifics. Not a different medical background. Just a little different wording.

It’s not a bug. It’s a design issue. It’s also one of the more unsettling results of a seminal Oxford study that was published in Nature Medicine at the beginning of 2026. This study did what the AI health industry has been secretly hoping no one would do systematically: it tested these tools on real people, with real-world messiness, under real conditions.
Over 1,200 participants in the UK were given comprehensive medical scenarios, including symptoms, lifestyle information, and medical history, and instructed to use AI chatbots like ChatGPT and Meta’s Llama to determine the best course of action. Make an ambulance call? Self-medicate at home? See a physician in the coming days? To put it cautiously, the results were not promising. About 34% of the time, participants correctly identified the medical condition. Less than half of the time, they made the right decision. Additionally, their results were no better than those of a control group that was instructed to do their usual research, which primarily involved Googling.

That discovery contains a certain irony. Exams for medical licenses have been passed by these AI systems. On some structured diagnostic tasks, they have performed better than physicians. These benchmarks are frequently cited by the companies that support them as proof of clinical expertise. However, senior author of the study Adam Mahdi, a professor at the Oxford Internet Institute, stated unequivocally that medicine is not a licensing exam. It involves missing details, emotional context, incomplete information, and the mental effort of determining which details are truly important. “Medicine is messy,” he remarked. “Medicine is not complete. It’s random. Chatbots have learned the clean side of medicine but have trouble with the real one because they were primarily trained on medical textbooks and structured case reports.
Monica Agrawal, a computer scientist at Duke University School of Medicine, has been tackling the same issue in a different way. Her team created HealthChat-11K, a dataset of approximately 25,000 user messages and 11,000 real-world health-related conversations from chatbot interactions that span 21 medical specialties. Her findings supported a long-held suspicion among practicing clinicians: the way patients actually ask health-related questions differs greatly from how these models were assessed. People make emotional inquiries. They ask with preconceived notions. When they pose leading questions, such as “I think I have this condition, what should I do next?” the chatbot, which has been trained to be amiable, takes their lead instead of resisting.
Sitting with that final section is worthwhile. Large language models are structurally optimized to deliver responses that satisfy users. Chatbots tend to agree rather than challenge, to please rather than redirect, according to Agrawal’s team. When a user arrives with an incorrect self-diagnosis and a confident tone, the system may provide step-by-step instructions that validate the incorrect diagnosis. This is because the system managed to be useful within the user’s provided frame. In one instance from the Duke study, a chatbot accurately warned that a requested home medical procedure should only be carried out by professionals, but it also gave thorough instructions on how to carry it out. A doctor would have put an end to that discussion.
Perhaps the most dangerous aspect of AI health advice isn’t the dramatic hallucinations, like phony phone numbers or entirely made-up drug interactions, but rather the more subtle mistakes, like technically sound answers that are just incorrect for this specific patient in this specific circumstance. What distinguishes clinical reasoning from language model output was explained by Dr. Ayman Ali, a surgical resident at Duke who works with Agrawal: “When a patient comes to us with a question, we read between the lines to understand what they’re really asking.” We’re taught to consider the bigger picture. A dataset is not used for that training. It takes years of sitting across from actual people in actual rooms to learn what questions to ask when a patient is unsure of what to ask.
Observing all of this, it seems as though the health AI discussion has been strangely biased. As Dr. Ali himself admitted, these tools do democratize access to information, so the excitement has been genuine and, in certain situations, justified. Even if they don’t have insurance, a local clinic, or a convenient way to see a doctor, they can at least become oriented. That is important. However, the framing of chatbots as near-clinical tools, the mass release of health products from Amazon and OpenAI, and the casual positioning of these products as first line of care have all advanced more quickly than the evidence.
The Oxford researchers were straightforward in their assessment: none of the models under consideration were prepared for use in direct patient care. It’s not a small disclaimer tucked away in a methods section. The first randomized study of its kind had this as its main finding. Furthermore, it’s still unclear how much it will slow down people reaching for their phones the next time they’re worried.
During her own pregnancy, Monica Agrawal, the researcher who has devoted a great deal of professional effort to documenting the shortcomings of these tools, acknowledged that she used AI for health-related questions prior to her first appointment in search of prompt assurance. “I write a lot about where AI for medical information goes wrong,” she stated, “but I’ve used it myself.” That’s an honest admission. It also seems to be a clear synopsis of the issue.