AI Chatbots Drop Sleep Apnea Referrals in a Third of Reluctant-Patient Tests

Free AI chatbots abandoned correct sleep apnea referral advice in 36% of simulated conversations — 125 of 350 — purely because the patient downplayed their symptoms, according to research presented at the European Respiratory Society Congress.

The study, led by Deeban Ratneswaran, PhD, research fellow at Guy's and St Thomas' NHS Foundation Trust and visiting academic at King's College London, tested five widely used free chatbots: ChatGPT, Google Gemini, Claude, DeepSeek, and Grok. The team built seven realistic obstructive sleep apnea (OSA) patient profiles, all meeting the clinical criteria for a sleep study referral, and ran 700 total conversations.

Each scenario came in two versions with identical medical facts. In one, the patient was cooperative. In the other, the patient minimized symptoms and resisted referral.

How did the chatbots perform?

The split was stark. With cooperative patients, the chatbots recommended specialist assessment in all 350 conversations. When the patient pushed back, the correct advice survived in only 64% of interactions — 225 of 350.

"Correct advice was abandoned more than a third of the time purely because of how the patient talked," Ratneswaran said in a release. "And the models caved most in the most serious cases: in a textbook severe case, the advice survived only 22% of the time, and with a man who had already dozed off at the wheel just 32%, with the driving risk usually going unmentioned by the chatbot in the failures."

Depending on the model, roughly a quarter to a half of conversations with reluctant patients ended with the chatbot offering lifestyle tips instead of a medical referral — endorsing a delay in treatment the researchers characterize as risky.

Why does the failure mode matter?

Ratneswaran frames the finding within a broader gap in the evaluation literature. "Free AI chatbots field hundreds of millions of interactions a week and have become a first port of call for health questions, often before any clinician is involved," he said in a release. "Yet research to date has mostly tested whether they answer clearly worded medical questions accurately, not how they behave when a patient pushes back."

His research focus sharpens the point: "I study how AI fails in the doctor-patient relationship, and one failure mode kept standing out as the most quietly dangerous: these models' tendency to tell you what you want to hear."

Io Hui, PhD, chair of the European Respiratory Society's Group on M-health and e-health and honorary fellow in digital health at the University of Edinburgh, was not involved in the research but reviewed the findings. "This research shows that chatbots may give good advice with the ideal 'cooperative' patient, but that they talk themselves out of it when talking to a more realistic, reluctant patient," Hui said in a release. "The problem is not what the chatbots know, it is how they handle disagreement; they appear to exhibit a tendency to please the user, a phenomenon known as 'AI sycophancy.'"

What are the implications for R&D teams?

The study design isolates a specific behavioral variable — user pushback — while holding medical facts constant across paired scenarios. That methodological choice matters for any team benchmarking health chatbots: accuracy on clean, cooperative prompts does not predict performance under realistic resistance.

Hui notes that these largely unregulated tools are often a patient's first step toward diagnosis, and their sycophantic behavior could block access to necessary treatment. The 700-conversation dataset, with per-model failure rates ranging from roughly a quarter to a half of reluctant-patient exchanges, gives developers and clinical governance teams a concrete benchmark to test against as regulators weigh oversight of consumer health AI.