- Why improving AI accuracy makes the danger harder to see, not easier
- How I found fabricated academic references inside my own published work
- Why most of what I once blamed on "AI being wrong" was actually me asking badly
- Why even a hallucination-free AI wouldn't fully solve this problem — and why that's the scarier part
- The exact habits I now use to keep myself from trusting AI more than it's earned
The biggest risk of AI in medical education is no longer frequent, obvious mistakes — it's increasing accuracy paired with decreasing verification. When AI was clearly unreliable, students checked everything out of habit. Now that it's usually right, checking starts to feel optional, which means the rare errors that do slip through are far more likely to be believed. The fix isn't using AI less; it's staying just as skeptical as when it used to fail constantly.
This morning I went looking for something specific: proof that the academic references scattered across my own blog were real. I had been citing the same handful of papers across nearly every article on NexoraMed, confident they were solid because they looked solid — proper DOI formatting, PMID numbers, journal names I recognized. Two of them were not what I thought. One PMID pointed to a demographics paper about fertility rates that had nothing to do with medicine or AI. The other DOI, formatted exactly like a real one, did not belong to any paper I could find.
I had read past those citations dozens of times without noticing, because they did not look wrong. That is the entire problem this article is about, and I am not writing it from the outside.
Where AI Actually Gets It Wrong — And Where It Almost Never Does
Not all AI mistakes carry the same weight, and not all AI tools fail at the same rate. Based on how I've actually used Claude, ChatGPT, Gemini, and DeepSeek across two years of medical school, the pattern has stayed consistent enough that I trust it:
| Category | Claude / ChatGPT | Gemini / DeepSeek |
|---|---|---|
| Abstract, self-contained topics (physiology, biochemistry, isolated mechanisms) |
Rarely wrong | Good, but noticeably more error-prone |
| Complex, multi-layered clinical scenarios (mixed presentations, overlapping systems) |
Same ranking holds, but errors rise sharply | Same ranking holds, errors rise even more |
| Drug dosing and clinical guidelines | Error rate jumps by a wide margin | Error rate jumps further still |
The relative order between the tools stays the same across all three categories — Claude and ChatGPT ahead of Gemini and DeepSeek — but that order matters less than the trend inside each row. The gap between "rarely wrong" and "noticeably higher error rate" is what should decide how much independent checking a given question deserves, not just which tool answered it.
I Used to Think the AI Was Wrong. Usually, I Was.
I struggled with this constantly through my first year. I'd ask a question, get an answer that felt off, and file it away as another example of AI being unreliable. Then, right at the start of second year, something shifted. I went back and reread some of my old prompts, and it was obvious: most of those answers weren't wrong because the AI failed. They were "wrong" because I had asked something ambiguous enough that any reasonable human would have understood what I meant, but the AI took it completely literally.
Most AI tools have a hard time saying "I'm not sure what you mean — can you rephrase that?" Instead, they pick the most probable interpretation of your question and answer it with full confidence, as if there was never any ambiguity to begin with. A human colleague would ask. AI guesses and moves forward.
Once I noticed this, a lot of what I used to blame on AI's limitations turned out to be a communication gap I could close myself — by being more specific, by stating what I already knew, by saying exactly what kind of answer I needed. The AI hadn't gotten smarter overnight. I had just started asking better questions.
The Paradox: Fewer Errors Should Feel Safer. It Doesn't.
Let's grant the premise fully: AI hallucinations are decreasing, consistently, at a genuinely impressive rate. On the surface, that sounds like unambiguous good news. I think it's the opposite, and here's the reasoning.
In the past, we double-checked everything an AI told us, because we knew — from direct, repeated experience — that it made mistakes often enough to justify the extra step. Today, the errors are rare enough that they rarely have a visible impact on how we act, which means the habit of checking quietly erodes. That is exactly the environment where blind trust grows fastest: not when something fails constantly, but when it almost never does.
This is where the fabricated references I mentioned at the start fit in perfectly. I had noticed AI occasionally producing sources with no real connection to the topic, or citations that don't exist at all, seemingly there just to make the answer feel more credible than it actually is. A citation formatted like a real one carries an unearned signal of legitimacy — and that signal is exactly what makes double-checking feel unnecessary.
"But What If the Errors Disappear Completely?"
Someone reading this might reasonably think: fine, rare-but-dangerous errors are worse than frequent-but-obvious ones, but doesn't that mean the fix is just to keep improving AI until the errors vanish? I'd argue that outcome is more dangerous, not less — for a reason that has nothing to do with hallucination rates at all.
A lot of what looks like an AI mistake isn't really the model getting something wrong. It's the model faithfully answering the question it was actually given, when the real situation was more complicated than what got communicated to it — sometimes by accident, sometimes because something was deliberately left out.
One of our pediatrics professors at Omdurman Islamic University once described a case that has stayed with me since. A mother had brought her infant in from a village in rural Al-Jazira state, with recurrent GI illness and poor weight gain — the kind of presentation that funnels naturally into a fairly standard set of differentials. What eventually uncovered the real cause wasn't a lab result. It was the professor noticing a hesitation in how the mother answered a routine question about feeding, something that didn't quite match her words. Only after gently circling back to it more than once did she admit she had been stretching a limited supply of formula with untreated water — something she hadn't mentioned earlier because she was too ashamed to say it out loud to a doctor.
A human clinician can sense when an answer doesn't quite fit and knows to keep gently asking. AI has no equivalent instinct. It cannot feel that a patient's story doesn't add up unless the inconsistency is spelled out explicitly in the text it's given — even a version of AI with zero hallucinations would still only know exactly what it was told, nothing more.
This is why "the errors are disappearing" isn't the finish line it sounds like. The ceiling on AI's usefulness in complex clinical reasoning was never just about hallucination rate — it's about the fact that the most important detail in a real clinical encounter is sometimes the one nobody said out loud.
How I Actually Try to Catch This Now
- I read the explanation from two different AI tools before treating anything as settled — never just one
- For sensitive or high-stakes information specifically, I never rely on a single tool at all, no exceptions
- When an answer feels slightly off, I sometimes ask the same question a second time, worded completely differently, just to see if the answer holds up
- I never accept a citation without opening the actual DOI or PubMed entry myself
- I never take a drug dose or clinical guideline from AI without checking it against an authoritative source
None of these habits assume AI is untrustworthy. They assume the opposite — that it's become good enough to be genuinely convincing when it's wrong, which demands a different, more deliberate kind of caution than the one I needed in first year, when the mistakes announced themselves.
Prompt — Forcing AI to Show Its UncertaintyThis doesn't fix the underlying problem — it just makes the model's uncertainty visible instead of hidden, which is the one thing a smoother, more polished answer tends to hide by default. Visible uncertainty is something I can act on. The invisible kind is what got past me for weeks.
The role AI plays in medical education was never going to be settled by whether a given answer happens to be right. It's shaped by how its growing reliability quietly changes what the person on the other end still bothers to check — and the honest answer, for me, was less than I assumed until this morning.
Frequently Asked Questions
Why does AI getting more accurate increase risk for medical students?
Because accuracy and verification move in opposite directions. Early AI models were wrong often enough that students checked everything by habit. Current models are right often enough that checking starts to feel unnecessary — which means the rare errors that do slip through are far more likely to be believed and acted on.
Is ChatGPT or Claude more accurate than Gemini or DeepSeek for medical topics?
In my testing, Claude and ChatGPT rarely make errors on abstract, self-contained topics like physiology or biochemistry, while Gemini and DeepSeek are good but slightly more error-prone in that same category. On complex, multi-layered clinical scenarios, all four can hallucinate — the same relative ranking holds, but the error rate rises sharply for every tool. On drug dosing and clinical guidelines, the error rate climbs even further, by a wide margin, regardless of which tool is used.
If AI hallucinations keep decreasing, doesn't that solve the problem eventually?
No — that would arguably make things more dangerous, not less. Many of what look like AI errors are actually the AI faithfully answering an incomplete or ambiguous question, often because the human didn't fully explain the situation, whether by mistake or because something was deliberately left out. A human clinician has intuition that can sense when something doesn't add up. Even a hallucination-free AI has no equivalent instinct — it only knows what it's told.
Can AI fabricate academic references that look real?
Yes. AI can generate citations with correctly formatted DOIs, PMIDs, and journal names that are either entirely invented or attached to a real paper on a completely unrelated topic. The formatting looking correct is not evidence the citation is real — only opening the actual database entry confirms that.
How can medical students verify AI-generated medical information?
Read the explanation from at least two different AI tools rather than one, never rely on a single tool for sensitive or high-stakes information, and occasionally ask the same question two different ways to check whether the answer changes. Never accept a citation, drug dose, or clinical guideline from AI without verifying it against the original source yourself.
References
- Lucas HC, Upperman JS, Robinson JR. A systematic review of large language models and their implications in medical education. Medical Education. 2024;58(11):1276-1285. DOI: 10.1111/medu.15402
- Mitigating hallucinations in healthcare AI: a systematic review of evidence-based strategies. PubMed. 2025. PMID: 42251377
- Zhang et al. Comparative performance of GPT and DeepSeek against AMBOSS user accuracy on USMLE-style questions. Cureus. 2025;17(8):e90212. DOI: 10.7759/cureus.90212
- Shi X, Jiang Z, Xiong L, Siu KC, Chen Z. Utilization of AI Among Medical Students and Development of AI Education Platforms in Medical Institutions: Cross-Sectional Study. JMIR Human Factors. 2026;13:e81652. PMID: 41505769
Medical Disclaimer: This article reflects personal experience as a medical student and does not constitute medical advice. Always verify medical information with authoritative sources. Never rely on AI for drug doses, clinical decisions, or as a substitute for checking a citation yourself.