The AI-Assist That Fails at 2 A.M.: How to Train Your ED for Algorithmic Failure

The AI-Assist That Fails at 2 A.M.: How to Train Your ED for Algorithmic Failure

If you’ve worked in a modern emergency department long enough, you’ve already met the most dangerous version of clinical AI: the one that’s mostly right.

Not the dramatic, sci-fi “robot doctor” nonsense. I mean the real stuff that shows up quietly in the background — triage scores, sepsis flags, risk predictions, radiology prioritization, documentation assistants — and earns your trust by being correct 80–95% of the time.

Then it fails.

And it doesn’t fail at 10:30 on a Tuesday when the informatics team is around. It fails at 02:00 with a waiting room full of chest pain, a trauma arriving unannounced, and a nurse telling you bed 12 “just looks bad.”

If your department is going to use AI (and you are), you need to train for that failure like you train for airway disasters: deliberately, repeatedly, and with an operational plan.

The problem isn’t AI error — it’s untrained dependence

Every clinical system has error. Humans miss STEMIs. Labs hemolyze. CT scanners go down. The reason AI feels different is that it changes how we think.

The predictable failure mode is not “the model is wrong.” The predictable failure mode is automation bias: the team defers to the output because it’s usually correct, and because it’s cognitively easier than re-deriving the decision from scratch.

That’s why your AI training plan can’t just be a one-hour vendor webinar.

Training has to build three habits on shift:

1. Interrogate the output (what data drove it?)
2. Translate the output into a specific action (what do we do now?)
3. Override safely when it’s wrong (how do we document/escalate/learn?)

Think of AI like a new medication: indication, contraindications, and adverse effects

When a new drug hits the formulary, you don’t say, “Here you go — it’s good.” You define:

• Indications
• Contraindications
• Dosing and monitoring
• Adverse effects
• What to do when the adverse effect happens

Clinical AI deserves the same discipline.

The AI Risk Management Framework from NIST is built around four functions — GOVERN, MAP, MEASURE, MANAGE — to help organizations manage AI risk across the lifecycle (NIST AI RMF 1.0 PDF).

You don’t need to memorize the framework. But you do need to translate that mindset to ED operations:

• Who governs this tool clinically?
• What is it intended to do — and what is it not intended to do?
• How do we measure performance in our shop?
• What’s the management plan when it underperforms?

Section 1: Build a “failure-mode inventory” before the tool goes live

Most departments roll out AI like a new order set: build it, announce it, hope it works.

Instead, do a pre-brief the way you’d do for a new resuscitation protocol. Create a one-page “failure-mode inventory” for the tool:

• What data elements does it rely on? (triage note text? vitals? problem list? prior encounters?)
• What are the predictable bad-data scenarios? (missing vitals, copy-forward histories, boarding-induced delays, triage shortcuts)
• What does “wrong” look like clinically? (false negatives vs false positives)
• What is the time-to-harm? (minutes for sepsis? hours for admission prediction?)
• What is the human backstop? (who is required to review? what triggers escalation?)

If you can’t answer those questions, you don’t have decision support. You have a black box with a UI.

Section 2: Train the team — not just the physicians

AI failure is rarely a solo-physician event. It’s a team event.

The nurse who’s watching the patient deteriorate. The resident who’s anchoring on the predicted risk score. The attending who’s trying to manage throughput and gets nudged into “trust the system.”

Training should include:

• Nursing: When to disregard the tool, how to escalate, what to chart.
• Providers: When to override, how to document, how to report the case for review.
• Charge + flow: What to do when AI-based throughput predictions conflict with lived reality.
• Registration/triage staff (yes): because garbage in, garbage out.

If you want a deeper dive on how simulation fits into AI readiness, I laid out the broader argument in this Medium piece: Medical Simulation and AI: Building the Training Ground for the Next Generation.

Section 3: Use simulation the way we actually use it: to hardwire response patterns

Here’s the blunt truth: reading about AI errors does not prepare you to catch one in real time.

Simulation does.

And you don’t need a multimillion-dollar sim center to do this. The ED already runs micro-simulations every day: huddles, case reviews, bedside walk-throughs, mock codes.

What an “AI failure sim” looks like

Pick 3–5 scenarios that represent your tool’s highest-risk failures. Examples:

• Sepsis alert false negative in an elderly patient with subtle vitals + abnormal appearance
• Triage acuity score false low because the chief complaint is vague and the free-text note is thin
• Admission prediction false high leading to premature inpatient bed requests and flow problems
• Radiology prioritization misses because the indication was poorly structured

Then embed one deliberate decision point:

• The model output suggests one course of action.
• The clinical picture suggests another.

Your training goal is not “prove the model wrong.” Your training goal is teach the team what to do next.

The three questions every sim must answer

1. Who owns the decision? (attending? charge? triage nurse?)
2. What is the override protocol? (do we ignore, silence, or annotate?)
3. Where does the case go for learning? (QI? informatics? safety event reporting?)

Section 4: Build an “AI near-miss” reporting loop that doesn’t punish clinicians

If you want safer AI, you need reporting. And if you want reporting, you need psychological safety.

The World Health Organization’s guidance on patient safety incident reporting emphasizes that incident reporting is meant to support learning and sustainable risk reduction — not just data collection (WHO technical guidance).

Translate that to ED AI use:

• Make “AI near-miss” reporting easy (a button, a dotphrase, a short form).
• Define what counts: false negative, false positive with harm, workflow disruption, documentation hallucination, alert fatigue overload.
• Close the loop: if someone reports a bad output, they should hear back what happened with it.

This is where most implementations die. Everyone is “too busy” to report. Then leadership assumes the tool is fine because there are no complaints.

No reports doesn’t mean no problems. It means no system.

Section 5: A practical 30-day checklist for ED leaders

If you’re the medical director, informatics lead, or the unlucky attending who got voluntold to “own AI,” here’s the 30-day plan I’d run.

1. Inventory every AI output that influences clinical decisions.
2. Write the failure-mode inventory for the highest-risk tool first.
3. Define the override + escalation rule. One paragraph. Make it real.
4. Run two short simulations (15–20 minutes each) that include nursing + providers.
5. Start the near-miss reporting loop and commit to a monthly 30-minute review.

That’s it. Not a 40-slide deck. A repeatable system.

Dr. Chet’s Take

AI isn’t dangerous because it’s inaccurate.

AI is dangerous because it’s persuasive.

The ED is a high-cognitive-load environment. Anything that reduces thinking effort will get adopted quickly — and anything that’s usually right will get trusted.

If you don’t train your department for the moment the tool is wrong, you’re not deploying decision support.

You’re deploying a new way to make old mistakes, faster.

Key Takeaways

• AI needs failure-mode training the same way airway management needs “can’t intubate/can’t oxygenate” planning.
• Build a one-page failure-mode inventory before go-live: bad data scenarios, time-to-harm, human backstop.
• Run short, realistic simulations where the AI output conflicts with the clinical picture.
• Create a non-punitive near-miss reporting loop and close the feedback loop.
• Governance isn’t bureaucracy; it’s how you keep persuasion from becoming harm.

FAQ

Q: We don’t have a sim center. Can we still do this?

Yes. Tabletop scenarios, charge-nurse huddles, mock cases during low-volume windows, and debriefs after real cases all count. The goal is rehearsed response patterns, not fancy mannequins.

Q: What if the vendor says the model is “validated”?

“Validated” means it performed on someone’s dataset under someone’s conditions. Your ED has different documentation habits, different triage patterns, and different operational pressures. You still need local measurement and a reporting loop.

Q: Who should be allowed to override the AI output?

Clinically, the licensed clinician responsible for the decision owns the override. Operationally, you should define escalation rules so a nurse or resident can raise concern when the tool output conflicts with what they’re seeing.

Q: What should we document when we disagree with the AI?

Document your clinical reasoning in plain language. If the tool has an official “override reason” field, use it. The goal is to create an audit trail that supports learning — not defensive charting.

Q: How often should we review performance?

Monthly at first, then quarterly once stable. Any significant workflow change, EHR upgrade, or model update should trigger a re-review.

Free Guide

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice -- not just the hype -- I put together a free practical guide. You can download it here: AI in EM Survival Guide

Read more from Dr. Shermer on Medium: Read more from Dr. Shermer on Medium

Sources

• NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). (NIST AI RMF 1.0 PDF)
• World Health Organization. Patient safety incident reporting and learning systems. 16 September 2020. (WHO technical guidance)