When the Triage Model Is Wrong: A Failure-Mode Playbook for the ED

When the Triage Model Is Wrong: A Failure-Mode Playbook for the ED

If you work in an emergency department long enough, you learn a painful truth: triage is where systems fail quietly.

And now we’re adding another system to triage: AI.

Not the flashy kind. The real kind—scores that influence acuity assignment, sepsis flags that re-order priorities, “risk of admission” predictions that change bed flow, and documentation tools that quietly shape the story a downstream team reads.

When these tools are right, they’re helpful. When they’re wrong, they don’t just miss a diagnosis—they misdirect the whole department.

This post is a practical failure‑mode playbook for emergency physicians and ED leaders: what goes wrong, how to spot it in real time, how to document overrides, and what governance has to look like if you want AI‑supported triage without AI‑driven harm.

---

Why triage is the most dangerous place to be wrong

Triage errors are different from most clinical errors because they’re multipliers.

A wrong triage assignment doesn’t just affect one decision. It determines:

• Who gets seen first
• Which nurse assignment you land on
• What room you go to (and what resources are available)
• How your story gets framed in the chart
• How long you wait before a physician even lays eyes on you

Add AI to that environment and you’ve created a new category of risk: automation bias in the most time‑compressed step in the workflow.

If you’ve never watched a busy triage nurse defer to a score because the waiting room is exploding, you haven’t seen the true failure mode.

---

Failure mode #1: the model is “right” on paper and wrong on Saturday night

Most triage‑adjacent models are trained and evaluated on historical ED data. That sounds reasonable—until you remember what historical ED data actually is:

• Missing vitals
• Chief complaints that don’t match the story
• Copy‑pasted nursing notes
• Documentation that lags reality by 30–90 minutes

A model can look strong in a retrospective dataset and still fail in operational reality.

What it looks like on shift

• The model under‑prioritizes vague but dangerous complaints (“weakness,” “dizziness,” “not acting right”).
• The model over‑prioritizes perfectly documented low‑risk complaints.
• The model becomes brittle when vitals are incomplete.

ED response (practical)

1. Treat the score as a flag, not a decision.
2. Identify the minimum data the tool needs to be interpretable (e.g., a blood pressure and O2 sat).
3. Build a hard rule: “No triage AI score is acted on without a human‑verified vital set.”

---

Failure mode #2: the tool inherits your department’s historical blind spots

If your department historically under‑triaged certain populations, your model will learn that.

This is not a moral accusation. It’s a data reality.

Historical data encodes:

• Communication barriers
• Differential documentation intensity
• Access issues that shape revisit rates
• The downstream influence of insurance and disposition patterns

If you deploy AI without auditing performance across subgroups, you’re essentially saying, “Our past decisions were correct enough to automate.”

The fix isn’t theoretical. It’s operational:

• Run subgroup audits (race, language, age, insurance, housing instability proxies).
• Track false negatives separately from false positives.
• Create an escalation channel for frontline staff when patterns emerge.

---

Failure mode #3: AI changes behavior even when nobody admits it

This is the part administrators underestimate.

The tool doesn’t have to force a decision. It just has to shift the conversation.

• “The score says low risk.”
• “The sepsis alert didn’t fire.”
• “The admission risk model says discharge.”

Once those statements exist in the workroom, the cognitive load changes. Clinicians start defending deviations instead of defending the patient.

Your counter‑measure: normalize the override.

A safe ED AI workflow has to make it easy—socially and operationally—to say:

The model is wrong. Here is why.

That’s not defiance. That’s governance.

If you want a deeper operational framing on how real‑world deployment creates integration challenges, this recent Medium piece is worth reading: How Running a Telehealth Network Prepared Me for the AI Integration Challenge.

---

Failure mode #4: documentation AI creates a “triage narrative” that becomes truth

Even if your department isn’t running an explicit triage model, you may be running documentation tools that behave like one.

Ambient documentation and templated triage notes can:

• Over‑normalize abnormal presentations
• Collapse nuance into a clean narrative
• Omit uncertainty (which is often the clinically relevant part)

If downstream teams read a polished triage note, they anchor.

Operational fix

• Train nurses and providers to insert uncertainty explicitly.
• Make “uncertainty phrases” part of triage templates (“unclear etiology,” “concern for high‑risk cause despite nonspecific symptoms”).
• Audit documentation tools for systematic omission patterns.

---

What ED leadership has to build (and most hospitals haven’t)

You don’t “implement” triage AI. You govern it.

Here’s the minimum governance stack I would want before I let a triage‑influencing model shape real patient flow:

1. An inventory of every AI system that touches triage, prioritization, or disposition.
2. Defined action pathways: what exactly changes when the tool fires (and what does not change).
3. An override protocol that is fast, standardized, and non‑punitive.
4. A feedback loop that converts overrides into monitoring data.
5. A quarterly review where clinicians—not vendors—review drift, subgroup performance, and near misses.

The Emergency Severity Index has always been a human system with structured rules; if you want the baseline reference point for how traditional triage is designed to work, start with the ESI materials from AHRQ (AHRQ ESI).

---

Dr. Chet’s Take

I’m not anti‑AI. I’m anti‑unowned AI.

Triage is the last place you can afford a tool that feels authoritative but isn’t accountable.

If your hospital can’t answer these questions, you’re not ready:

• Who owns triage model performance—clinically and legally?
• How do overrides get logged, reviewed, and translated into model monitoring?
• What is the plan when the model drifts during flu season, RSV season, or a staffing collapse?

If the answer is “the vendor monitors it,” that’s not a plan. That’s a liability strategy.

---

Key Takeaways

• Triage AI errors are multipliers: they misdirect flow, documentation, and downstream decision-making.
• The most common failure mode is dataset cleanliness vs. real‑shift chaos.
• If you don’t audit subgroup performance, you automate historical blind spots.
• Safe deployment depends on fast, normalized overrides and a monitoring loop.

---

FAQ

1) Should EDs use AI for triage at all? Yes—but only as decision support with defined limits, clear accountability, and a documented override pathway.

2) What’s the single most important safeguard? A frictionless override protocol that clinicians actually use, paired with routine review of override patterns.

3) How do we know if the model is drifting? Track performance over time (not just at go‑live), stratify by subgroup, and review near misses quarterly.

4) Is this regulated by the FDA? Some clinical decision support functions fall under FDA oversight depending on how they’re marketed and used; start with the FDA’s clinical decision support overview (FDA CDS).

5) Where do we start if we’re already live and worried? Start with an inventory + override logging this week. Then build a 30‑day review of cases where clinicians disagreed with the tool.

---

Next Step (Free Guide)

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice -- not just the hype -- I put together a free practical guide. You can download it here: AI in EM Survival Guide

Read more from Dr. Shermer on Medium: Read more from Dr. Shermer on Medium

---

Sources

• AHRQ ESI
• FDA CDS
• NIST AI RMF 1.0
• WHO AI ethics & governance for health