AI Clinical Decision Support in the ED: The 'Show Your Work' Rule

AI Clinical Decision Support in the ED: The 'Show Your Work' Rule

You don’t need AI in your emergency department to be smart. You need it to be legible.

Right now, a lot of clinical decision support (CDS) is arriving in the ED as a black box: a score, a flag, a recommendation, a risk percentile. It looks authoritative. It shows up inside the EHR. It has a confidence number that feels scientific.

And at 2 a.m., when you’re covering three rooms, one psych hold, and the waiting room is stacking, the easiest thing in the world is to let that output become the default plan.

Here’s the operational rule I want you to adopt:

If the tool can’t show its work in a way a clinician can review quickly, it doesn’t get to steer the care.

This is not anti-AI. It’s pro-accountability.

(If you want the broader, clinician-side reality check on what AI still can’t do, read my related piece: I’ve Practiced Emergency Medicine for 25 Years. Here’s What AI Still Can’t Do..)

Why the ED is the worst place for “trust me” software

Emergency medicine is a hostile environment for brittle decision support:

• We work with incomplete information.
• We make decisions under time pressure.
• The data stream is messy (triage notes, copied histories, delayed labs).
• Our risk tolerance is asymmetric: one missed sick patient matters more than ten unnecessary tests.

That’s exactly why CDS is tempting. It promises “pattern recognition at scale” and faster sorting.

But operationally, the ED is also where a CDS failure becomes a systems event: downstream admissions, ICU delays, lawsuits, morale damage, and loss of physician trust.

So before you argue about whether a model’s AUC is 0.84 or 0.87, ask the more important question:

Can a working clinician independently review what the model is basing its recommendation on?

That question isn’t just philosophical. It’s embedded in how regulators describe “non-device” CDS.

The FDA’s line in the sand: “independently review the basis”

The FDA’s Clinical Decision Support Software guidance lays out a practical concept that maps to bedside reality: some CDS functions can be excluded from device regulation only if they let the health care professional independently review the basis for the recommendation (FDA guidance PDF).

In plain language, the “good” kind of CDS:

• Shows you what inputs it used.
• Shows you how it reasoned (at least at a human level).
• Lets you decide whether the recommendation applies to this patient.

The guidance also makes an important operational point: if the decision is time-critical, FDA doesn’t consider that environment compatible with independent review (FDA guidance PDF).

Read that again.

If your workflow is too time-pressured for a clinician to review the basis, then the tool shouldn’t be designed as “trust me, act now.” In the ED, that is most of our day.

The ED “Show Your Work” checklist (what I actually want to see)

When I’m evaluating a CDS tool for emergency department use, I’m looking for a fast, shift-friendly way for clinicians to answer four questions:

1) What problem is it solving — exactly?

If the vendor can’t state the problem in one sentence, you’re buying a dashboard, not a tool.

• “Predicts sepsis within 6 hours” is a problem statement.
• “Improves outcomes with AI” is marketing.

Operational test: Can I write a one-line policy describing when we use it and what action it triggers? If not, it will drift into noisy non-use.

2) What inputs did it use for this patient?

A score without inputs is not decision support; it’s a decision replacement.

At minimum, the UI should show:

• Vitals and trends that mattered
• Lab values and timestamps
• Key comorbidities or problem list items used
• Free-text features (if used) with a clear warning label

If a model is using triage free text, copied notes, or nursing comments as a major driver, that can be fine — but it has to be visible. Otherwise, you’ll never know when a hallucinated history or a copied ROS contaminated the output.

3) Why is it recommending this?

I’m not asking for the full math. I’m asking for a clinician-readable explanation that can be audited.

Examples of acceptable “why”:

• “Risk increased due to persistent tachycardia + rising lactate + hypotension trend.”
• “Recommendation driven by: age, troponin delta, ECG interpretation, and prior CAD.”

Examples that fail:

• “High risk.”
• “Model confidence 92%.”

If the tool can’t explain itself, you can’t train people to use it, and you can’t do post-event review when it fails.

4) What is it not seeing?

Every ED clinician knows the classic model failure: the patient who looks bad but the labs are still normal.

Good CDS should explicitly show known blind spots:

• Missing data the model expects
• Situations where validation is limited (peds, pregnancy, immunocompromised, unusual comorbidities)
• Operational caveats (downtime workflows, delayed labs)

This isn’t a “nice to have.” It’s how you keep clinicians from over-trusting a clean-looking number.

Governance that fits real departments (NIST has the map)

If you’re trying to operationalize this, the best high-level framework I’ve seen is the NIST AI Risk Management Framework (AI RMF 1.0) with its four functions: Govern, Map, Measure, Manage (NIST AI RMF 1.0).

I translate that into ED language like this:

• Govern: Who owns the tool, the thresholds, the training, and the failure review?
• Map: Where does it touch workflow, and what does it change on shift?
• Measure: What are we tracking that matters (false negatives, alert burden, subgroup performance, clinician trust)?
• Manage: What do we do when the model drifts, breaks, or the workflow changes?

NIST’s “trustworthy AI” characteristics line up with what clinicians actually care about: valid/reliable, safe, secure/resilient, accountable/transparent, privacy-enhanced, and fair with harmful bias managed (NIST AI RMF 1.0).

That’s not a policy document to hang on the wall. It’s a vocabulary you can use to run a real governance meeting.

Implementation pattern: start with the “audit lane,” not the “action lane”

Here’s my practical rollout recommendation for CDS in the ED:

1. Run it in the background first. No clinician-facing alerts. Collect outputs.
2. Create an “audit lane.” A small group (physician + nursing + quality) reviews cases where the model would have changed care.
3. Define an escalation rulebook. When the model fires, what happens? Who is notified? What’s the required clinician action?
4. Only then go clinician-facing — with guardrails.

Why this matters: if your first experience of a model is that it yelled at you all night and was wrong twice, you’ve lost physician buy-in for a year.

The audit lane approach builds trust by making the model earn influence.

Dr. Chet’s Take

ED clinicians are not “end users.” We’re the last safety layer.

If your CDS tool can’t show its work, it will create substitution bias: clinicians anchor on the output and stop doing the productive doubt that keeps patients safe.

So make “show your work” non-negotiable.

If the tool can explain itself, we can train it, govern it, and improve it.

If it can’t, it doesn’t belong in a time-critical environment where the wrong nudge becomes an event.

Key Takeaways

• The ED is the wrong environment for black-box CDS; legibility beats novelty.
• A usable CDS tool must let clinicians independently review the basis of recommendations (FDA guidance PDF).
• If the decision is time-critical, “independent review” is often unrealistic — design accordingly (FDA guidance PDF).
• Use a risk framework (Govern/Map/Measure/Manage) so you’re not improvising governance after the first failure (NIST AI RMF 1.0).
• Start with an audit lane, then earn the right to alert.

FAQ

Q: Do I need to understand the math to safely use CDS?

A: No. You need to understand inputs, limits, and failure modes. “Show your work” means the tool explains itself at a clinician level and exposes what it used and what it missed.

Q: If a vendor won’t reveal their algorithm, should I walk away?

A: You don’t need their source code, but you do need enough transparency to independently review the basis and run local validation. If they can’t provide that, the tool is asking for trust it hasn’t earned.

Q: What’s the biggest operational risk with CDS in the ED?

A: Substitution bias. The model becomes the default plan because it’s convenient. That’s how you miss the sick patient who doesn’t fit the training data.

Q: How do we keep alerts from becoming noise?

A: Tie alerts to specific actions and track alert burden like a safety metric. If the tool fires constantly with low yield, you will train the department to ignore it.

Q: What if the model performs well at another hospital?

A: Great. Now prove it in your environment. Patient mix, workflow, lab turnaround times, and documentation culture all change model performance.

Lead Magnet

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice -- not just the hype -- I put together a free practical guide. You can download it here: AI in EM Survival Guide

Read more from Dr. Shermer on Medium: https://medium.com/@chet.shermer

Sources

• FDA. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff (Jan 29, 2026). https://www.fda.gov/media/109618/download
• NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0) (Jan 2023). https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf