Before AI Changes ED Care, Run It in Shadow Mode

The screen is quiet. The model has been connected to the ED feed for two weeks, calculating an admission prediction for every arrival. No clinician sees the score yet. The operations team compares it with what happened, records missing inputs, and asks: “Would this output have changed care for the better?”

That is shadow mode. The algorithm runs on live or representative data, but its recommendation is withheld from clinical decision-makers. The team studies the output, the data feed, and the workflow around it before allowing the tool to influence a patient. For an ED, that pause is not wasted time. It is the first real test of whether a model belongs in the department.

Shadow mode is a safety test, not a demo

AI research in emergency medicine has moved faster than bedside implementation. A 2025 scoping review found 605 studies of AI-based clinical decision support in the ED, but 94.5% remained in the earliest phase of preclinical development and only 5.5% reached later phases of testing (Wang et al., Academic Emergency Medicine). A retrospective result has not shown that a model improves care in your department.

The distinction matters because the ED is a moving system. Triage language changes, interfaces are replaced, documentation templates change, and surges alter who is waiting and when disposition decisions are made. Clinical AI can lose performance when the environment or data-generating process changes (Andersen et al., JBI Evidence Synthesis).

Shadow mode lets the department observe those conditions without asking clinicians to act on an unproven output. A review group compares the prediction with the outcome, the clinician’s assessment, and the timing of care. That is an implementation test, not autonomous readiness.

The evidence supports this staged mindset. In a 2025 prospective multisite study, a machine-learning model was compared with nurse predictions for hospital admission across seven hospitals and nearly 50,000 ED visits. The model outperformed nurse estimates, but real-time workflow implementation and outcomes such as boarding time were described as a future phase (Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System). Prediction performance is not evidence that a department should change its admission or staffing decisions tomorrow.

What to measure before anyone sees the score

Start with the input feed. Record missingness, stale or impossible values, units, timestamps, and fields that have changed meaning. If a respiratory rate is suddenly populated with a default value, the model may return a number while operating outside its tested conditions. A 2024 review found limited practical guidance and little evidence that one monitoring method works everywhere (Andersen et al.).

Next, measure output behavior: no-result frequency, score distribution, alert frequency, and the time between input and output. Compare the output with the action clinicians took, but treat disagreement as a case for review, not proof that either side was wrong.

Then measure clinical performance when the outcome becomes known. Depending on the use case, include sensitivity, specificity, predictive value, calibration, time-to-intervention, and subgroup performance. Ground-truth data may be delayed, difficult to obtain, or affected by the model itself (Andersen et al.). A dashboard with one attractive metric is not a safety system.

Finally, measure the human work around the score. Did it arrive before the decision? Could the receiving clinician tell what it meant, when it was generated, and what data it used? Would it add clicks without changing a plan? Learn whether the model changes attention, action, or delay.

For a practical monitoring companion, see Chet’s “When an AI Model Fails Silently”. It treats monitoring as input integrity, output behavior, clinical performance, and human/workflow feedback, with an owner and clock for every threshold.

Turn the trial into a go-live gate

Before shadow mode begins, write the intended use in one sentence. “Predict hospital admission from triage data to support bed planning” is testable. “Improve patient flow” is not. Name the patient population, the decision the output may support, prohibited decisions, and the fallback if the model is unavailable. 2025 Canadian emergency-medicine consensus work recommends a relevant problem and expert team, attention to data quality and quantity, AI-specific reporting, and ethics and privacy principles (Kareemi et al., CJEM).

Set the observation period and review sample before the first case enters the feed. Review local slices such as overnight arrivals, older adults, psychiatric presentations, high-acuity arrivals, and sparse records. Match the slices to the intended use and patient population. FDA’s 2025 draft guidance for AI-enabled device software emphasizes lifecycle risk management, validation, performance monitoring, and known limitations for regulated device software (FDA).

Define the stop rule. Write triggers such as a broken feed, sustained alert-rate change, unexplained subgroup gap, software release, or repeated clinician reports that the output no longer fits. Assign an owner and response time. NIST’s voluntary AI Risk Management Framework treats trustworthiness across design, development, use, and evaluation, a useful structure for an ED governance file (NIST AI RMF).

Go-live should be conditional. If the model meets the agreed thresholds, activate it for a narrow use with an override path. Show prediction time and distinguish “no score” from “low risk.” Keep the fallback visible. Recheck after a defined interval and after EHR, interface, pathway, or model changes. A 2024 framework describes preventive, preemptive, responsive, and reactive maintenance because no single approach covers every clinical shift (Davis, Embí, and Matheny, JAMIA).

The interface deserves its own test. A statistically sound model can still push a clinician toward the wrong action if its presentation invites over-trust. In a randomized cross-over study, automation bias accounted for 45.5% of mistakes in the AI-assisted round; suppressing AI advice in high-risk zones reduced misleading events more than correcting events in the simulated strategy (Wang et al., JAMIA). Test what people do when the model is wrong.

A two-minute pause rehearsal can be run as an ED simulation rather than a vendor presentation; EM-Sim is a natural home for that kind of clinician-focused practice. If the model crosses the prehospital-to-ED interface, include the handoff and fallback in the scenario; EMS-MedSim covers that operational training space.

Dr. Chet's Take

I have watched emergency departments adopt new tools because the demo was clean and the shift was not. A model that performs well in a slide deck has not earned a place in the bed-management meeting or the resuscitation bay. Shadow mode gets the order right. Let the tool watch the work before it changes the work. In HEMS and in the Guard, we test a capability under conditions that resemble the mission. The ED deserves the same discipline.

That being said, shadow mode is not a permission slip to stop thinking. A score can look stable while the feed is wrong, or arrive too late. A clinician can disagree for a good reason, or miss a signal because the interface made the number feel authoritative. The honest answer is that we need to measure the interaction, not just the algorithm. Automation bias belongs in training, governance, and screen design.

If you are leading an emergency department, pick one model and write its failure envelope this week. Name the inputs it requires, the conditions that invalidate it, the person who can pause it, and the fallback for the next patient. Run the drill until the team can state that fallback without opening a policy manual. A safe AI deployment is one your clinicians can stop.

Key Takeaways

• Shadow mode tests the model, its data feed, and the workflow around it without changing patient care.
• A retrospective accuracy result is not proof of safe ED implementation; prospective workflow and outcome evaluation still matter (Wang et al.).
• Monitor inputs, outputs, clinical performance, subgroup behavior, and human response. Assign an owner and response clock to every stop rule (Andersen et al.).
• Go-live should require a written intended use, a visible fallback, an override path, and a rehearsed pause.
• The final accountable decision remains with the clinical team, not the score.

FAQ

What is shadow mode for an AI tool in the emergency department?

Shadow mode runs the model on real or representative ED data while withholding its recommendation from clinicians. The department compares predictions with outcomes and workflow conditions before allowing the output to influence care.

How long should an ED run an AI model in shadow mode?

There is no universal number of days. Set the period and sample before starting, include the shifts and patient mix covered by the intended use, and base the stop on predefined thresholds, not a calendar date.

What should an ED monitor before going live with AI?

Monitor input integrity, output availability and distribution, clinical performance, subgroup behavior, timing, clinician interpretation, and actions after the output. A model can fail through its feed or workflow even when the algorithm has not changed (Andersen et al.).

Who can pause a clinical AI tool?

The ED’s governance plan should name a role with authority to pause the tool, define the trigger, preserve evidence, notify users, and activate the fallback. NIST’s AI RMF can organize responsibilities across design, use, and evaluation, but the department must translate it into local policy (NIST AI RMF).

Does an AI explanation prevent automation bias?

No single display feature can be assumed to prevent automation bias. A randomized clinician-AI study found that automation bias still contributed to mistakes and proposed suppressing AI advice in high-risk zones as one possible mitigation (Wang et al.). Test the interface with realistic cases, including cases in which the model is wrong.

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice — not just the hype — I put together a free practical guide. You can download it here: AI in EM Survival Guide.

Sources

• Wang et al., “Artificial intelligence–based clinical decision support in the emergency department: A scoping review”
• Andersen et al., “Monitoring performance of clinical artificial intelligence in health care: a scoping review”
• Nover et al., “Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System”
• Kareemi et al., “Establishing methodological standards for the development of artificial intelligence-based Clinical Decision Support in emergency medicine”
• U.S. Food and Drug Administration, “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations”
• National Institute of Standards and Technology, “AI Risk Management Framework”
• Davis, Embí, and Matheny, “Sustainable deployment of clinical prediction tools—a 360° approach to model maintenance”
• Wang et al., “Artificial intelligence suppression as a strategy to mitigate artificial intelligence automation bias”
• Shermer, “When an AI Model Fails Silently: Detection and Response Protocols for Emergency Physicians”
• Related simulation training: EM-Sim, EMS-MedSim, MilMedSim
• Books by Dr. Shermer