Automation Bias in the ED: Explanations Won't Catch a Bad Model

At 0200 an 84-year-old retiree who lives alone arrives short of breath, with bilateral crackles and a BNP still pending. The chest radiograph lands on the worklist with an AI flag for pneumonia and a colored overlay across the lower lung fields. The resident orders ceftriaxone and azithromycin before the BNP comes back. Nobody in the room asks whether the model weighs age more heavily than the film.

Bottom line: clinician review is a weak safety net for a flawed model, and heatmap explanations do not fix it; before an ED AI tool goes live, require performance data broken out by the patient groups you see, and review every case where the tool and the final diagnosis disagreed.

What the JAMA Vignette Trial Tested

Jabbour and colleagues ran a multicenter randomized clinical vignette survey study from April 2022 to January 2023 with hospitalist physicians, nurse practitioners, and physician assistants from 13 US states [1]. Of 572 clinicians randomized, 457 completed at least 1 vignette and formed the primary analysis: 231 saw AI predictions alone and 226 saw predictions with explanations [1]. The explanations were image-based Grad-CAM heatmaps overlaid on the chest radiograph [1].

Each participant reviewed 9 vignettes of patients hospitalized with acute respiratory failure and rated the likelihood of pneumonia, heart failure, and COPD from 0 to 100, with 50 or higher counted as a positive diagnosis [1]. Two vignettes had no AI and set baseline accuracy. Six showed AI predictions: 3 from a standard model and 3 from a systematically biased model [1].

The trial defined "systematically biased" precisely. Each participant was randomized to 1 of 3 biases: a model that called pneumonia likely in patients 80 or older, a model that called heart failure likely when BMI was 30 or higher, or a model that called COPD likely when the radiograph was blurred [1]. The biased model's heatmaps highlighted the shortcut it used, such as low bone density or soft tissue, rather than lung [1]. When biased predictions were triggered, the biased models were 33% accurate compared with 67% for the standard model [1].

Explanations Did Not Rescue Accuracy

Baseline diagnostic accuracy without AI was 73.0% (95% CI 68.3% to 77.8%) [1]. Standard AI raised it by 2.9 percentage points (95% CI 0.5 to 5.2), and standard AI with heatmaps raised it by 4.4 points (95% CI 2.0 to 6.9) [1].

Biased AI dropped accuracy to 61.7%, a loss of 11.3 points (95% CI minus 15.5 to minus 7.2) [1]. Adding heatmaps to the biased model left accuracy at 64.0%, still 9.1 points below baseline (95% CI minus 13.2 to minus 4.9) [1]. The difference between biased AI with and without explanations was 2.3 points (95% CI minus 2.7 to 7.2, P = 0.37) [1]. The loss came mostly from specificity, which fell from 73% at baseline to 53% with biased predictions [1].

Treatment choices followed the diagnoses. Treatment selection accuracy was 70.3% without AI, 77.0% with correct predictions, and 55.1% with incorrect predictions, a drop of 15.3 points (95% CI minus 19.1 to minus 11.4) [1]. Explanations did not significantly improve treatment accuracy when predictions were wrong [1].

The direction of effect was similar in exploratory subgroups by diagnosis, by profession (274 physicians and 142 NPs or PAs), and by prior decision-support experience, though the paper does not report numerical estimates inside those subgroups [1]. Two-thirds of participants (66.7%) were unaware that AI could be systematically biased by patient demographics [1]. The authors concluded that image-based explanations did not mitigate biased models and questioned whether clinicians can serve as a backstop against flawed AI at all [1].

Experience Did Not Protect Mammography Readers

Dratsch and colleagues tested the same failure in radiology. In a prospective experiment, 27 radiologists read 50 mammograms with a purported AI system suggesting BI-RADS categories [2]. The first 10 cases had correct suggestions; in the remaining 40, the AI suggested an incorrect category for 12 [2]. The study defined automation bias as the propensity to favor suggestions from automated systems [2].

Every experience tier lost accuracy when the AI was wrong. Inexperienced readers fell from 79.7% correct with accurate suggestions to 19.8% with inaccurate ones [2]. Moderately experienced readers fell from 81.3% to 24.8%, and very experienced readers from 82.3% to 45.5% [2]. Inexperienced readers followed an incorrectly high BI-RADS suggestion more than very experienced readers (mean degree of bias 4.0 versus 1.2, P = 0.009) [2]. The authors concluded that readers at every experience level were prone to automation bias [2].

Emergency Physicians Followed Bad Advice Too

Gaube and colleagues included emergency physicians directly. In a 2 by 2 factorial experiment, 138 radiologists and 127 internal and emergency medicine physicians (80 internal medicine, 47 emergency medicine) from the US and Canada reviewed 8 chest radiograph cases [3]. Six cases came with accurate advice and 2 with inaccurate advice, and the advice was labeled as coming from either an AI system or an experienced radiologist [3]. All advice was actually written by human experts [3].

Inaccurate advice cut accuracy in both groups regardless of the label [3]. Among internal and emergency medicine physicians, 41.73% gave the wrong diagnosis every time the advice was wrong, compared with 27.54% of radiologists [3]. Only 17.32% of the internal and emergency medicine group rejected all incorrect advice, compared with 28.26% of radiologists [3]. Radiologists rated advice labeled as AI lower in quality, but the authors found that this skepticism did not change how much they relied on it [3].

What This Means Before Go-Live

The biases in the JAMA trial were not exotic. Age, BMI, and image quality are variables every ED population spreads across, and portable films at 0200 are rarely the cleanest images in the archive. The trial's lesson is that the clinician at the workstation will not reliably detect a model leaning on one of them.

That moves the safety work upstream. A vendor's overall accuracy figure hides exactly the failure these studies describe, because a model can be accurate overall and wrong in a specific subgroup. The data that matter are performance by age band, body habitus, and image source in a population like yours, and a running log of cases where the tool and the final diagnosis disagreed.

Key Takeaways

• Require subgroup performance data (age bands, BMI, portable versus department films) before go-live, then audit every AI-discordant final diagnosis; bedside review alone does not catch a biased model [1,3].
• Systematically biased AI cut clinician diagnostic accuracy by 11.3 points and treatment accuracy by 15.3 points in the JAMA trial [1].
• Heatmap explanations recovered only 2.3 points against a biased model (P = 0.37), so an overlay is no evidence that the model looked at the right thing [1].
• In the Gaube experiment, 41.73% of internal and emergency medicine physicians accepted every piece of wrong advice they saw [3].

FAQ

What is automation bias in emergency medicine?

It is the tendency to favor a suggestion from an automated system over your own judgment or contradictory data [2]. In a randomized vignette trial, clinicians shown systematically biased AI predictions lost 11.3 points of diagnostic accuracy compared with working without AI [1].

Do AI heatmaps help clinicians catch a wrong prediction?

In the JAMA trial, heatmaps that highlighted the biased model's shortcut improved accuracy by only 2.3 points over the biased model alone, a difference that was not statistically significant [1]. Accuracy stayed 9.1 points below the no-AI baseline [1].

Are experienced physicians immune to automation bias?

No. Very experienced mammography readers fell from 82.3% to 45.5% correct when AI suggestions were wrong, a smaller drop than less experienced readers but still a large one [2]. In a chest radiograph study, 27.54% of radiologists accepted every incorrect suggestion [3].

Dr. Chet's Take

I have been an emergency physician for more than 25 years at a Level I academic trauma center, and the phrase I distrust most in AI sales meetings is "the physician stays in the loop." This article gets the core problem right. The loop is the person at 0200 with twelve patients and a flag on the screen, and these studies show that person bends toward the flag. That is not a character flaw. It is what a busy brain does with a confident suggestion. The argument holds because three different research teams, three different tasks, and three different clinician groups ended up in the same place. When a vendor tells you the clinician is the safety mechanism, they are describing the failure mode.

That being said, the evidence is all simulated, and I would be careful about how far we push it. Vignettes and reading sessions remove the nurse who says the patient looks wet, the family who mentions the furosemide ran out, and the second look at the patient in the hallway. Those signals are real, and they catch errors that no study session captures. The honest answer is that nobody has measured automation bias inside a live ED with all of those inputs running. My bet is that it is still there, and possibly worse under real volume. I also think the deeper risk is training. A resident who learns radiograph reading alongside a flag never builds the pattern library to argue with it.

If you are leading an emergency department, put one person's name on the AI-discordance log and give them protected time to read it monthly. Ask your residency or APP leads to teach a session in which the tool is deliberately wrong. Get the vendor to state in writing who investigates when your log shows a pattern, and how quickly a corrected model reaches your site. Then track whether residents' independent reads hold up over a year of using the tool. If you are the attending at the workstation, read the film and form your impression before you look at the flag, then compare. Teach your residents to do the same in that order. The model gets a vote only after you have cast yours.

— Chester Shermer, MD, FACEP | Emergency Medicine, 25+ Years Clinical Experience | State Surgeon

Continue Your Training

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice — not just the hype — I put together a free practical guide. You can download the AI in EM Survival Guide here.

Browse All Courses at Global MedOps Command

Relevant Reading on Global MedOps Command:

How to Avoid Becoming an AI Casualty — Dr. Shermer's guide to navigating AI tools in clinical and operational settings without compromising judgment or patient outcomes.

Emergency Department Efficiency Playbook — Practical systems for throughput, triage optimization, and operational efficiency.

Read more from Dr. Shermer on Medium

Connect with Dr. Shermer: LinkedIn — Chester "Chet" Shermer, MD, FACEP.

References

1. Jabbour S, Fouhey D, Shepard S, et al. Measuring the impact of AI in the diagnosis of hospitalized patients: a randomized clinical vignette survey study. JAMA. 2023;330(23):2275-2284. PubMed 38112814
2. Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology. 2023;307(4):e222176. PubMed 37129490
3. Gaube S, Suresh H, Raue M, et al. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ Digit Med. 2021;4(1):31. PubMed 33608629

Related simulation training: EM-Sim · Books by Dr. Shermer