Social Determinants of Health and the AI Blind Spots Hurting Patients

A 2026 study from Ann & Robert H. Lurie Children's Hospital of Chicago, published in May, analyzed more than 74,000 pediatric emergency department visits and found that undertriage — assigning a lower acuity score than the child actually needed — was significantly more common for Black children, Hispanic children, and children whose families preferred to communicate in Spanish. The same study has been reported widely in the medical equity literature this spring, and the lead author has been careful to attribute the pattern to a combination of implicit bias, communication breakdowns from underuse of professional interpreters, and structural factors inside the healthcare system that have been documented for decades.

That study is not new in its diagnosis. The triage disparity it describes has been measured, replicated, and discussed in emergency medicine journals for at least twenty years. What is new is the context in which the study lands. In 2026, emergency departments across the country are deploying AI-augmented triage, AI-augmented disposition, AI-augmented length-of-stay prediction, and AI-augmented clinical decision support at a pace that has outrun the specialty's ability to audit the tools for bias. The same departments where Black, Hispanic, and Spanish-preferring children have been systematically undertriaged for two decades are now feeding that historical triage data into the training sets and the prospective decision support that will operate on the next generation of patients. The disparity does not disappear when it is encoded into software. It accelerates.

What the Algorithm Learns When the System Has Been Unequal

The mechanism is well-understood by everyone who builds clinical AI and is poorly understood by most physicians who use it. A machine-learning model trained on historical clinical data learns the patterns in that data. If the historical data reflects systematic undertriage of certain patient populations, the model learns that those populations are lower acuity. The model is not making a moral judgment. It is making a statistical prediction based on the labels it has been given. The label is wrong, and the model is faithful to the wrong label.

The Frontiers in Digital Health post-pandemic equity review published in May 2026 is direct about this dynamic. AI in healthcare can either narrow or widen existing disparities, and the variable that determines which way the tool moves is the rigor of the data and the deployment process. Tools trained on representative data, deployed with fairness audits, monitored for differential performance across subgroups, and integrated into a clinical workflow that preserves physician judgment can narrow disparities. Tools trained on non-representative data, deployed without audits, monitored only against aggregate outcomes, and integrated as autonomous decision-makers will widen disparities, often with a veneer of objectivity that makes the widening harder to challenge.

Proxy variables are the specific failure mode that emergency physicians need to understand. ZIP code correlates strongly with race and income in most American cities. Insurance type correlates with employment, immigration status, and household structure. Past no-show rates correlate with transportation access, work flexibility, and child care. Prior healthcare utilization correlates with access to a primary care physician, which correlates with insurance, employment, and ZIP code. An algorithm that uses any of those variables — and most clinical AI tools use several of them — is operating on a set of proxies for race and class. The algorithm has plausible deniability built into its architecture. The patient still gets the same wrong answer.

The Specific Categories Where Emergency Physicians Should Be Looking

There are five categories of clinical AI deployment in the emergency department where the SDOH blind spot is most likely to be operating, and physicians who lead departments should be auditing each one this quarter.

The first is AI-augmented triage. The Lurie Children's study is the warning. Any AI triage system trained on the institution's historical Emergency Severity Index data will inherit the historical undertriage pattern. The audit is straightforward in principle. Compare the AI's acuity assignment across race, ethnicity, language preference, and insurance type, and look for systematic differences after controlling for chief complaint and vital signs. Most institutions have not done that audit. The data is available; the analytic infrastructure is the missing piece.

The second is AI-augmented disposition. The decision to admit, observe, or discharge has historically been influenced by social factors — admission rates differ by race, by insurance status, by language, even after adjustment for clinical severity. A disposition model trained on those decisions inherits the pattern. The audit looks at the model's disposition recommendations across demographic subgroups for clinically equivalent presentations and asks whether the recommendations differ in ways that the clinical data does not justify.

The third is length-of-stay and boarding-prediction tools. These tools are increasingly used to prioritize bed placement and to allocate inpatient capacity. If the underlying historical data reflects that certain populations have longer documented stays because of social factors — interpreter delays, transportation issues affecting discharge, family-coordination challenges — the model will predict those populations as low-priority for fast-tracked bed assignment. The audit asks whether the prediction errors differ across populations and whether the resource-allocation downstream of the prediction is being driven by the population variable rather than the clinical variable.

The fourth is AI-augmented payer interactions and authorization. The WLRN reporting in May 2026 on the unintended consequences of AI in health insurance coverage decisions, including the State Farm and UnitedHealth Group lawsuits cataloged in the spring 2026 coverage of the NAIC AI Systems Evaluation Tool pilot, makes the pattern clear. Insurance algorithms have been documented to deny or delay claims at differential rates for Black homeowners, for Spanish-speaking patients, and for low-income ZIP codes. When the same algorithms touch the emergency department — through prior authorization for observation, through admission denials, through documentation requirements — the patient sitting in the bay is the one absorbing the difference. The emergency physician is the last line of defense against an algorithmic denial that is operating on a proxy for race.

The fifth is documentation and ambient AI scribes. Ambient scribes are trained on clinician speech and chart language, both of which have been shown to differ by patient race, gender, and language. A scribe that learns to document a Black patient's chest pain differently from a white patient's chest pain is propagating the bias into the next generation of training data. The audit asks whether the scribe's output differs across subgroups in ways that the underlying clinical encounter does not justify. The audit is harder than the others, but the institutions that have run it have found differences.

The Regulatory Window Is Closing on Excuses

The institutional excuse for not auditing — "we do not have the analytic infrastructure" — has a shrinking shelf life. The EU AI Act, with most of its rules applying from August 2026, designates AI systems used in health and life insurance as high-risk and imposes binding requirements for bias testing, technical documentation, and post-deployment monitoring. The National Association of Insurance Commissioners launched its AI Systems Evaluation Tool pilot in April 2026 with twelve participating states. California's SB 1120, effective January 2025, prohibits health insurers from denying coverage based solely on an AI algorithm. Colorado's SB 21-169 requires insurers to inventory every algorithm and external data source used in pricing and to test for discriminatory outcomes annually.

The momentum is moving from insurance into clinical AI. The Joint Commission's 2025 guidance on AI in healthcare explicitly endorses pre-deployment bias assessment and post-deployment monitoring. The American Medical Association's November 2025 AI literacy policy calls for clinician training on AI failure modes including bias. The 2025 systematic review on AI-driven pediatric ED triage in PubMed explicitly identifies bias amplification as one of the primary risks that has not been adequately addressed. The regulatory and professional infrastructure is now writing the rules that the institutional leadership will be evaluated against in the next inspection cycle.

When the algorithm produces a recommendation that the clinician believes is wrong on social grounds — a discharge for a patient the clinician believes needs admission, a low-acuity assignment for a child whose family is struggling to communicate the severity, a deferred consult for a patient whose insurance is being flagged by an upstream system — the clinician needs a framework for overriding the tool and documenting the override. I wrote about that framework on Medium in When the Algorithm Disagrees With Your Clinical Judgment: A Resolution Framework, and the principles in that essay translate directly into the SDOH context. The override is the data the institution needs to evaluate whether the tool is systematically wrong in the same direction across the populations the department serves.

Dr. Chet's Take

I have watched emergency medicine grapple with health disparities for twenty-five years, and I have watched every well-intentioned intervention bend back toward the same structural pattern when the underlying system did not change. AI is not a neutral tool that bends toward equity by default. It bends in whatever direction the data, the deployment, and the institutional leadership push it. The institutions that build fairness audits into their AI governance — that look at the tool's output across race, ethnicity, language, insurance, and ZIP code, that publish the audit, that act on the findings — will be the institutions whose patients get better care from clinical AI. The institutions that deploy the tools and do not audit will be the institutions where the next twenty years of disparity research will be conducted.

The emergency physician sitting in the chair has more leverage in this conversation than the literature credits. Procurement decisions reach the committee one way when the emergency medicine leadership has read the bias literature and another way when they have not. Tool selection bends one direction when the medical staff has read the relevant systematic reviews and another direction when they have not. The override patterns of the clinicians on the floor become the data the institution evaluates the vendor against, or they do not. The choice is one the emergency department leadership is making whether or not the leadership recognizes it as a choice.

AI Won't Wait. Neither Should You.

If your emergency department has deployed AI-augmented triage, disposition, length-of-stay prediction, or ambient documentation without a fairness audit across the populations you serve, you are operating clinical AI on the assumption that the historical data is correct. The Lurie Children's study, the spring 2026 systematic review, and the Frontiers equity review all argue the assumption is wrong. Consider enrolling in my course: AI in Emergency Medicine: Becoming AI Bulletproof. The course walks through the audit framework, the override documentation standard, and the procurement-side questions every department should be asking the vendor before the contract is signed.

If you're an emergency physician (or any clinician treating patients daily) trying to understand how AI will actually impact your clinical practice — not just the hype — I put together a free practical guide. You can download it here: AI in EM Survival Guide.

Dr. Chester "Chet" Shermer, MD, FACEP is a Professor of Emergency Medicine, TeleHealth, HEMS and Critical Care Transport, and State Surgeon for the Army National Guard. He is the founder of Global MedOps Command and the creator of AI in Emergency Medicine: Becoming AI Bulletproof.

Also on Medium: Read more from Dr. Shermer on Medium

Sources

1. Journal of Medical Internet Research, "Can Digital Tools Fix Bias in Mental Health Triage?" — citing the 2026 Lurie Children's pediatric ED triage study, May 2026, https://www.jmir.org/2026/1/e100947
2. PubMed, "Artificial Intelligence-Driven Triage in Pediatric Emergency Medicine," systematic review, May 2026, https://pubmed.ncbi.nlm.nih.gov/42137483/
3. WLRN, "The 'unintended consequences' of using AI in health insurance coverage decisions," May 2026, https://www.wlrn.org/health/2026-05-19/the-unintended-consequences-of-using-ai-in-health-insurance-coverage-decisions
4. Frontiers in Digital Health, "Artificial intelligence and digital health equity: a post-pandemic perspective," May 2026, https://www.frontiersin.org/journals/digital-health/articles/10.3389/fdgth.2026.1785700/full
5. HealthManagement.org, "Ethics of AI in Emergency Care," May 2026, https://healthmanagement.org/c/hospital/pharmacy/ethics-of-ai-in-emergency-care
6. Ground News, "Insurance AI Under Fire for Alleged Racial Bias" — NAIC AI Systems Evaluation Tool pilot, May 2026, https://ground.news/article/insurance-ai-under-fire-for-alleged-racial-bias