Sepsis AI: Why Early Promise Can Mislead Investors

The hype around AI in healthcare is that it can spot critical conditions like sepsis way earlier and better than we can now. But the path from a cool algorithm in a lab to a tool that actually works in a hospital is littered with methodological traps that can make a model look way better than it is. The Epic Sepsis Model (ESM) is a perfect case study of this problem, it’s a widely used proprietary algorithm, but its reported accuracy is all over the map depending on how and when you measure its performance.

The Epic Sepsis Model: Initial Promise and Independent Scrutiny

The Epic Sepsis Model, baked into Epic Systems’ electronic health record (EHR), is supposed to predict sepsis, the body’s life-threatening overreaction to infection. Early marketing from Epic suggested it was a strong performer which helped it get adopted in hospitals all over the US. But independent research quickly started telling a different story, making people question the model’s real-world value and the secrecy around these proprietary tools. A 2021 study in JAMA Internal Medicine tore through data on about 38,500 hospitalizations and found a shocking discrepancy with the original ESM: the model flat-out missed two-thirds of sepsis cases JAMA Internal Medicine 2021 Epic Sepsis Model study. As researchers told Fierce Healthcare, “the increase and growth in deployment of proprietary models has led to an underbelly of confidential, non-peer-reviewed model performance documents that may not accurately reflect real-world model performance.” That quote hit a nerve with clinicians and hospital AI governance teams. It pointed to a growing fear of a “data moat,” where vendors control all the performance data and validation methods, effectively hiding how well (or poorly) their tools actually work. In response to this and other critiques, Epic did release an updated version, ESM v2, sometime in late 2021 or early 2022.

The Nuance of “Accuracy”: Patient-Level vs. Prediction-Level Evaluation

The huge differences in reported performance for the ESM aren’t just about one version being “good” and another “bad.” It’s often about a fundamental disagreement in how to measure performance, specifically the difference between patient-level and prediction-level accuracy. Understanding this is key to figuring out if an AI tool is actually useful in a clinical setting. Researchers at the University of Michigan, including Tom Valley, Jenna Wiens, and Karandeep Singh, really drove this point home in a separate study covered by TechTarget and University of Michigan News. Their work, published in NEJM AI in February 2024, showed that if you looked across a patient’s entire hospital stay, the Epic Sepsis Model (likely an earlier version) seemed great, correctly flagging high-risk patients 87% of the time. This is a “patient-level” view. But that number can be very misleading for a doctor who needs an alert they can act on right now. The real story came out when the researchers looked only at the clinically useful time windows. When they measured accuracy only for predictions made before a patient officially met sepsis criteria, the model’s performance plummeted to 62%. It got even worse, down to 53%, when they restricted it to predictions made before a doctor ordered a blood culture, often the very first chance to intervene NEJM AI Epic Sepsis Model study. This “prediction-level” analysis proves a model might eventually get it right, but if the alert comes too late to change how you manage the patient, its clinical value is close to zero. The same team also raised the possibility that the model wasn’t discovering sepsis on its own but was just encoding “clinician suspicion that the patient has sepsis” already present in the chart data. So is the AI an early warning system, or just an expensive echo of what doctors are already thinking?

Epic’s Rebuttal and the Importance of Context

To be fair, you have to hear Epic’s side of the story. Epic has consistently disputed the methods of some of these independent studies. For example, responding to that 2021 JAMA Internal Medicine paper, Epic argued that the researchers used a hypothetical, super-low alert threshold that no hospital would actually configure in a live setting. They pointed out that even by the paper’s own math, the original model would have flagged 183 patients who might otherwise have been missed, showing how much these methodological choices can skew the results. Epic then launched its updated ESM v2 model in late 2021 or early 2022 to improve both accuracy and timing. And some early adopters have seen good things. Saint Luke’s Health System reported a 32% drop in their order-to-antibiotic turnaround time and a 16% reduction in their sepsis mortality index. This whole back-and-forth reveals the biggest challenge in AI validation: context. An AI model’s performance isn’t a fixed number. It changes based on how it’s plugged into clinical workflows, what alert thresholds are set, and the quality of the data feeding it. That’s why following Good Machine Learning Practice (GMLP) principles is so important, it’s about making sure models are built for the real world and continuously monitored so their performance doesn’t degrade over time. That said, a multicenter prospective validation of ESM v2, published in JAMA Network Open in February 2026, threw some cold water on the update. While the new model was better at early prediction, its performance varied wildly between hospitals, it still had a low positive predictive value (0.13-0.26), and it generated a huge number of alerts. That study, looking at 227,091 inpatient encounters, basically concluded that any hospital using ESM v2 needs to do its own local validation and build workflows to deal with the firehose of false positives.

Lessons for Clinicians and AI Governance Teams

So what does the whole Epic Sepsis Model story teach those of us working in health systems? * Scrutinize “Accuracy” Metrics: “Accuracy” is a loaded term. You have to ask if a vendor’s number reflects patient-level identification over an entire stay (which is mostly useless for timing) or prediction-level identification in a clinically actionable window. For a fast-moving killer like sepsis, only the second one matters.

• Demand Transparency and External Validation: The flood of proprietary, black-box algorithms means we must demand more transparency. A vendor’s “data moat” might be a business strategy, but it can’t come at the cost of patient safety. Health systems and regulators need to insist on seeing complete, independent, external validation data, not just confidential marketing PDFs.
• Consider the “What Responsible AI Does Differently” Model: Building a good AI isn’t just about hitting a high score in a lab. It means deeply understanding the clinical workflow and figuring out the exact moment an intervention will help. It means designing a model that finds a problem before a clinician is already suspicious, actually adding a new signal instead of just amplifying an existing one.
• Beware of Over-Reliance on Vendor-Reported Data: A vendor’s data is a starting point, nothing more. The discrepancies with the ESM show exactly why health systems must run their own validation studies or lean on solid, independent research before deploying any AI tool that could impact patient care. Frameworks for independent AI model validation The Epic Sepsis Model saga isn’t a failure of AI in healthcare. It’s a reality check. It’s a powerful lesson in just how complicated it is to deploy these technologies responsibly. The true success of a clinical AI lies in its ability to predict a problem when and how it matters most for the patient in the bed. As AI gets further embedded in clinical practice, a critical, evidence-based, and frankly skeptical approach to validation is the only way we’ll ensure patient safety and see any real benefit.