Clinical AI: Why Good Models Fail in the Real World

People talk about AI in healthcare as either a total revolution or a complete disaster, but that misses the point for the teams doing AI governance, the clinical informaticists on the ground, and the investors trying to figure out deployment risk. A clinical AI tool can be brilliant in one hospital and a dud in the next, even with the same core algorithm. We have to get granular about why these performance gaps happen, because there’s a world of difference between a model’s logic being unreliable and its operational environment being a mess. The real question isn’t “did the deployment fail?”, it’s “where, specifically, was the point of failure?”

The Bifurcated Reality: Core Logic vs. Data Pipeline Integrity

Any clinical AI’s effectiveness boils down to two things that can succeed or fail on their own: the recommendation logic itself (the AI’s “brain”) and the data pipeline feeding it patient information from the hospital’s systems (its “nervous system”). If you mistake a pipeline failure for a logic failure, you’ll come to the wrong conclusions about the AI’s clinical value and whether it’s worth investing in. The classic example is IBM Watson for Oncology. Its core treatment recommendations were initially developed and tuned with oncologists from Memorial Sloan Kettering Cancer Center. Even after its assets were sold to Francisco Partners in 2022 and became Merative, that core logic designed to help clinicians was tested over and over in different hospitals around the world.

Concordance Studies: Evidence of Core Logic Performance

A whole body of peer-reviewed concordance studies, all indexed on PubMed Central, gives us a clear window into how Watson for Oncology’s core logic performed in the wild. These studies simply measured how often the AI’s recommendations agreed with the hospital’s own human oncology teams. The results showed consistent, and often high, rates of agreement. For example, in China, studies on advanced non-small-cell lung cancer found agreement around 73.3% which jumped to 90.3% when the AI’s “for consideration” options were included peer-reviewed study on Watson for Oncology concordance in China. A different cohort study in Thailand covering breast, colon, lung, and rectal cancers saw agreement rates in the 60-70% range peer-reviewed study on Watson for Oncology concordance in Thailand. Other research on colorectal, gastric, and thyroid cancers found similar significant, though variable, concordance. What this collection of international data tells us is that the tool’s underlying clinical reasoning, when given good data, was very much in line with expert human judgment.

MD Anderson’s Pilot: A Case Study in Deployment Infrastructure Failure

Then you have the highly publicized story of the IBM Watson pilot getting canceled at MD Anderson Cancer Center which seems to tell the opposite story. But it doesn’t. A University of Texas audit found the problem wasn’t a documented failure in the AI’s recommendations, but a total breakdown at the infrastructure level. Specifically, a failure to integrate with the EHR meant the tool was using outdated data University of Texas audit report on MD Anderson Watson for Oncology pilot. This is a critical distinction. The AI was reasoning correctly, but it was reasoning with bad inputs. It’s like asking a top physician to make a diagnosis using a patient’s chart from six months ago, completely unaware of new lab results that just came in. The doctor’s skill is intact, but the information is garbage, so the outcome is compromised. This is a massive lesson for any health system AI governance team: a great AI model is completely useless without clean data pipelines and solid EHR integration.

The Critical Distinction: Model Reliability vs. Data Currency

The gap between the concordance study results and the MD Anderson audit report demonstrates a core principle for deploying AI responsibly: an AI tool’s performance is chained to the quality and timeliness of its input data. The model’s reasoning, which was refined by oncologists at Memorial Sloan Kettering, was shown to be clinically plausible again and again. The MD Anderson story shows that even that well-trained model will fail if the operational plumbing can’t deliver clean, real-time patient data. Both the audit and the studies are valid. They just describe different problems. The audit points to a failure in deployment infrastructure and a dependency on data currency, which should be a top diligence item for any investor or hospital. The concordance studies show what the model’s core engine is capable of when it’s properly fueled.

Implications for AI Governance and Investment

This distinction gives AI governance teams a clear action plan. When you’re looking at an AI solution, you have to dig into the integration pathways and data governance just as hard as you look at the clinical validation studies. What’s the vendor’s QMS and ISO 13485 certification status? Can they demonstrate GMLP adherence? How do they plan to manage algorithmic drift over time? These are the questions that matter. Clinical informaticists need to be obsessed with ensuring data pipelines are solid, secure (meaning HIPAA / HITRUST / SOC 2 compliant), and built for real-time data sync. We talk about a “data moat” in terms of having proprietary training sets, but the operational data moat, the ability to consistently feed an AI high-quality, current real-world evidence (RWE), is just as important. Investors also need to get smarter than the headlines. A “failed deployment” does not mean the model is bad. Due diligence means asking what, specifically, broke. Was it a fundamental flaw in the AI’s logic, or did the surrounding infrastructure, data feeds, or hospital protocols fall apart? A company with a fantastic core AI but a messy integration plan has a completely different risk profile from a company with a fundamentally weak algorithm. Understanding that difference is how you spot the truly viable AI-native companies and evaluate real deployment risk. The path for clinical AI will have plenty of wins and operational stumbles. By digging into incidents like the MD Anderson pilot and putting them in context with the broader evidence from concordance studies, we can develop the evidence-literacy needed to innovate and deploy these tools safely.