Mammography’s AI Failure: A Warning for Healthcare Investors

The buzz around diagnostic AI, especially with generative models claiming they can spot things better than ever, feels a lot like déjà vu from 25 years ago. Back then, computer-aided detection (CAD) for mammography was supposed to be a revolution in finding cancer early. The way that story actually played out in clinics offers some tough lessons for today’s Healthcare AI Investors and Radiologist Group Executives, because it shows exactly why short-term reader studies are terrible predictors of long-term value and why you don’t see any meaningful return on investment without rigorous, longitudinal validation.

The Rise and Fall of First-Generation CAD in Mammography

In the late 90s and early 2000s, companies like Hologic were rolling out the first CAD systems for mammography. The idea was simple: the software would act as a “second reader,” flagging suspicious spots on a mammogram for the radiologist. This promised to improve cancer detection, picking up on subtle lesions the human eye might otherwise skim past. The initial diagnostic sensitivity data looked great, mostly coming from reader studies where radiologists would read cases with and without CAD’s help. That early data, combined with the general excitement for new tech, fueled massive adoption and, importantly, got reimbursement codes established. The FDA cleared these systems via the premarket approval (PMA) pathway, basing its decisions largely on those promising, but limited, diagnostic sensitivity figures FDA historical PMA documents for early CAD systems. But the excitement started to die down once long-term clinical data began to surface. What looked like better detection in a lab was, in practice, a huge spike in false positives. Radiologists, nudged by the CAD marks, would recall patients for more imaging or biopsies only to find nothing. This created a ton of patient anxiety, led to procedures that weren’t needed, and drove up healthcare costs, all without a corresponding improvement in patient outcomes.

Clinical Utility vs. Diagnostic Sensitivity: A Critical Distinction

The main problem with early CAD was a complete focus on diagnostic sensitivity while ignoring actual clinical utility. Sure, the CAD software flagged more things, but its inability to tell the difference between a harmless finding and a malignant one created a downstream nightmare. This wasn’t some secret. Major journals like the New England Journal of Medicine documented the whole mess. For example, large-scale research from the National Cancer Institute (NCI) revealed that despite CAD’s widespread use, it didn’t meaningfully improve cancer detection rates or reduce the number of cancers missed between screenings when compared to radiologists reading films on their own in real-world clinical settings NCI studies on CAD effectiveness in mammography. These conclusions blew the initial, optimistic projections out of the water. The data from these peer-reviewed studies told a consistent story: while CAD made radiologists identify more potential issues, the overwhelming majority of these extra findings were benign. In practical terms, this meant the positive predictive value of a CAD-assisted read was often low, creating a higher burden for both patients and the healthcare system. When researchers analyzed the historical sensitivity, specificity, and false positive rates from traditional CAD in large, long-term patient groups, they couldn’t show a net clinical benefit. Instead, they often found lower specificity and no real gain in cancer detection over unassisted reads.

“Without a PCCP, every time your cardiac AI model retrains on new data, you need a new 510(k), that’s unscalable.” This point about adaptive AI gets right to the regulatory and clinical headache of proving an algorithm still works over time, a lesson CAD learned the hard way.

Lessons for Modern Diagnostic AI Developers and Investors

The history of CAD is a powerful cautionary tale for anyone investing in or running a radiology group today. The hype around early CAD and the current claims about generative diagnostic AI are eerily similar. Both promise to make humans better, but the only measure of success that matters is demonstrable, long-term clinical utility, which is a far cry from just hitting technical performance metrics in a lab.

Why Long-Term Clinical Utility Trials are Superior to Short-Term Reader Studies

CAD got its FDA clearance and initial market traction based almost entirely on short-term reader studies. These studies can establish a baseline for diagnostic performance, but they are poor predictors of real-world impact. Why? They’re often conducted in a vacuum with a limited set of cases (frequently “enriched” with known cancers) and don’t reflect the chaos of daily clinical practice. Any investor doing due diligence on a modern diagnostic AI company should be demanding evidence from long-term clinical utility trials. These are the studies that track actual patient outcomes over time, measuring things like:

• A real reduction in false positives and the associated downstream costs of workups.
• Actual improvement in cancer detection rates (like a lower number of interval cancers).
• Changes to patient management and treatment decisions.
• The overall cost-effectiveness and return on investment for a hospital or clinic.

A company that builds its product and validation strategy around Good Machine Learning Practice (GMLP), planning for problems like algorithmic drift from the start, is in a much stronger position to deliver a product that’s still being used and paid for in five years.

The Imperative of Real-World Evidence (RWE)

Randomized controlled trials are great, but AI is constantly changing, which means you need a strong plan for collecting and analyzing Real-World Evidence (RWE). This means continuously tracking how the AI performs in different clinics with different patient populations, pulling data from electronic health records, patient registries, and insurance claims. RWE is how you spot early signs of a model’s performance degrading or developing biases that you’d never see in the clean data of the initial validation studies. For an investor, seeing a company with a clear strategy for generating and using RWE is a strong sign that you’re dealing with a mature, responsible AI-native organization.

Moving Beyond the Hype: A Call for Responsible AI Validation

The CAD story in mammography shows us a simple truth: being technologically sophisticated doesn’t guarantee a clinical benefit. The initial promise of any AI has to be tested against the messy reality of patient care and whether it actually saves money or makes a clinic run better. The lessons from CAD aren’t about giving up on AI’s potential. They’re about being much smarter in how we validate and deploy it. For investors, that means looking past a slick demo with impressive sensitivity numbers. It means digging into the clinical trial methodology and backing companies that can prove long-term clinical utility and cost-effectiveness. It means asking the hard questions, like how an AI model will hold up across diverse populations and what the plan is to monitor its performance a year after deployment. For radiology group executives, it means demanding proof that a new AI tool will actually improve patient outcomes and clinic efficiency, not just add another layer of alerts and increase the false positive rate. The history of CAD is the perfect reminder that real progress in healthcare AI is measured by what it improves for patients and providers, not just by what it can detect on a scan. Review of current FDA guidance on AI/ML-based SaMD validation