Listen to this article · 8 min listen

AI in healthcare has a lot of potential, but getting it to work safely is a messy, complicated business. If you’re a buyer or a procurement lead, you have to learn from the documented failures to sort out the real breakthroughs from the overblown marketing. Looking at the patterns from incidents on aihealthrisks.com gives you a practical way to vet these AI health solutions.

The Illusion of “AI” as a Monolith: What Was Actually Cleared?

A pattern I see over and over is vendors being deliberately fuzzy about what the “AI” in their product actually is, especially when it comes to regulatory clearance. Whether a product has “AI” is not the point. The real question is about specific, provable claims for its function and clearance. You have to distinguish between things like the hardware versus the software, or a simple wellness app versus a regulated medical device. The WHOOP wellness-claim dispute is a perfect example of this confusion. WHOOP sells a whole wellness platform, but the regulatory heat comes down on very specific claims and functions. Is the AI just giving you tips on well-being, or is it spitting out statements that sound like a diagnosis? The first gets a pass with little oversight, but the second demands serious validation and clearance as a SaMD (Software as a Medical Device), as outlined in the FDA guidance on SaMD definitions. Buyers need to hammer vendors on these points:

  • Which exact parts of your product have regulatory clearance (e.g., FDA 510(k), De Novo, CE Mark under EU MDR)?
  • What’s the precise scope of that clearance? Does it apply to the whole AI system, or just one small part of it? A company like Hello Heart does a good job of this, clearly separating its blood pressure monitor (the hardware) from its app and AI, and being specific about what’s cleared for which use. You need that kind of transparency.
  • Is the AI set up for Clinical Decision Support (CDS) or is it a full-on Diagnostic AI? The regulatory paths are completely different. A CDS might suggest things, but a Diagnostic AI makes calls on its own and is regulated like a device. Without getting this level of detail, buyers end up with products in a regulatory no-man’s-land, which is a huge risk for their organization and for patients.

    Unmasking the Model’s True Purpose: What is it Actually Trained to Predict?

Another big red flag I see in these cases is a gap between what the sales pitch claims and what the AI model was actually built to do. The 2019 algorithmic-bias study Original 2019 algorithmic bias study in Science is the classic cautionary tale here. An algorithm that was supposed to flag patients for extra care ended up being racist because it was trained to optimize for healthcare costs, not for actual health needs. The model worked as designed (predicting cost) but completely failed its real-world job (fair patient care), showing the massive difference between what an AI can predict and what it should predict to be useful in a clinic. You have to get to the bottom of the AI’s design:

  • What’s the specific outcome variable the model is trained on? Are we talking a direct clinical result like disease presence, or is it a proxy like billing codes or utilization, which can be full of bias and not reflect reality?
  • What datasets were used to train and test it? You absolutely have to know the demographics, clinical details, and sources of the training data to judge bias and see if the model will even work for your patient population. A big “data moat” might be a business advantage for the vendor, but that just means you have to be even more skeptical about what’s inside it.
  • How does the vendor handle algorithmic drift? AI models in a busy hospital setting will get worse over time because the real world changes and no longer looks like the old training data, a problem explained in this Explanation of algorithmic drift in healthcare AI. What’s their plan for monitoring performance, retraining the model, and re-validating it? When a vendor can give you straight, specific answers backed by clear methods and validation studies, that’s a good sign you’re dealing with a serious, responsible company.

    The Human Element: Who Verifies Outputs and Under What Time Pressure?

As we push more AI into healthcare, we have to seriously reconsider the role of human oversight and the very real danger of automation bias. The fact that there’s now Regulatory guidance on mitigating automation bias in AI-enabled medical devices shows how big this concern is. When an AI gives a recommendation, doctors might just trust it blindly and miss something critical. Even if the AI is incredibly accurate, the context of its use and the human review process are what really matter. You have to ask about the real-world workflow:

  • What is the human’s expected role when this AI is running? Is the AI supposed to be on its own, or is it a tool that requires a human to sign off on its findings every time?
  • What are the time pressures on the human reviewer? Forcing someone in a high-pressure job to quickly check AI outputs is just asking for automation bias to take over. How does the system’s design account for that reality?
  • What quality management system (QMS) and Good Machine Learning Practice (GMLP) are built in? These frameworks, like the FDA/Health Canada/MHRA GMLP principles, are how you build and maintain safe and effective AI/ML devices. A good AI solution is about more than a fancy algorithm. It’s about a smart, well-designed collaboration between a human and the AI that puts patient safety first. Vendors need to show that they know exactly where human experts are still essential.

    Conclusion

The common thread in all these documented failures is a desperate need for specificity and transparency. Vague claims about “AI” just don’t cut it. Procurement teams and buyers have to dig past the marketing slides and insist on real answers to these hard questions. By digging into what was actually cleared, what the model really predicts, and how the human fits into the loop, your organization can actually lower the risk of its AI investments. This focus ensures these powerful tools actually make patient care better instead of creating new ways for things to go wrong. This kind of due diligence isn’t a “best practice”, it’s the absolute minimum for adopting AI in healthcare responsibly.

Frequently Asked Questions

How do I determine if an AI health solution is genuinely innovative or just making overstated claims?

Buyers and procurement professionals should critically examine documented AI failures and patterns from incidents like those on aihealthrisks.com. Focus on specific, verifiable claims about the AI’s function and regulatory clearance status, rather than vague general statements about ‘AI’ presence.

What specific regulatory clearances should I look for when evaluating an AI health product?

Buyers must press vendors for clarity on what specific components have received regulatory clearance, such as FDA 510(k), De Novo, or CE Mark under EU MDR. It’s crucial to understand the precise scope of that clearance, whether it covers the entire AI system or only a discrete module, and if it’s cleared as Clinical Decision Support or Diagnostic AI.

How can I assess the potential for bias or misrepresentation in an AI health model?

Buyers need to investigate what specific outcome variable the AI model is trained to optimize or predict, ensuring it aligns with direct clinical endpoints rather than proxy variables. Additionally, understanding the demographics, clinical characteristics, and data sources of the training and validation datasets is fundamental to assessing potential biases and generalizability.

What questions should I ask about the data used to train and validate an AI health solution?

Inquire about the specific outcome variable the AI model is trained to optimize or predict, and whether it’s a direct clinical endpoint or a proxy. Also, ask about the datasets used for training and validation, including their demographics, clinical characteristics, and data sources, to assess potential biases and generalizability.

What is the importance of human oversight in AI health solutions?

Even with accurate AI, the context of its use and the role of human review are paramount to mitigate automation bias. Buyers need to understand the expected role of human oversight in the AI’s operational workflow and whether the AI is intended to be fully autonomous or requires human verification of its outputs.