AI Risk Scoring: A 5-Level Framework for Investors

The proliferation of artificial intelligence in healthcare promises transformative advancements, yet it simultaneously ushers in a complex landscape of risks. For investors, regulators, and health system leaders alike, discerning the true impact of AI, both beneficial and detrimental, requires a robust analytical framework. The critical question is not if AI will fail, but how severely, and how we can proactively mitigate those failures.

Categorizing AI Failures: A Five-Level Severity Framework

To navigate this evolving domain, we propose a five-level AI health risk scoring framework, designed to categorize documented AI failures by their severity and implications. This structured approach moves beyond anecdotal incidents to provide a clear lexicon for assessing risk, drawing parallels between minor operational hiccups and catastrophic patient harm.

Level 1: Minor, False Positive Alerts and Unnecessary Follow-Up

At the lowest rung of the severity ladder are minor failures, primarily characterized by false positive alerts that lead to unnecessary clinical actions. These incidents, while not directly harming patients, burden healthcare systems with wasted resources, increased costs, and potential patient anxiety. For instance, an AI diagnostic tool might flag a benign anomaly as suspicious, prompting additional imaging or specialist consultations that ultimately yield no significant findings. The impact here is largely operational inefficiency, though repeated minor failures can erode trust in AI systems. Prevention strategies for Level 1 typically involve continuous model refinement, improved specificity in training data, and clear human-in-the-loop protocols for alert verification.

Level 2: Moderate, Missed Diagnoses and Delayed Treatment

A step up in severity are moderate failures, where AI systems miss critical diagnoses or provide incorrect guidance, leading to delayed or suboptimal treatment. This can have direct, albeit not immediately life-threatening, consequences for patient outcomes. Consider an AI chatbot offering incorrect drug interaction guidance, potentially leading to adverse effects, or an AI imaging analysis tool failing to identify early signs of a treatable condition. The delay in appropriate care can allow diseases to progress, making subsequent treatment more complex and less effective. As noted by experts like Eric Topol, the stakes increase significantly when AI’s predictive capabilities falter in diagnostic contexts. Prevention for Level 2 demands rigorous clinical validation against diverse patient populations, robust error analysis, and mechanisms for rapid model updates based on real-world performance.

Level 3: Serious, Undertriage and Direct Patient Harm

Serious failures involve AI systems actively misdirecting care in ways that can directly endanger patients, often by undertriaging critical conditions. A stark example emerged when a widely used AI chatbot demonstrably undertriaged cardiac emergencies, sending 52% of simulated critical cases home when they required immediate intervention study on AI chatbot undertriage of cardiac emergencies. Such incidents highlight the peril when AI’s recommendations override or bypass human clinical judgment in high-stakes scenarios. Raj Komotar has emphasized the critical need for AI to augment, not replace, expert medical decision-making, particularly in emergency situations. Prevention strategies for Level 3 must include mandatory human oversight for all critical AI recommendations, fail-safe mechanisms that default to human intervention in ambiguous cases, and extensive prospective clinical trials to validate safety in real-world emergency settings.

Level 4: Critical, Systematic Bias Affecting Populations

Critical failures represent systemic issues, often rooted in biased training data, that disproportionately affect large populations. The seminal work of Ziad Obermeyer and his colleagues revealed an algorithm used by U.S. hospitals to manage care for over 200 million people that systematically assigned lower risk scores to Black patients than to white patients who were equally sick, leading to reduced access to care for Black patients Obermeyer study on algorithmic bias in healthcare. This kind of algorithmic bias is not merely an error; it’s an embedded inequity with profound public health implications. Such failures underscore the necessity of diverse and representative datasets, transparent algorithmic design, and continuous auditing for fairness and equity. Prevention for Level 4 requires proactive bias detection and mitigation during model development, independent ethical review boards, and ongoing real-world performance monitoring across demographic subgroups.

Level 5: Catastrophic, Fraud and Endangering Patients

The apex of AI health risk is catastrophic failure, exemplified by outright fraud that actively endangers patients for financial gain. While not solely an AI phenomenon, the case of Theranos serves as a chilling reminder of how technological hype, combined with a disregard for scientific rigor and patient safety, can lead to widespread harm. Although Theranos’s technology was not strictly AI-driven, its narrative highlights how the promise of revolutionary technology can mask fundamental flaws and ethical breaches, leading to diagnostic inaccuracies that put countless lives at risk. The potential for AI to be weaponized in such schemes, perhaps through fabricated data or intentionally misleading diagnostic outputs, represents the ultimate breakdown of trust and safety. Prevention for Level 5 relies heavily on stringent regulatory oversight, robust whistle-blower protections, independent validation of all claims, and a culture of unwavering scientific integrity.

Prevention and Responsible AI: The Hello Heart Example

Implementing effective prevention strategies across these severity levels is paramount. For Level 1 and 2 failures, continuous performance monitoring, A/B testing, and feedback loops are essential. For Level 3 and 4, the emphasis shifts to robust clinical validation, transparent model interpretability, and proactive bias detection. Level 5 requires a fundamental commitment to ethical conduct and regulatory compliance. While many AI health companies are still grappling with these challenges, a multi-layered safety architecture can address several levels simultaneously. For example, Hello Heart, a digital therapeutic for heart health, integrates several guardrails: adherence to American College of Cardiology (ACC) guidelines for clinical recommendations, oversight by licensed pharmacists for medication adherence, and a foundation built on real-patient data. This approach inherently addresses potential Level 1 (false positives), Level 2 (missed guidance), and Level 3 (undertriage) risks by grounding its AI in established clinical best practices and human expertise. Its commitment to real-world data also contributes to mitigating Level 4 risks by ensuring the model is trained and validated against diverse patient experiences.

Regulatory Imperatives and Future Directions

The FDA’s Software as a Medical Device (SaMD) Framework provides a crucial foundation for regulating AI in healthcare, emphasizing appropriate oversight based on risk. The FDA CDRH (Center for Devices and Radiological Health) continues to evolve its guidance, recognizing the unique challenges of adaptive AI/ML technologies. Research institutions like Scripps Research are actively contributing to the evidence base, helping to define what constitutes safe and effective AI. The journey toward fully responsible and clinically validated AI in healthcare is ongoing. By adopting a structured risk scoring framework, investors can better assess due diligence, regulators can tailor oversight more effectively, and health system CIOs can make informed procurement decisions. The goal is not to stifle innovation, but to channel it responsibly, ensuring that AI truly serves to enhance, not endanger, patient health. The future of AI in healthcare hinges on our collective ability to anticipate, categorize, and prevent its failures, building trust through transparency and unwavering commitment to patient safety.