How to build a safe path to AI in Healthcare

Healthcare adheres to an elevated standard. This is evident in the rigorous training required for providers, the stringent safety protocols for life sciences professionals, and the stringent data and privacy requirements for healthcare analytics software. The stakes in healthcare are higher, as errors can have life-or-death consequences. Therefore, every innovation must be approached with utmost caution.

Accordingly, the adoption of AI in healthcare has faced significant challenges and controversies, more so than in other industries. This is largely due to widespread skepticism regarding ambitious claims about AI’s potential to revolutionize cancer treatment, coupled with the relatively slow integration of AI technologies across various healthcare disciplines. Concerns about data security, privacy, and accuracy have been at the forefront of these discussions. However, with the rapid advancement of AI and generative AI (GenAI) technologies and the emergence of several promising real-world applications, a clearer and safer path for AI integration in healthcare is beginning to take shape.

Based on my firm’s work developing AI and GenAI solutions to help the world’s leading healthcare payers work more effectively with providers, we’ve identified the following criteria as the keys to building trust in the technology.

1. Healthcare Domain Expertise: It cannot be said enough that anyone developing AI-driven models for healthcare needs to understand the unique use cases and stringent data security and privacy requirements – and the detailed nuances of how this information will be used – in the specific healthcare setting where the technology will be deployed. For solutions used in the prior authorization process, for example, developers need to understand that any missing data point or any break in continuity in tracking the complete patient journey could result in coverage denials or approvals that could have life-threatening consequences. Similarly, they must build safety protocols and checks and balances into the technology to validate data lineage and recognize potentially erroneous or incomplete information.
2. Data Security, Privacy, and Accuracy: One of the major hurdles to implementing AI in healthcare is the risk of accidental exposure to private health information. In order for any AI-powered healthcare analytics to be accurate enough to provide meaningful insights into real-world patient journeys, they must be populated with robust de-identified data. However, to avoid the risk of reidentification or breach of privacy when using that data in a large language model (LLM), it is important to implement several risk mitigation strategies. These include multiple techniques such as role-based access control in multiple layers of AI applications, fine-grained authorization for vector databases, protected health information (PHI) and personally identifiable information (PII) encryption in vector database metadata, PHI/PII masking before sending to LLM as context, hosting the LLMs behind firewalls, generating safe responses using prompt engineering, fine-tuning the open-source LLM to classify prompts and responses as safe or unsafe, and use of secure retrieval augmented generation (RAG) architecture to prevent hallucinations. In enterprise implementations, different combinations of these techniques will be applied. The decision over which to use when will be based on factors such as performance and cost.
3. Observability and Explainability: AI-supported decisions in healthcare must be transparent and explainable to ensure trust and facilitate informed decision-making by healthcare providers. Observability and explainability are critical to understanding AI behavior, identifying errors, and ensuring compliance with regulatory standards. Some important steps that need to be taken to monitor and address these issues include specific communication and documentation regarding GenAI usage parameters, real-time input and output logging, and consistent evaluation against performance metrics and benchmarks.
4. Focus on Scalability: Beyond all of the task-specific fine-tuning and data privacy controls that need to be incorporated into the process, it is also important to remember that the technology needs to be accessible, affordable, and able to evolve and grow as new technology is developed and as new challenges emerge. This is where the marriage of domain and data expertise comes in. To build effective GenAI solutions that help nurse care managers effectively review dozens of claims in the time it took them to review just one or to help spot anomalies that trigger an earlier intervention on a chronic condition, technology teams need to take a modular approach to tech development, one that leans on experts with domain and tech expertise who continually update underlying tech infrastructure based on new data and new capabilities.

The fact is AI is currently revolutionizing the way we diagnose and treat disease and the companies that embrace it responsibly are finding a faster path to earlier interventions and better outcomes. While getting there may not be as easy as firing up ChatGPT and asking it to identify at-risk patients or evaluate patient medical history to gauge whether or not it is safe for them to receive an experimental new therapy, the technology is transforming the way care is delivered.

To learn more, visit us here.

About the author
Jay Nambiar is chief technology officer, healthcare, and Prashant Renu is vice president, AI solutions, healthcare at EXL, a leading data analytics and digital operations and solutions company.