Beyond One-Time Validation: A Framework for Adaptive Validation of Prognostic and Diagnostic AI-based Medical Devices

Abstract

Prognostic and diagnostic AI-based medical devices hold immense promise for advancing healthcare, yet their rapid development has outpaced the establishment of appropriate validation methods. Existing approaches often fall short in addressing the complexity of practically deploying these devices and ensuring their effective, continued operation in real-world settings. Building on recent discussions around the validation of AI models in medicine and drawing from validation practices in other fields, a framework to address this gap is presented. It offers a structured, robust approach to validation that helps ensure device reliability across differing clinical environments. The primary challenges to device performance upon deployment are discussed while highlighting the impact of changes related to individual healthcare institutions and operational processes. The presented framework emphasizes the importance of repeating validation and fine-tuning during deployment, aiming to mitigate these issues while being adaptable to challenges unforeseen during device development. The framework is also positioned within the current US and EU regulatory landscapes, underscoring its practical viability and relevance considering regulatory requirements. Additionally, a practical example demonstrating potential benefits of the framework is presented. Lastly, guidance on assessing model performance is offered and the importance of involving clinical stakeholders in the validation and fine-tuning process is discussed.