Overfitting in Histopathology Model Training: The Need for Customized Architectures

Abstract

This study investigates the critical problem of overfitting in deep learning models applied to histopathology image analysis. We show that simply adopting and fine-tuning large-scale models designed for natural image analysis often leads to suboptimal performance and significant overfitting when applied to histopathology tasks. Through extensive experiments with various model architectures, including ResNet variants and Vision Transformers (ViT), we show that increasing model capacity does not necessarily improve performance on histopathology datasets. Our findings emphasize the need for customized architectures specifically designed for histopathology image analysis, particularly when working with limited datasets. Using Oesophageal Adenocarcinomas public dataset, we demonstrate that simpler, domain-specific architectures can achieve comparable or better performance while minimizing overfitting.