Scalable Federated Learning for Massive Medical Image Classification: Tackling Noisy and Imbalanced Data

Abstract

The proliferation of medical imaging data, coupled with stringent privacy regulations, necessitates scalable and reliable classification methods to address the challenges of big data. Motivated by the critical need for accurate pneumonia detection from Chest X-rays (CXR) across diverse clinical settings, this research confronts the five fundamental challenges of big data: volume, velocity, variety, value, and veracity. In such scenarios, traditional centralized methods are often constrained by data heterogeneity, computational bottlenecks, and privacy risks. To overcome these constraints, we propose a Federated Learning (FL) system that distributes model training across multiple clients, thereby ensuring data privacy while efficiently managing large-scale datasets. Our method leverages transfer learning by fine-tuning a pre-trained VGG11 model and employs FedProx regularization to mitigate client drift arising from non-Independent and non-Identically Distributed (non-IID) data distributions. Furthermore, we introduce an innovative data partitioning technique that simulates real-world conditions by generating imbalanced label distributions with a Dirichlet process and injecting Gaussian noise to mimic image quality variations. By enabling distributed local training and dynamic learning rate adjustments, our approach effectively manages high-volume, high-velocity data while preserving data privacy. Experimental results demonstrate that our proposed method efficiently aggregates diverse and noisy client updates while achieving competitive performance in pneumonia classification.