<scp>HADA</scp> : A Hybrid Anomaly Detection Approach Using Unsupervised Machine Learning

Abstract

ABSTRACT This paper introduces HADA, a novel hybrid unsupervised anomaly detection approach that addresses critical limitations in financial fraud detection through innovative integration of principal component analysis (PCA), isolation forest (IF), and agglomerative hierarchical clustering (AHC). Financial fraud detection faces fundamental challenges including severe class imbalance, label scarcity, and evolving fraud patterns in high‐dimensional transactional data. Unlike standalone methods or partial hybrids, HADA's architectural novelty enables simultaneous dimensionality stabilization, efficient anomaly scoring, and interpretable structure discovery within a unified pipeline. Comprehensive experiments on five financial datasets (credit card, IEEE‐CIS fraud detection, bank account fraud, mobile banking, and synthetic financial data) demonstrate that HADA significantly outperforms state‐of‐the‐art baselines and recent methods, achieving ROC‐AUC scores up to 0.94, 0.91, and 0.92 on key benchmarks. More importantly for operational deployment, HADA improves Precision@100 by 23%–32% over standalone IF while maintaining competitive recall. The method incurs moderate computational overhead but remains scalable with O ( n log n + k 2 ) complexity, making it suitable for real‐world deployment. Statistical validation across 10 independent runs confirms the robustness of these results (standard deviation &lt; 0.5% for performance metrics). Comparative analysis with eight recent anomaly detection methods shows HADA's superior performance, achieving 3.2%–7.9% higher ROC‐AUC than deep learning approaches while requiring less training time. Operational metrics demonstrate a 32% reduction in false positive costs and a 41% improvement in investigator efficiency compared to baseline methods. The proposed method addresses key limitations of existing approaches by providing both detection effectiveness and operational interpretability through hierarchical cluster groupings, making HADA particularly suitable for real‐world fraud monitoring systems where investigator review capacity is limited and explanatory capabilities are essential for regulatory compliance.