Performance Analysis of ML Algorithms for Fraud Detection in Digital Financial Transactions

Abstract

Digital financial transactions are increasingly vulnerable to fraud, driving the adoption of machine learning (ML) for detection. However, ML models often overfit on imbalanced transaction data. This results in high apparent accuracy but poor generalization to real-world scenarios. This study compares seven ML algorithms—Logistic Regression (LR), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), Gradient Boosting Machine (GBM), Isolation Forest (IF), and K-Means (KM) clustering—for detecting credit card fraud. We use a large, publicly available Kaggle dataset comprising approximately 1.85 million simulated transactions from January 2019 to December 2020. Models are trained on the original imbalanced data and a resampled version using SMOTE-ENN to correct for class imbalance. To account for the skewed class distribution, performance is evaluated using multiple metrics: accuracy, precision, recall, F1-score, Matthews correlation coefficient, and Cohen’s kappa. This multi-metric approach ensures a more balanced assessment of model performance, particularly in detecting minority-class fraud instances. Results show that ensemble models (RF, GBM) achieve the highest fraud-detection rates but are prone to overfitting under class imbalance. LR proves most resilient on unbalanced data, while resampling significantly improves recall for fraud detection across all models. Feature importance analysis highlights transaction amount, merchant category, transaction timing, and customer demographics as key fraud indicators.