Enhancing Android Malware Detection with System Call-Based Adversarial Learning Using GANs: Evaluating and Improving IDS Robustness

Abstract

Introduction: The widespread use of Android devices has made them prime targets for sophisticated malware. This study proposes a novel Android malware detection method based on dynamic system call behavior. Methods: We use Generative Adversarial Networks (GANs) to create adversarial feature vectors from system call frequencies. Unlike traditional approaches that focus on static features or network traffic, our method emphasizes dynamic analysis, offering deeper insights into malware detection and IDS robustness. Five Intrusion Detection System (IDS) models—Random Forest (RF), Linear Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), and Multilayer Perceptron (MLP)—were evaluated against GAN-generated adversarial samples. Results: Experimental results show that adversarial examples, as assessed using the Fréchet Inception Distance (FID), closely resemble real data, resulting in notable performance drops across all IDS models. For example, RF accuracy fell from 99.36\% to 38.43\%. Discussion: Adware and Banking malware were especially affected, as revealed by confusion matrix analysis. To address this, we applied adversarial training, which substantially improved IDS resilience. Conclusion: These results underscore the need for robust IDS models that can withstand adversarial manipulation in dynamic system call analysis.