Deep Multimodal Learning for Heart Sound Classification Using CNN, Transformer, and BiLSTM with Attention

Abstract

Phonocardiogram (PCG) signals offer a non-invasive, low-cost screening tool for cardiovascular diseases. However, their noisy and non-stationary nature makes automated classification challenging, and traditional methods often fail to capture complex spectral-temporal patterns. This study proposes a multimodal deep learning architecture for the binary classification of heart sounds (Healthy vs. Unhealthy). The hybrid model integrates Convolutional Neural Networks (CNNs), Transformer encoders, and Bidirectional Long Short-Term Memory (BiLSTM) networks with an attention mechanism. It utilizes an early-fusion feature extraction pipeline combining MFCCs, Mel-spectrograms, and Chroma descriptors. To ensure robust evaluation and prevent data leakage, SMOTE is applied exclusively to the training folds within a strict zero-leakage, patient-wise 5-fold cross-validation protocol. The proposed framework demonstrates exceptional performance, achieving an average accuracy of 91.67%, a sensitivity of 80.95%, a specificity of 94.46%, and an AUC-ROC of 96.50%. An ablation study confirms that integrating Transformer and BiLSTM modules significantly enhances diagnostic stability over baseline CNNs. Furthermore, with exactly 858,434 parameters (3.27 MB) and interpretable attention maps, this highly optimized model provides a robust assistive solution suitable for deployment in digital stethoscopes and mobile telemedicine systems.