Skip to main content
SpringerLink
Log in

Multimodal image fusion on ECG signals for congestive heart failure classification

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The electrocardiogram, or ECG, records electrical signals from the heart to detect various heart conditions. It helps in the diagnosis of important cardiovascular diseases such as myocardial infarction (MI), cardiac arrhythmias, and congestive heart failure (CHF). Existing machine learning models and sophisticated deep learning techniques have primarily focused on using electrocardiogram (ECG) signals exclusively. However, to overcome the limitations of this approach, researchers have explored multimodal image-based fusion techniques, which have shown promising results, particularly when applied to real-time data such as ECG signals. In this work, two Multimodal Image-based Fusion techniques, Multimodal Image Fusion (MIF) and Multimodal Feature Fusion (MFF) are being used for the better classification of the ECG signals. The raw ECG data is first converted to create images of different modalities and then fused for classification. The different modality images formed are Gramian Angular Field (GAF), Markov Transition Field (MTF), and Recurrent Plot (RP) images. In the MIF method, the fused image of the three different modality images is provided to the convolutional neural network (CNN) for classification. When applying the MFF method, the features are first extracted from the obtained modality images by entering them in CNN and then these images are fused to get the required unique information. In this research, ECG5000 dataset is used for analysis. A newly designed model is used to classify the severity of Congestive Heart Failure (CHF) and attains accuracy using the proposed model is 97.79% for MIF and 98.19% for MFF methods respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability and access

The study’s data was taken from website and is freely accessible. We thank the authors and collaborators for making the original data freely available.

References

  1. Sun L, Lu Y, Yang K, Li S (2012) ECG analysis using multiple instance learning for myocardial infarction detection. IEEE Trans Biomed Eng 59(12):3348–3356

    Article  Google Scholar 

  2. Tiwari S, Agarwal S (2021) A Shrewd Artificial Neural Network-Based Hybrid Model for Pervasive Stress Detection of Students Using Galvanic Skin Response and Electrocardiogram Signals. Big Data 9(6):427–442. Epub 2021 Nov 30. PMID: 34851743. https://doi.org/10.1089/big.2020.0256

  3. Xia Y, Liu X, Wu D, Xiong H, Ren L, Xu L, Wu W, Zhang H (2017) Influence of beat-to-beat blood pressure variability on vascular elasticity in hypertensive population. Sci Rep 7(1):1–8

    Article  Google Scholar 

  4. Tiwari S, Agarwal S (2022) An optimized hybrid solution for IoT based lifestyle disease classification using stress data

  5. Bravi A, Longtin A, Seely AJ (2011) Review and classification of variability analysis techniques with clinical applications. BioMedical Engineering OnLine 10(1):1–27

    Article  Google Scholar 

  6. İşler Y, Kuntalp M (2007) Combining classical HRV indices with wavelet entropy measures improves to performance in diagnosing congestive heart failure. Comput Biol Med 37(10):1502–1510

    Article  Google Scholar 

  7. Hussain L, Awan IA, Aziz W, Saeed S, Ali A, Zeeshan F, Kwak KS (2020) Detecting Congestive Heart Failure by Extracting Multimodal Features and Employing Machine Learning Techniques. BioMed Res Int 2020, Article ID 4281243, 19 pages . https://doi.org/10.1155/2020/4281243

  8. Tiwari S, Agarwal S (2022) Data Stream Management for CPS-based Healthcare: A Contemporary Review. IETE Tech Rev 39(5):987–1010. https://doi.org/10.1080/02564602.2021.1950578

    Article  Google Scholar 

  9. Sidek KA, Khalil I, Jelinek HF (2014) ECG biometric with abnormal cardiac conditions in remote monitoring system. IEEE Trans Syst Man Cybern Syst 44(11):1498–1509

    Article  Google Scholar 

  10. Huang B, Yang F, Yin M, Mo X, Zhong C (2020) A Review of Multimodal Medical Image Fusion Techniques. Comput Math Methods Med 2020, Article ID 8279342, 16 pages. https://doi.org/10.1155/2020/8279342

  11. Yu SN, Lee MY (2012) Bispectral analysis and genetic algorithm for congestive heart failure recognition based on heart rate variability. Comput Biol Med 42(8):816–825

    Article  Google Scholar 

  12. Al Rahhal MM, Bazi Y, Almubarak H, Alajlan N, Al Zuair M (2019) Dense Convolutional Networks With Focal Loss and Image Generation for Electrocardiogram Classification. IEEE Access 7:182225–182237. https://doi.org/10.1109/ACCESS.2019.2960116

    Article  Google Scholar 

  13. Diker A, Cömert Z, Avcı E, Toğaçar M, Ergen B (2019) A Novel Application based on Spectrogram and Convolutional Neural Network for ECG Classification. 2019 1st International Informatics and Software Engineering Conference (UBMYK), Ankara, Turkey, pp. 1–6. https://doi.org/10.1109/UBMYK48245.2019.8965506

  14. Chen W, Zheng L, Li K, Wang Q, Liu G, Jiang Q (2016) A Novel and Effective Method for Congestive Heart Failure Detection and Quantification Using Dynamic Heart Rate Variability Measurement. Plos One 11(11):e0165304

    Article  Google Scholar 

  15. Chen W, Liu G, Su S, Jiang Q, Hung N (2017) A CHF detection method based on deep learning with RR intervals. In: International conference of the IEEE engineering in medicine and biology society IEEE

  16. Wang L, Zhou W, Liu N, Xing Y, Zhou X (2018) CHF Detection with LSTM Neural Network. 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, pp. 514–517. https://doi.org/10.1109/EMBC.2018.8512300

  17. Zhang Z, Dong J, Luo X, Choi KS, Wu X (2014) Heartbeat classification using disease-specific feature selection. Comput Biol Med 46:79–89

    Article  Google Scholar 

  18. RecuPlots and CNNs for Time-Series Classification. [Online]. Available: https://www.kaggle.com/tigurius/ recuplots-and-cnns-for-time-series-classification. Accessed 14 July 2021

  19. Wang Z, Oates T (2015) Imaging time-series to improve classification and imputation. In: Proc. 24th Int. Joint Conf. Artif. Intell., pp. 1–7

  20. Yang C-L, Chen Z-X, Yang C-Y (2019) Sensor classification using convolutional neural network by encoding multivariate time series as two dimensional colored images. Sensors 20(1):168

    Article  Google Scholar 

  21. Eckmann JP, Kamphorst SO, Ruelle D (1995) Recurrence plots of dynamical systems. World Sci. Ser. Nonlinear Sci. A 16:441–446

    Google Scholar 

  22. . RecuPlots and CNNs for Time-Series Classification. [Online]. Available: https://www.kaggle.com/tigurius/recuplots-and-cnns-for-time-series-classification. Accessed 14 July 2021

  23. Wang Z, Oates T (2015) Encoding time series as images for visual inspection and classification using tiled convolutional neural networks. In: Proc. 29th AAAI Conf. Artif. Intell., pp. 1–7

  24. Ahmad Z, Khan N (2021) CNN-based multistage gated average fusion (MGAF) for human action recognition using depth and inertial sensors. IEEE Sensors J 21(3):3623–3634

    Article  Google Scholar 

  25. Ahmad Z, Tabassum A, Guan L, Khan NM (2021) ECG Heartbeat Classification Using Multimodal Fusion. In: IEEE Access, vol. 9, pp. 100615–100626. https://doi.org/10.1109/ACCESS.2021.3097614

  26. Pedro M, Duarte F, Hugo G, André C (2021) Robust Anomaly Detection in Time Series through Variational AutoEncoders and a Local Similarity Score. 91–102. https://doi.org/10.5220/0010320500910102

  27. Asif M, Mishra S, Vinodbhai MT, Tiwary US (2022) Emotion Recognition with Temporarily Localized ’Emotional Events’ in Naturalistic Context. arXiv:2211.02637

Download references

Acknowledgements

The authors are grateful to the Ministry of Education and the Indian Institute of Information Technology, Allahabad, for providing the necessary materials required to complete this work.

Author information

Author notes

  1. Sonali Agarwal contributed equally to this work.

Authors and Affiliations

  1. Indian Institute of Information Technology, Allahabad, Jhalwa, Prayagraj, 211001, Uttar Pradesh, India

    Riya Panchal, Sadhana Tiwari & Sonali Agarwal

Authors
  1. Riya Panchal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sadhana Tiwari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sonali Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The work was equally contributed to by all authors.

Corresponding author

Correspondence to Sadhana Tiwari.

Ethics declarations

Ethical and informed consent for data used

The data used in this work has already been published and freely accessible to all. No attempt has been taken to re-identify any of those individuals or things that the data represents.

Competing interests

The authors have no competing interests to declare that are relevant to the content of this work.

Conflicts of interests

All authors declare that they have no conflicts of interest in the presented work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Panchal, R., Tiwari, S. & Agarwal, S. Multimodal image fusion on ECG signals for congestive heart failure classification. Multimed Tools Appl (2024). https://doi.org/10.1007/s11042-024-19052-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-024-19052-8

Keywords

Search

Navigation