Skip to main content
SpringerLink
Log in

Hybrid densenet with long short-term memory model for multi-modal emotion recognition from physiological signals

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recognition of emotions from multi-modal physiological signals is one among the toughest tasks prevailing amid the research communities. Most existing works have focused on emotion recognition (ER) from single modal signals, which is now ineffective. Certain models considered multiple modalities, but the results obtained are not satisfactory, and there is still a possibility for improvement in accuracy. Therefore, this work introduces a novel and effective mechanism by embedding multiple techniques to achieve the required task. The projected approach includes stages like pre-processing, signal-to-image conversion, feature extraction, feature selection and classification. Each signal modality is separately pre-processed, and the results are provided to a complex dual tree with fast lifting wavelet transform (CTFL-WT) to convert the signals into images. The converted images are sent to the channel attentive squeezenet (CASN) model for feature extraction. The obtained features are then reduced with the help of an adaptive arithmetic optimization algorithm (AAOA). The reduced features are then provided to the hybrid densenet with long short term memory (DLSTM) for accurate labelling. The projected work resulted in the classification of three different emotions such as neutral, stress and amusement. The implementations are performed in the Python platform, and the evaluations are done using the wearable stress and affect detection (WESAD) dataset. In comparison, the proposed work resulted in an overall accuracy value of 99% and an overall F1-score value of 97.84%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Data availability

Data sharing is not applicable to this article.

References

  1. Jiang Y, Li W, Hossain MS, Chen M, Alelaiwi A, Al-Hammadi M (2020) A snapshot research and implementation of multi-modal information fusion for data-driven emotion recognition. Inf Fusion 53:209–221

    Article  Google Scholar 

  2. Ahmad Z, Khan N (2022) A Survey on Physiological Signal-Based Emotion Recognition. Bioengineering 9(11):688

    Article  Google Scholar 

  3. Hasnul MA, Aziz NA, Alelyani S, Mohana M, Aziz AA (2021) Electrocardiogram-based emotion recognition systems and their applications in healthcare—a review. Sensors 21(15):5015

    Article  Google Scholar 

  4. Saganowski S, Dutkowiak A, Dziadek A, Dzieżyc M, Komoszyńska J, Michalska W, Polak A, Ujma M, Kazienko P (2020) Emotion recognition using wearables: A systematic literature review-work-in-progress. In2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), p 1–6

  5. Liakopoulos L, Stagakis N, Zacharaki EI, Moustakas K (2021) CNN-based stress and emotion recognition in ambulatory settings. In2021 12th international conference on information, intelligence, systems & applications (IISA), p 1–8

  6. Wijasena HZ, Ferdiana R, Wibirama S (2021) A survey of emotion recognition using physiological signal in wearable devices. In2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS), p 1–6

  7. Baghizadeh M, Maghooli K, Farokhi F, Dabanloo NJ (2020) A new emotion detection algorithm using extracted features of the different time-series generated from ST intervals Poincaré map. Biomed Signal Process Control 59:101902

    Article  Google Scholar 

  8. Li W, Zhang Z, Song A (2021) Physiological-signal-based emotion recognition: An odyssey from methodology to philosophy. Measurement 172:108747

    Article  Google Scholar 

  9. Mendoza A, Cuno A, Condori-Fernandez N, Lovón WR (2020) An evaluation of physiological public datasets for emotion recognition systems. In Information Management and Big Data: 7th Annual International Conference, SIMBig 2020, Lima, Peru, October 1–3, 2020, Proceedings. Springer International Publishing, Cham, p 90–104

  10. Montero Quispe KG, Utyiama DM, Dos Santos EM, Oliveira HA, Souto EJ (2022) Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals. Sensors 22(23):9102

    Article  Google Scholar 

  11. Abdullah SM, Ameen SY, Sadeeq MA, Zeebaree S (2021) Multi-modal emotion recognition using deep learning. J Appl Sci Technol Trends 2(02):52–58

    Google Scholar 

  12. Zhang J, Yin Z, Chen P, Nichele S (2020) Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review. Inf Fusion 59:103–126

    Article  Google Scholar 

  13. Egger M, Ley M, Hanke S (2019) Emotion recognition from physiological signal analysis: A review. Electron Notes Theor Comput Sci 343:35–55

    Article  Google Scholar 

  14. Hassan MM, Alam MG, Uddin MZ, Huda S, Almogren A, Fortino G (2019) Human emotion recognition using deep belief network architecture. Inf Fusion 51:10–18

    Article  Google Scholar 

  15. Suhaimi NS, Mountstephens J, Teo J (2020) EEG-based emotion recognition: a state-of-the-art review of current trends and opportunities. In: Computational intelligence and neuroscience, vol 2020, p 8875426. https://doi.org/10.1155/2020/8875426

    Chapter  Google Scholar 

  16. Yan M, Deng Z, He B, Zou C, Wu J, Zhu Z (2022) Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion. Biomed Signal Process Control 71:103235

    Article  Google Scholar 

  17. Bhatti A, Behinaein B, Hungler P, Etemad A (2022) AttX: Attentive cross-connections for fusion of wearable signals in emotion recognition. arXiv preprint arXiv:2206.04625

  18. Mekruksavanich S, Hnoohom N, Jitpattanakul A (2022) A Deep Residual-based Model on Multi-Branch Aggregation for Stress and Emotion Recognition through Biosignals. In 2022 19th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), p 1–4

  19. Theerthagiri P (2023) Stress emotion recognition with discrepancy reduction using transfer learning. Multimed Tools Appl 82(4):5949–5963

    Article  Google Scholar 

  20. Fouladgar N, Alirezaie M, Främling K (2022) CN-waterfall: a deep convolutional neural network for multi-modal physiological affect detection. Neural Comput Appl: 1–20

  21. Widrow B, Hoff ME (1960) Adaptive switching circuits. IRE WESCON Convention Record 4(1):96–104

    Google Scholar 

  22. Widrow B, McCool J, Ball M (1975) The complex LMS algorithm. Proc IEEE 63(4):719–720

    Article  Google Scholar 

  23. Wan EA, Van Der Merwe R (2001) The unscented Kalman filter. In: Kalman filtering and neural networks, pp 221–280

    Chapter  Google Scholar 

  24. Wan EA, Van Der Merwe R (2000) The unscented Kalman filter for non-linear estimation. In Proceedings of the IEEE 2000 Adaptive Systems for Signal Processing, Communications, and Control Symposium (Cat. No. 00EX373),p 153–158

  25. Tuncer T, Dogan S, Plawiak P, Subasi A (2022) A novel Discrete Wavelet-Concatenated Mesh Tree and ternary chess pattern based ECG signal recognition method. Biomed Signal Process Control 72:103331

    Article  Google Scholar 

  26. Valens C (1999) The fast lifting wavelet transform. In: The math forum, pp 10–12

    Google Scholar 

  27. Abualigah L, Diabat A, Mirjalili S, Abd Elaziz M, Gandomi AH (2021) The arithmetic optimization algorithm. Comput Methods Appl Mech Eng 376:113609

    Article  MathSciNet  Google Scholar 

  28. Schmidt P, Reiss A, Duerichen R, Marberger C, Van Laerhoven K (2018) Introducing wesad, a multi-modal dataset for wearable stress and affect detection. In Proceedings of the 20th ACM international conference on multi-modal interaction. p 400–408

  29. Liapis A, Faliagka E, Katsanos C, Antonopoulos C, Voros N (2021) Detection of Subtle Stress Episodes During UX Evaluation: Assessing the Performance of the WESAD Bio-Signals Dataset. InHuman-Computer Interaction–INTERACT 2021: 18th IFIP TC 13 International Conference, Bari, Italy, August 30–September 3, 2021, Proceedings, Part III 18. Springer International Publishing, p 238–247

  30. Sarkar P, Etemad A (2020) Self-supervised ECG representation learning for emotion recognition. IEEE Trans Affect Comput 13(3):1541–1554

    Article  Google Scholar 

  31. Bhatti A, Behinaein B, Rodenburg D, Hungler P, Etemad A (2021) Attentive cross-modal connections for deep multimodal wearable-based emotion recognition. In: 2021 9th international conference on affective computing and intelligent interaction workshops and demos (ACIIW). Nara, pp 1–5. https://doi.org/10.1109/ACIIW52867.2021.9666360

    Chapter  Google Scholar 

  32. Dissanayake V, Seneviratne S, Rana R, Wen E, Kaluarachchi T, Nanayakkara S (2022) Sigrep: Toward robust wearable emotion recognition with contrastive representation learning. IEEE Access 10:18105–18120

    Article  Google Scholar 

  33. Weinert HL (2007) Efficient computation for Whittaker-Henderson smoothing. Comput Stat Data Anal 52(2):959–974

    Article  MathSciNet  Google Scholar 

  34. Yamada H (2020) A note on Whittaker-Henderson graduation: Bisymmetry of the smoother matrix. Commun Stat-Theory Methods 49(7):1629–1634

    Article  MathSciNet  Google Scholar 

  35. Selesnick IW, Baraniuk RG, Kingsbury NC (2005) The dual-tree complex wavelet transform. IEEE Signal Process Mag 22(6):123–151

    Article  Google Scholar 

  36. Nigam K, Godani K, Sharma D, Jain S (2021) An improved approach for stress detection using physiological signals. EAI Endorsed Trans Scalable Inf Syst 8(33). https://doi.org/10.4108/eai.14-5-2021.169919

  37. Jimenez IAC, Acevedo JSG, Marcolin F, Vezzetti E, Moos S (2023) Towards an integrated framework to measure user engagement with interactive or physical products. Int J Interact Des Manuf (IJIDeM) 17(1):45–67

    Article  Google Scholar 

  38. Chatterjee D, Dutta S, Shaikh R, Saha SK (2022) A lightweight deep neural network for detection of mental states from physiological signals. Innov Syst Softw Eng:1–8

  39. Wen W, Liu G, Cheng N, Wei J, Shangguan P, Huang W (2014) Emotion recognition based on multi-variant correlation of physiological signals. IEEE Trans Affect Comput 5(2):126–140

    Article  Google Scholar 

  40. Cheng WX, Gao R, Suganthan PN, Yuen KF (2022) EEG-based emotion recognition using random Convolutional Neural Networks. Eng Appl Artif Intell 116:105349

    Article  Google Scholar 

  41. Aung ST, Hassan M, Brady M, Mannan ZI, Azam S, Karim A, Zaman S, Wongsawat Y (2022) Entropy-based emotion recognition from multichannel EEG signals using artificial neural network. In: Computational intelligence and neuroscience, vol 2022, p 6000989. https://doi.org/10.1155/2022/6000989

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of ECE, National Institute of Technology Patna, Bihar, 800005, India

    Anushka Pradhan & Subodh Srivastava

Authors
  1. Anushka Pradhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Subodh Srivastava
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors read and approved the final manuscript.

Corresponding author

Correspondence to Anushka Pradhan.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants or animals performed by authors.

Consent to participate

All the authors involved have agreed to participate in this submitted article.

Consent to Publish

All the authors involved in this manuscript give full consent to publish this submitted article.

Conflict of interest

Authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pradhan, A., Srivastava, S. Hybrid densenet with long short-term memory model for multi-modal emotion recognition from physiological signals. Multimed Tools Appl 83, 35221–35251 (2024). https://doi.org/10.1007/s11042-023-16933-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16933-2

Keywords

Search

Navigation