Skip to main content
SpringerLink
Log in

Bi-Model Emotional AI for Audio-Visual Human Emotion Detection Using Hybrid Deep Learning Model

  • Chapter
  • First Online:
Artificial Intelligence for Sustainable Development

Part of the book series: EAI/Springer Innovations in Communication and Computing ((EAISICC))

  • 66 Accesses

Abstract

The present attention of computer vision study is on AI emotion identification, which comprises the automatic acknowledgment of facial terminologies of feeling and the evaluation of sentiment in visual database. In order for artificially intelligent systems with visual capabilities to comprehend human interactions, the study of human–machine interaction is essential. Artificial emotional intelligence, sometimes referred to as affective computing and emotional AI, is a subfield of artificial intelligence that concentrates on the comprehension, examination, and replication of human emotions. Its goal is to advance the sincerity and organic nature of interactions between people and robots. Textual content, voice tone, facial expressions, and gestures are just a few of the cues that emotional AI uses to understand people’s emotions and alter its answers accordingly. Using computer vision technology, Visual Emotion AI analyzes facial expressions in photos and videos to determine a person’s emotional state. This study uses both audio and visual inputs to investigate the recognition of emotions using artificial intelligence.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. J. Deng and F. Ren, "Multi-Label Emotion Detection via Emotion-Specified Feature Extraction and Emotion Correlation Learning," in IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 475–486, 1 Jan.-March 2023, doi: https://doi.org/10.1109/TAFFC.2020.3034215.

  2. H. Kim, J. Ben-othman, L. Mokdad and P. Bellavista, "A Virtual Emotion Detection Architecture With Two-Way Enabled Delay Bound toward Evolutional Emotion-Based IoT Services," in IEEE Transactions on Mobile Computing, vol. 21, no. 4, pp. 1172–1181, 1 April 2022, doi: https://doi.org/10.1109/TMC.2020.3024059.

  3. Y. Yu et al., "Cloud-Edge Collaborative Depression Detection Using Negative Emotion Recognition and Cross-Scale Facial Feature Analysis," in IEEE Transactions on Industrial Informatics, vol. 19, no. 3, pp. 3088–3098, March 2023, doi: https://doi.org/10.1109/TII.2022.3163512.

  4. F. Anzum and M. L. Gavrilova, "Emotion Detection From Micro-Blogs Using Novel Input Representation," in IEEE Access, vol. 11, pp. 19512–19522, 2023, doi: https://doi.org/10.1109/ACCESS.2023.3248506.

  5. C. Suman, R. Chaudhari, S. Saha, S. Kumar and P. Bhattacharyya, "Investigations in Emotion Aware Multimodal Gender Prediction Systems From Social Media Data," in IEEE Transactions on Computational Social Systems, vol. 10, no. 2, pp. 470-479, April 2023, doi: https://doi.org/10.1109/TCSS.2022.3158605.

  6. Z. Xu and S. Wang, "Emotional Attention Detection and Correlation Exploration for Image Emotion Distribution Learning," in IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 357-369, 1 Jan.-March 2023, doi: https://doi.org/10.1109/TAFFC.2021.3071131.

  7. N. Aslam, F. Rustam, E. Lee, P. B. Washington and I. Ashraf, "Sentiment Analysis and Emotion Detection on Cryptocurrency Related Tweets Using Ensemble LSTM-GRU Model," in IEEE Access, vol. 10, pp. 39313–39324, 2022, doi: https://doi.org/10.1109/ACCESS.2022.3165621.

  8. F. Alrowais et al., "Modified Earthworm Optimization With Deep Learning Assisted Emotion Recognition for Human Computer Interface," in IEEE Access, vol. 11, pp. 35089-35096, 2023, doi: https://doi.org/10.1109/ACCESS.2023.3264260.

  9. E. A. Veltmeijer, C. Gerritsen and K. V. Hindriks, "Automatic Emotion Recognition for Groups: A Review," in IEEE Transactions on Affective Computing, vol. 14, no. 1, pp. 89–107, 1 Jan.–March 2023, doi: https://doi.org/10.1109/TAFFC.2021.3065726.

  10. S. Lee, S. Lee and H. Kim, "Differential Security Barriers for Virtual Emotion Detection in Maritime Transportation Stations With Cooperative Mobile Robots and UAVs," in IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 2, pp. 2461–2471, Feb. 2023, doi: https://doi.org/10.1109/TITS.2022.3172668.

  11. L. Chen, M. Zhou, W. Su, M. Wu, J. She, and K. Hirota, “Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction,” Information Sciences, vol. 428, pp. 49–61, 2018.

    Article  MathSciNet  Google Scholar 

  12. A. Hassouneh, A. M. Mutawa, and M. Murugappan, “Development of a real-time emotion recognition system using facial expressions and EEG based on machine learning and deep neural network methods,” Informatics in Medicine Unlocked, vol. 20, Article ID 100372, 2020.

    Google Scholar 

  13. O. Ekundayo and S. Viriri, “Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation,” PeerJ Computer Science, vol. 7, p. e736, 2021.

    Google Scholar 

  14. K. Jayanthi and S. Mohan, “An integrated framework for emotion recognition using speech and static images with deep classifier fusion approach,” International Journal of Information Technology, pp. 1–11, 2022.

    Google Scholar 

  15. P. Babajee, G. Suddul, S. Armoogum, and R. Foogooa, “Identifying human emotions from facial expressions with deep learning,” in Proceedings of the 2020 Zooming Innovation in Consumer Technologies Conference (ZINC), pp. 36–39, IEEE, Novi Sad, Serbia, May 2020.

    Google Scholar 

  16. C. Tan, M. Šarlija, and N. Kasabov, “NeuroSense: short-term emotion recognition and understanding based on spiking neural network modelling of spatio-temporal EEG patterns,” Neurocomputing, vol. 434, pp. 137–148, 2021.

    Article  Google Scholar 

  17. M. S. Hossain and G. Muhammad, “An Audio-Visual Emotion Recognition System Using Deep Learning Fusion for a Cognitive Wireless Framework,” IEEE Wireless Communications, vol. 26, no. 3, pp. 62–68, Jun. 2019, doi: https://doi.org/10.1109/mwc.2019.1800419.

  18. Yining Chen, “The Past, Present, and Future of Enterprise Resource Planning”, Journal of Enterprise and Business Intelligence, vol.3, no.2, pp. 065–074, April 2023. doi: https://doi.org/10.53759/5181/JEBI202303007.

  19. B. Jackson, S. R, B. Balachander, A. Haldorai, V. S., and A. N. A. Sahaya, “Modified Design Structure Of A Metamaterial Microstrip Patch Array Antenna For Rf Energy Optimization,” Materiali in tehnologije, vol. 56, no. 2, Apr. 2022, doi: 10.17222/mit.2022.384

    Google Scholar 

  20. Sim Sze Yin and Yoni Danieli, “An Evaluation of Smart Livestock Feeding Strategies”, Journal of Robotics Spectrum, vol.1, pp. 066–077, 2023. doi: https://doi.org/10.53759/9852/JRS202301007.

  21. A. Kumar, K. Abhishek, X. Liu, and A. Haldorai, “An Efficient Privacy-Preserving ID Centric Authentication in IoT Based Cloud Servers for Sustainable Smart Cities,” Wireless Personal Communications, vol. 117, no. 4, pp. 3229–3253, Nov. 2020, doi: https://doi.org/10.1007/s11277-020-07979-8.

  22. Kazuo Adachi and Ryo Sato, “Customer Relationship Management in an Era of Digital Disruption and Business Model Evolution”, Journal of Enterprise and Business Intelligence, vol.3, no.1, pp. 001–011, January 2023. doi: https://doi.org/10.53759/5181/JEBI202303001.

  23. S. Ntalampiras, “Speech emotion recognition via learning analogies,” Pattern Recognition Letters, vol. 144, pp. 21–26, Apr. 2021, doi: https://doi.org/10.1016/j.patrec.2021.01.018.

  24. M. Bejani, D. Gharavian, and N. M. Charkari, “Audiovisual EmotionRecognition using ANOVA Feature Selection Method and Multi-ClassifierNeural Networks,” Neural Computing and Applications, vol. 24, no. 2,pp. 399–412, 2014.

    Article  Google Scholar 

  25. N. E. D. Elmadany, Y. He, and L. Guan, “Multiview Learning via DeepDiscriminative Canonical Correlation Analysis,” in IEEE InternationalConference on Acoustics, Speech and Signal Processing. IEEE, 2016,pp. 2409–2413.

    Google Scholar 

  26. S. Zhang, S. Zhang, T. Huang, and W. Gao, “Multimodal DeepConvolutional Neural Network for Audio-Visual Emotion Recognition,“in ACM on International Conference on Multimedia Retrieval. ACM,2016, pp. 281–284.

    Google Scholar 

  27. W. Mellouk and W. Handouzi, “Facial emotion recognition using deep learning: review and insights,” Procedia Computer Science, vol. 175, pp. 689–694, 2020, doi: https://doi.org/10.1016/j.procs.2020.07.101.

    Article  Google Scholar 

  28. R. Gross, S. Baker, I. Matthews, and T. Kanade, “Face Recognition Across Pose and Illumination,” Handbook of Face Recognition, pp. 193–216, doi: 10.1007/0-387-27257-7_10.

    Google Scholar 

  29. A. Dhall, R. Goecke, S. Lucey, and T. Gedeon, “Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark,” 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Nov. 2011, doi: https://doi.org/10.1109/iccvw.2011.6130508.

  30. P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, and I. Matthews, “The Extended Cohn-Kanade Dataset (CK+): A complete dataset for action unit and emotion-specified expression,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition – Workshops, Jun. 2010, doi: https://doi.org/10.1109/cvprw.2010.5543262

  31. M. F. Valstar, B. Jiang, M. Mehu, M. Pantic, and K. Scherer, “The first facial expression recognition and analysis challenge,” Face and Gesture 2011, Mar. 2011, doi: https://doi.org/10.1109/fg.2011.5771374.

  32. I. J. Goodfellow et al., “Challenges in representation learning: A report on three machine learning contests,” Neural Networks, vol. 64, pp. 59–63, Apr. 2015, doi: https://doi.org/10.1016/j.neunet.2014.09.005.

  33. Lijun Yin, Xiaozhou Wei, Yi Sun, Jun Wang, and M. J. Rosato, “A 3D Facial Expression Database For Facial Behavior Research,” 7th International Conference on Automatic Face and Gesture Recognition (FGR06), doi: 10.1109/fgr.2006.6.

    Google Scholar 

  34. W.-J. Yan et al., “CASME II: An Improved Spontaneous Micro-Expression Database and the Baseline Evaluation,” PLoS ONE, vol. 9, no. 1, p. e86041, Jan. 2014, doi: https://doi.org/10.1371/journal.pone.0086041.

  35. G. Zhao, X. Huang, M. Taini, S. Z. Li, and M. Pietikäinen, “Facial expression recognition from near-infrared videos,” Image and Vision Computing, vol. 29, no. 9, pp. 607–619, Aug. 2011, doi: https://doi.org/10.1016/j.imavis.2011.07.002.

  36. A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, Jan. 2019, doi: https://doi.org/10.1109/taffc.2017.2740923.

  37. S. Li, W. Deng, and J. Du, “Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild,” 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 2017, doi: https://doi.org/10.1109/cvpr.2017.277.

  38. O. Langner, R. Dotsch, G. Bijlstra, D. H. J. Wigboldus, S. T. Hawk, and A. van Knippenberg, “Presentation and validation of the Radboud Faces Database,” Cognition & Emotion, vol. 24, no. 8, pp. 1377–1388, Dec. 2010, doi: https://doi.org/10.1080/02699930903485076.

  39. A. V. Savchenko, L. V. Savchenko and I. Makarov, "Classifying Emotions and Engagement in Online Learning Based on a Single Facial Expression Recognition Neural Network," in IEEE Transactions on Affective Computing, vol. 13, no. 4, pp. 2132–2143, 1 Oct.-Dec. 2022, doi: https://doi.org/10.1109/TAFFC.2022.3188390.

  40. T. Dar, A. Javed, S. Bourouis, H. S. Hussein and H. Alshazly, "Efficient-SwishNet Based System for Facial Emotion Recognition," in IEEE Access, vol. 10, pp. 71311–71328, 2022, doi: https://doi.org/10.1109/ACCESS.2022.3188730.

  41. F. Huang, X. Li, C. Yuan, S. Zhang, J. Zhang and S. Qiao, "Attention-Emotion-Enhanced Convolutional LSTM for Sentiment Analysis," in IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 9, pp. 4332–4345, Sept. 2022, doi: https://doi.org/10.1109/TNNLS.2021.3056664.

  42. B. García-Martínez, A. Fernández-Caballero, R. Alcaraz and A. Martínez-Rodrigo, "Application of Dispersion Entropy for the Detection of Emotions With Electroencephalographic Signals," in IEEE Transactions on Cognitive and Developmental Systems, vol. 14, no. 3, pp. 1179–1187, Sept. 2022, doi: https://doi.org/10.1109/TCDS.2021.3099344.

  43. S. Kakuba, A. Poulose and D. S. Han, "Deep Learning-Based Speech Emotion Recognition Using Multi-Level Fusion of Concurrent Features," in IEEE Access, vol. 10, pp. 125538–125551, 2022, doi: https://doi.org/10.1109/ACCESS.2022.3225684.

Download references

Author information

Authors and Affiliations

  1. Centre for Future Networks and Digital Twin, Sri Eshwar College of Engineering, Coimbatore, Tamil Nadu, India

    Anandakumar Haldorai

  2. Sri Eshwar College of Engineering, Coimbatore, Tamil Nadu, India

    Babitha Lincy R, Suriya Murugan & Minu Balakrishnan

Authors
  1. Anandakumar Haldorai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Babitha Lincy R
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suriya Murugan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minu Balakrishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2024 European Alliance for Innovation

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Haldorai, A., R, B.L., Murugan, S., Balakrishnan, M. (2024). Bi-Model Emotional AI for Audio-Visual Human Emotion Detection Using Hybrid Deep Learning Model. In: Artificial Intelligence for Sustainable Development. EAI/Springer Innovations in Communication and Computing. Springer, Cham. https://doi.org/10.1007/978-3-031-53972-5_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-53972-5_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-53971-8

  • Online ISBN: 978-3-031-53972-5

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation