Skip to main content
SpringerLink
Log in

Multimodal Emotion Recognition and Intention Understanding in Human-Robot Interaction

  • Chapter
  • First Online:
Developments in Advanced Control and Intelligent Automation for Complex Systems

Part of the book series: Studies in Systems, Decision and Control ((SSDC,volume 329))

Abstract

Emotion recognition and intention understanding are important components of human-robot interaction. In multimodal emotion recognition and intent understanding, feature extraction and selection of recognition methods are related to the calculation of affective computing and the diversity of human-robot interaction. Therefore, by studying multimodal emotion recognition and intention understanding we to create an emotional and human-friendly human-robot interaction environment. This chapter introduces the characteristics of multimodal emotion recognition and intention understanding, presents different modalities emotion feature extraction methods and emotion recognition methods, proposes the intention understanding method, and finally applies them in practice to achieve human-robot interaction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

PCA:

Principal component analysis

LDA:

Linear discriminant analysis

SIFT:

Scale-invariant feature transform

SURF:

Speed-up robost features

ELM:

Extreme learning machine

BPNN:

Back propagation neural networks

CNN:

Convolution Neural Network

DBN:

Deep Belief Network

RNN:

Recurrent Neural Network

SVM:

Support vector machine

TLWFSVR:

Three-layer weighted fuzzy support vector regression

FSVR:

Fuzzy support vector regressions

ROI:

Regions of Interest

SC:

Sparse coding

FFT:

Fast Fourier Transformation

SR:

Softmax Regression

DSAN:

Deep Sparse Autoencoder Network

BPA:

Basic Probability Assignment

RF:

Random Forest

FCM:

Fuzzy c-means

References

  1. Mehrabian, A.: Communication without words. Psychol. Today 2(4), 53–56 (1968)

    Google Scholar 

  2. Schuller, B., Batliner, A., Steidl, S., Seppi, D.: Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge. Speech Commun. 53(9), 1062–1087 (2011)

    Article  Google Scholar 

  3. Chen, L., Zheng, S.K.: Speech emotion recognition: Features and classification models. Digital Signal Process. 22(6), 1154–1160 (2012)

    Article  MathSciNet  Google Scholar 

  4. Chen, L.F., Zhou, M.T., Su, W.J., Wu, M., She, J.H., Hirota, K.: Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Inf. Sci. 428, 49–61 (2018)

    Article  MathSciNet  Google Scholar 

  5. Chen, L.F., Feng, Y., Maram, M.A., Wang, Y.W., Wu, M., Hirota, K., Pedrycz, W.: Multi-SVM based dempster-shafer theory for gesture intention understanding using sparse coding feature. Appl. Soft Comput. (2019). https://doi.org/10.1016/j.asoc.2019.105787

    Article  Google Scholar 

  6. Chen, L.F., Su, W.J., Feng, Y., Wu, M., She, J.H., Hirota, K.: Two-layer fuzzy multiple random forest for speech emotion recognition in human-robot interaction. Inf. Sci. 509, 150–163 (2020)

    Article  Google Scholar 

  7. Kim, Y., Provost, E.M.: ISLA: temporal segmentation and labeling for audio-visual emotion recognition. IEEE Trans. Affect. Comput. 2702653, 2017 (2017). https://doi.org/10.1109/TAFFC

    Article  Google Scholar 

  8. Barros, P., Jirak, D., Weber, C.: Multimodal emotional state recognition using sequence-dependent deep hierarchical features. Neural Netw. 72, 140–151 (2015)

    Article  Google Scholar 

  9. Soleymani, M., Asghariesfeden, S., Fu, Y.: Analysis of EEG signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7(1), 17–28 (2016)

    Article  Google Scholar 

  10. Wegner, D.M.: The Illusion of Conscious Will. MIT Press, Cambridge (2002)

    Book  Google Scholar 

  11. Emotion-Age-Gender-Nationality based intention understanding in human-robot interaction using two-layer fuzzy support vector regression. Int. J. Soc. Robot. 7(5), 709–729 (2015)

    Google Scholar 

  12. Chen, L.F., Wu, M., Zhou, M.T.: Dynamic emotion understanding in human-robot interaction based on two-layer fuzzy SVR-TS model. IEEE Trans. Syst. Man Cybern. Syst. 50(2), 490–501 (2017)

    Article  Google Scholar 

  13. Chen, L.F., Zhou, M.T., Wu, M.: Three-layer weighted fuzzy support vector regression for emotional intention understanding in human-robot interaction. IEEE Trans. Fuzzy Syst. 26(5), 2524–2538 (2018)

    Article  Google Scholar 

  14. Lu, X., Bao, W., Wang, S.: Three-dimensional interfacial stress decoupling method for rehabilitation therapy robot. IEEE Trans. Ind. Electron. 64(5), 3970–3977 (2017)

    Article  Google Scholar 

  15. Turner, J., Meng, Q., Schaefer, G.: Distributed task rescheduling with time constraints for the optimization of total task allocations in a multi-robot system. IEEE Trans. Cybern. 48(9), 2583–2597 (2018)

    Article  Google Scholar 

  16. Lui, J.H., Samani, H., Tien, K.Y.: An affective mood booster robot based on emotional processing unit. In: Proceedings of 2017 International Automatic Control Conference, Pingtung, China, pp. 1–6 (2018)

    Google Scholar 

  17. Somchanok, T., Michiko, O.: Emotion recognition using ECG signals with local pattern description methods. Int. J. Affect. Eng. 15(2), 51–61 (2016)

    Article  Google Scholar 

  18. Klug, M., Zell, A.: Emotion-based human-robot interaction. In: Proceedings of the 9th IEEE International Conference on Computational Cybernetics, Tihany, Hungary, pp. 365–368 (2013)

    Google Scholar 

  19. Boccanfuso, L., Barney, E., Foster, C.: Emotional robot to examine differences in play patterns and affective response of children with and without ASD. In: Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction, Christchurch, New Zealand, pp. 19–26 (2016)

    Google Scholar 

  20. Phaisangittisagul, E., Thainimit, S., Chen, W.: Predictive high-level feature representation based on dictionary learning. Expert Syst. Appl. 69(2017), 101–109 (2017)

    Article  Google Scholar 

  21. Li, B., Zhao, F., Su, Z.: Example-based image colorization using locality consistent sparse representation. IEEE Trans. Image Process 26(11), 5188–5202 (2017)

    Article  MathSciNet  Google Scholar 

  22. Stefania, B., Alfonso, C., Peelen, M.V.: View-invariant representation of hand postures in the human lateral occipitotemporal cortex. NeuroImage 181(2018), 446–452 (2018)

    Google Scholar 

  23. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: SURF: speeded up robust features. In: Proceedings of Computer Vision and Image Understanding, pp. 346–359 (2008)

    Google Scholar 

  24. Zhu, X., Li, X., Zhang, S.: Robust joint graph sparse coding for unsupervised spectral feature selection. IEEE Trans. Neural Netw. Learn. Syst. 28(6), 1263–1275 (2017)

    Article  MathSciNet  Google Scholar 

  25. Eyben, F., Wollmer, M., Graves, A.: Online emotion recognition in a 3-D activation-valence-time continuum using acoustic and linguistic cues. J. Multimodal User Interfaces 3, 7–19 (2010)

    Article  Google Scholar 

  26. Chen, C., Jafari, R., Kehtarnavaz, N.: Improving human action recognition using fusion of depth camera and inertial sensors. IEEE Trans. Human-Mach. Syst. 45(1), 51–61 (2015)

    Article  Google Scholar 

  27. Wang, J.N., Xiong, R., Chu, J.: Facial feature points detecting based on Gaussian mixture models. Pattern Recognit. Lett. 53, 62–68 (2015)

    Article  Google Scholar 

  28. Asteriadis, S., Nikolaidis, N., Pitas, I.: Facial feature detection using distance vector fields. Pattern Recognit. 42(7), 1388–1398 (2009)

    Article  Google Scholar 

  29. Chen, L.F., Liu, Z.T., Wu, M., Ding, M., Dong, F.Y., Hirota, K.: Emotion-age-gender-nationality based intention understanding in human-robot interaction using two-layer fuzzy support vector regression. Int. J. Soc. Robot. 7(5), 709–729 (2015)

    Article  Google Scholar 

  30. Lin, K.P.: A novel evolutionary kernel intuitionistic fuzzy c-means clustering algorithm. IEEE Trans. Fuzzy Syst. 22(5), 1074–1087 (2014)

    Article  Google Scholar 

  31. Nguyen, D.D., Ngo, L.T.: Multiple kernel interval type-2 fuzzy c-means clustering. In: IEEE International Conference on Fuzzy Systems, Hyderabad, India, pp. 1–8 (2013)

    Google Scholar 

  32. Dong, W.M., Wong, F.S.: Fuzzy weighted averages and implementation of the extension principle. Fuzzy Sets Syst. 21(2), 183–199 (1987)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Automation, China University of Geosciences, Wuhan, 430074, China

    Luefeng Chen, Zhentao Liu & Min Wu

  2. Hubei Key Laboratory of Advanced Control and Intelligent Automation for Complex Systems, Wuhan, 430074, China

    Luefeng Chen, Zhentao Liu & Min Wu

  3. Tokyo Institute of Technology, Yokohama, 226-8502, Japan

    Kaoru Hirota

  4. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, TR6 2G7, Canada

    Witold Pedrycz

Authors
  1. Luefeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhentao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kaoru Hirota
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Witold Pedrycz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luefeng Chen .

Editor information

Editors and Affiliations

  1. School of Automation, China University of Geosciences, Wuhan, China

    Min Wu

  2. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  3. School of Automation, China University of Geosciences, Wuhan, China

    Luefeng Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Chen, L., Liu, Z., Wu, M., Hirota, K., Pedrycz, W. (2021). Multimodal Emotion Recognition and Intention Understanding in Human-Robot Interaction. In: Wu, M., Pedrycz, W., Chen, L. (eds) Developments in Advanced Control and Intelligent Automation for Complex Systems. Studies in Systems, Decision and Control, vol 329. Springer, Cham. https://doi.org/10.1007/978-3-030-62147-6_10

Download citation

Publish with us

Policies and ethics

Search

Navigation