Abstract
Facial expressions are one of the most practical and straightforward ways to communicate emotions. Facial Expression Recognition has been used in lots of fields such as human behaviour understanding and health monitoring. Deep learning models can achieve excellent performance in facial expression recognition tasks. As these deep neural networks have very complex nonlinear structures, when the model makes a prediction, it is not easy for human users to understand what is the basis for the model’s prediction. Specifically, we do not know which facial units contribute to the classification more or less. Developing affective computing models with more explainable and transparent feedback for human interactors is essential for a trustworthy human-robot interaction. Compared to “white-box" approaches, “black-box” approaches using deep neural networks, which have advantages in terms of overall accuracy but lack reliability and explainability. In this work, we introduce a multimodal affective human-robot interaction framework, with visual-based and verbal-based explanation, by Layer-Wise Relevance Propagation (LRP) and Local Interpretable Mode-Agnostic Explanation (LIME). The proposed framework has been tested on the KDEF dataset, and in human-robot interaction experiments with the Pepper robot. This experimental evaluation shows the benefits of linking deep learning emotion recognition systems with explainable strategies.
Supported by University of Manchester and UKRI Node on Trust (EP/V026682/1).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access 6, 52138–52160 (2018)
Arrieta, A.B., et al.: Explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115 (2020)
Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.R., Samek, W.: On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS One 10(7), e0130140 (2015)
Baltrusaitis, T., Zadeh, A., Lim, Y.C., Morency, L.P.: Openface 2.0: facial behavior analysis toolkit. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 59–66. IEEE (2018)
Dubey, A.K., Jain, V.: Automatic facial recognition using vgg16 based transfer learning model. J. Inf. Optim. Sci. 41(7), 1589–1596 (2020)
Ekman, P., Friesen, W.V.: Facial action coding system. Environ. Psychol. Nonverbal Behav. (1978)
Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the human face: Guidelines for research and an integration of findings, vol. 11. Elsevier (2013)
Ivanovs, M., Kadikis, R., Ozols, K.: Perturbation-based methods for explaining deep neural networks: a survey. Pattern Recogn. Lett. 150, 228–234 (2021)
Kavila, S.D., Bandaru, R., Gali, T.V.M.B., Shafi, J.: Analysis of cardiovascular disease prediction using model-agnostic explainable artificial intelligence techniques. In: Principles and Methods of Explainable Artificial Intelligence in Healthcare, pp. 27–54. IGI Global (2022)
Lien, J.J., Kanade, T., Cohn, J.F., Li, C.C.: Automated facial expression recognition based on facs action units. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp. 390–395. IEEE (1998)
Lundqvist, D., Flykt, A., Öhman, A.: Karolinska directed emotional faces. Cogn. Emot. (1998)
Malik, S., Kumar, P., Raman, B.: Towards interpretable facial emotion recognition. In: Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–9 (2021)
Martinez, M., et al.: Emotion detection deficits and decreased empathy in patients with alzheimer’s disease and parkinson’s disease affect caregiver mood and burden. Front. Aging Neurosci. 10, 120 (2018)
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., Müller, K.R.: Layer-wise relevance propagation: an overview. In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 193–209 (2019)
Nie, W., Zhang, Y., Patel, A.: A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. In: International Conference on Machine Learning, pp. 3809–3818. PMLR (2018)
Rathod, J., Joshi, C., Khochare, J., Kazi, F.: Interpreting a black-box model used for scada attack detection in gas pipelines control system. In: 2020 IEEE 17th India Council International Conference (INDICON), pp. 1–7. IEEE (2020)
Ribeiro, M.T., Singh, S., Guestrin, C.: “why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Robnik-Šikonja, M., Bohanec, M.: Perturbation-based explanations of prediction models. In: Zhou, J., Chen, F. (eds.) Human and Machine Learning. HIS, pp. 159–175. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-90403-0_9
Rosenberg, E.L., Ekman, P.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (2020)
Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.): Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6
Taheri, A., Meghdari, A., Alemi, M., Pouretemad, H.: Human-robot interaction in autism treatment: a case study on three pairs of autistic children as twins, siblings, and classmates. Int. J. Social Rob. 10(1), 93–113 (2018)
Tian, Y.I., Kanade, T., Cohn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach, Intell. 23(2), 97–115 (2001)
Yao, L., Wan, Y., Ni, H., Xu, B.: Action unit classification for facial expression recognition using active learning and svm. Multimedia Tools Appl. 80(16), 24287–24301 (2021)
Yin, P., Huang, L., Lee, S., Qiao, M., Asthana, S., Nakamura, T.: Diagnosis of neural network via backward deduction. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 260–267. IEEE (2019)
Yu, C.: Robot Behavior Generation and Human Behavior Understanding in Natural Human-Robot Interaction. Ph.D. thesis, Institut polytechnique de Paris (2021)
Yu, C., Tapus, A.: Interactive robot learning for multimodal emotion recognition. In: Salichs, M.A., et al. (eds.) ICSR 2019. LNCS (LNAI), vol. 11876, pp. 633–642. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-35888-4_59
Yu, C., Tapus, A.: Multimodal emotion recognition with thermal and rgb-d cameras for human-robot interaction. In: Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, pp. 532–534 (2020)
Zhang, H., Yu, C., Tapus, A.: Why do you think this joke told by robot is funny? the humor style matters. In: 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), pp. 572–577. IEEE (2022)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhu, H., Yu, C., Cangelosi, A. (2022). Affective Human-Robot Interaction with Multimodal Explanations. In: Cavallo, F., et al. Social Robotics. ICSR 2022. Lecture Notes in Computer Science(), vol 13817. Springer, Cham. https://doi.org/10.1007/978-3-031-24667-8_22
Download citation
DOI: https://doi.org/10.1007/978-3-031-24667-8_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-24666-1
Online ISBN: 978-3-031-24667-8
eBook Packages: Computer ScienceComputer Science (R0)