Affective Conversational Interfaces



In order to build artificial conversational interfaces that display behaviors that are credible and expressive, we should endow them with the capability to recognize, adapt to, and render emotion. In this chapter, we explain how the recognition of emotional aspects is managed within conversational interfaces, including modeling and representation, emotion recognition from physiological signals, acoustics, text, facial expressions, and gestures and how emotion synthesis is managed through expressive speech and multimodal embodied agents. We also cover the main open tools and databases available for developers wishing to incorporate emotion into their conversational interfaces.


EmotionML Emotion recognition Physiological signals Paralinguistic features Facial expressions Gestures Emotion synthesis 


  1. Allwood J, Cerrato L, Jokinen K, Naravetta C, Paggio P (2008) The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Lang Resour Eval 41(3/4):273–287. doi: 10.1007/s10579-007-9061-5 Google Scholar
  2. Baccianella S, Esuli A, Sebastiani F (2010) SentiWordNet 3.0: an enhanced lexical resource for sentiment analysis and opinion mining. In: International conference on language resources and evaluation (LREC2010). European Language Resources Association (ELRA), Malta, 17–23 May 2010, pp 2200–2204Google Scholar
  3. Batliner A, Schuller B, Seppi D, Steidl S, Devilliers L, Vidrascu L, Vogt T, Aharonson V, Amir N (2011) The automatic recognition of emotions in speech. In: Cowie R, Pelachaud C, Petta P (eds) Emotion-oriented systems. Springer Berlin Heidelberg, Berlin, Heidelberg, pp 71–99. doi: 10.1007/978-3-642-15184-2_6
  4. Ben Mimoun MS, Poncin I, Garnier M (2012) Case study—embodied virtual agents: an analysis on reasons for failure. J Retail Consum Serv 19(6):605–612. doi: 10.1016/j.jretconser.2012.07.006 CrossRefGoogle Scholar
  5. Boersma P, Weenink D (2016) Praat: doing phonetics by computer.
  6. Bos DO (2006) EEG-based emotion recognition; the influence of visual and auditory stimuli.
  7. Burkhardt F (2005) Emofilt: the simulation of emotional speech by prosody-transformation. In: Proceedings of the 9th European conference on speech communication and technology (Interspeech2005), Lisbon, Portugal, 4–8 Sept 2005, pp 509–512.
  8. Cafaro A, Vilhjálmsson HH, Bickmore T, Heylen D, Pelachaud C(2014) Representing communicative functions in SAIBA with a unified function markup language. In: Bickmore T, Marsella S, Sidner C (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 81–94. doi: 10.1007/978-3-319-09767-1_11
  9. Callejas Z, Griol D, López-Cózar R (2011) Predicting user mental states in spoken dialogue systems. EURASIP J Adv Signal Process 1:6. doi: 10.1186/1687-6180-2011-6 CrossRefGoogle Scholar
  10. Calvo RA, D’Mello S, Gratch J, Kappas A (eds) (2014) The Oxford handbook of affective computing, 1st edn. Oxford University Press, Oxford. doi: 10.1093/oxfordhb/9780199942237.001.0001 Google Scholar
  11. Castellano G, Leite I, Pereira A, Martinho C, Paiva A, McOwan PW (2010) Affect recognition for interactive companions: challenges and design in real world scenarios. J Multimodal User Interfaces 3(1–2):89–98. doi: 10.1007/s12193-009-0033-5 CrossRefGoogle Scholar
  12. Clavel C, Callejas Z (2016) Sentiment analysis: from opinion mining to human-agent interaction. IEEE Trans Affect Comput 7(1):74–93. doi: 10.1109/TAFFC.2015.2444846
  13. Cowie R, Cornelius R (2003) Describing the emotional states that are expressed in speech. Speech Commun 40(1–2):5–32. doi: 10.1016/S0167-6393(02)00071-7 CrossRefMATHGoogle Scholar
  14. Ekman P (1999) Basic emotions. In: Dalgleish T, Power MJ (eds) Handbook of cognition and emotion. Wiley, Chichester, pp 45–60. doi: 10.1002/0470013494.ch3
  15. Ekman P (2003) Emotions revealed: recognizing faces and feelings to improve communication and emotional life, 1st edn. Times Books, New YorkGoogle Scholar
  16. Ekman P, Rosenberg EL (eds) (2005) What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS), 2nd edn. Oxford University Press, Oxford. doi: 10.1093/acprof:oso/9780195179644.001.0001
  17. Eyben F, Weninger F, Gross F, Schuller B (2013) Recent developments in openSMILE, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM international conference on multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 835–838. doi: 10.1145/2502081.2502224
  18. Feldman R (2013) Techniques and applications for sentiment analysis. Commun ACM 56(4):82. doi: 10.1145/2436256.2436274 CrossRefGoogle Scholar
  19. Govind D, Prasanna SRM (2012) Expressive speech synthesis: a review. IJST 16(2):237–260. doi: 10.1007/s10772-012-9180-2 Google Scholar
  20. Gratch J, Marsella S (eds) (2013) Social emotions in nature and artifact. Oxford University Press, Oxford  10.1093/acprof:oso/9780195387643.001.0001
  21. Haag A, Goronzy S, Schaich P, Williams J (2004) Emotion recognition using bio-sensors: first steps towards an automatic system. In: André E, Dybkjær L, Minker W, Heisterkamp P (eds) Affective dialogue systems. Springer Berlin Heidelberg, New York, pp 36–48. doi: 10.1007/978-3-540-24842-2_4
  22. Jang E-H, Park B-J, Kim S-H, Chung M-A, Park M-S, Sohn J-H (2014) Emotion classification based on bio-signals emotion recognition using machine learning algorithms. In: Proceedings of 2014 international conference on information science, Electronics and Electrical Engineering (ISEEE), Sapporo, Japan, 26–28 April 2014, pp 104–109. doi: 10.1109/InfoSEEE.2014.6946144
  23. Jerritta S, Murugappan M, Nagarajan R, Wan K (2011) Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th international colloquium on signal processing and its applications (CSPA), Penang, Malaysia, 4–6 March 2011, pp 410–415. doi: 10.1109/CSPA.2011.5759912
  24. Kätsyri J, Förger K, Mäkäräinen M, Takala T (2015) A review of empirical evidence on different uncanny valley hypotheses: support for perceptual mismatch as one road to the valley of eeriness. Front Psychol 6:390. doi: 10.3389/fpsyg.2015.00390 CrossRefGoogle Scholar
  25. Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal 30(12):2067–2083. doi: 10.1109/TPAMI.2008.26 CrossRefGoogle Scholar
  26. Kim KH, Bang SW, Kim SR (2004) Emotion recognition system using short-term monitoring of physiological signals. Med Biol Eng Comput 42(3):419–427. doi: 10.1007/BF02344719 CrossRefGoogle Scholar
  27. Kipp M (2012) ANVIL: a universal video research tool. In: Durand J, Gut U, Kristofferson G (eds) Handbook of corpus phonology. Oxford University Press, Oxford. doi: 10.1093/oxfordhb/9780199571932.013.024
  28. Kopp S, Krenn B, Marsella S, Marshall AN, Pelachaud C, Pirker H, Thórisson KR, Vilhjálmsson H (2006) Towards a common framework for multimodal generation: the behavior markup language. In: Gratch J, Young M, Aylett R, Ballin D, Olivier P (eds) Intelligent virtual agents. Springer International Publishing, Switzerland, pp 205–217. doi: 10.1007/11821830_17
  29. Liu B (2015) Sentiment analysis: mining opinions, sentiments, and emotions. Cambridge University Press, New York. doi: 10.1017/CBO9781139084789
  30. Marsella SC, Gratch J (2009) EMA: a process model of appraisal dynamics. Cogn Syst Res 10(1):70–90. doi: 10.1016/j.cogsys.2008.03.005 CrossRefGoogle Scholar
  31. Mathur MB, Reichling DB (2016) Navigating a social world with robot partners: a quantitative cartography of the Uncanny Valley. Cognition 146:22–32. doi: 10.1016/j.cognition.2015.09.008 CrossRefGoogle Scholar
  32. Nasoz F, Alvarez K, Lisetti CL, Finkelstein N (2003) Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Technol Work 6(1):4–14. doi: 10.1007/s10111-003-0143-x CrossRefGoogle Scholar
  33. Nass C, Lee KM (2000) Does computer-generated speech manifest personality? An experimental test of similarity-attraction. In: Proceedings of the SIGCHI conference on human factors in computing systems (CHI’00), The Hague, Netherlands, 1–6 April 2000, pp 329–336. doi: 10.1145/332040.332452
  34. Nass C, Yen C (2012) The man who lied to his laptop: what we can learn about ourselves from our machines. Penguin Group, New YorkGoogle Scholar
  35. Niewiadomski R, Hyniewska SJ, Pelachaud C (2013) Computational models of expressive behaviors for a virtual agent. In: Gratch J, Marsella S (eds) Social emotions in nature and artifact. Oxford University Press, Oxford, pp 143–161. doi: 10.1093/acprof:oso/9780195387643.003.0010
  36. Paggio P, Navarretta C (2011) Head movements, facial expressions and feedback in danish first encounters interactions: a culture-specific analysis. In: Stephanidis C (ed) Universal access in human-computer interaction users diversity. Springer Berlin Heidelberg, New York, pp 583–590. doi: 10.1007/978-3-642-21663-3_63
  37. Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Delac K, Grgic M (eds) Face recognition. I-Tech Education and Publishing, Vienna, Austria, pp 377–416. doi: 10.5772/4847
  38. Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE International conference on multimedia and expo (ICME), Amsterdam, The Netherlands, 6–8 July 2005, pp 317–321. doi: 10.1109/ICME.2005.1521424
  39. Pelachaud C (2009) Modelling multimodal expression of emotion in a virtual agent. Philos Trans R Soc B Biol Sci 364(1535):3539–3548. doi: 10.1098/rstb.2009.0186 CrossRefGoogle Scholar
  40. Petta P, Pelachaud C, Cowie R (eds) (2011) Emotion-oriented systems: the Humaine handbook. Springer, Berlin Heidelberg. doi: 10.1007/978-3-642-15184-2
  41. Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal 23(10):1175–1191. doi: 10.1109/34.954607 CrossRefGoogle Scholar
  42. Polzin TS, Waibel A (2000) Emotion-sensitive human-computer interfaces. In: International speech communication association (ISCA) tutorial and research workshop on speech and emotion. Newcastle, Northern Ireland, UK, pp 201–206Google Scholar
  43. Ryan A, Cohn JF, Lucey S, Saragih J, Lucey P, De La Torre F, Rossi A (2009) Automated facial expression recognition system. In: 43rd annual international Carnahan conference on security technology, Zurich, Switzerland, 5–8 Oct 2009, pp 172–177. doi: 10.1109/CCST.2009.5335546
  44. Schröder M (2009) Expressive speech synthesis: past, present, and possible futures. In: Tao J, Tan T (eds) Affective information processing. Springer, London, pp 111–126. doi: 10.1007/978-1-84800-306-4_7
  45. Schuller B, Batliner A (2013) Computational paralinguistics: emotion, affect and personality in speech and language processing. Wiley, Chichester, UK. doi: 10.1002/9781118706664
  46. Van Santen J, Mishra T, Klabbers E (2008) Prosodic processing. In: Benesty J, Sondhi MM, Huang Y (eds) Springer handbook of speech processing. Springer, Berlin Heidelberg, pp 471–488. doi: 10.1007/978-3-540-49127-9_23
  47. Väyrynen E (2014) Emotion recognition from speech using prosodic features. Doctoral Dissertation, University of Oulu, Finland.
  48. Ververidis D, Kotropoulos C (2006) Emotional speech recognition: resources, features and methods. Speech Commun 48(9):1162–1181. doi: 10.1016/j.specom.2006.04.003 CrossRefGoogle Scholar
  49. Wagner J, Lingenfelser F, Baur T, Damian I, Kistler F, André E (2013) The social signal interpretation (SSI) framework: multimodal signal processing and recognition in real-time. In: Proceedings of the 21st ACM international conference on Multimedia (MM’13), Barcelona, Spain, 21–25 Oct 2013, pp 831–834 doi: 10.1145/2502081.2502223
  50. Zhang C, Zhang Z (2010) A survey of recent advances in face detection. Microsoft TechReport MSR-TR-2010-66.

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.School of Computing and MathematicsUlster UniversityNorthern IrelandUK
  2. 2.ETSI Informática y TelecomunicaciónUniversity of GranadaGranadaSpain
  3. 3.Department of Computer ScienceUniversidad Carlos III de MadridMadridSpain

Personalised recommendations