A module-based framework to emotion recognition by speech: a case study in clinical simulation

  • Luana Okino SawadaEmail author
  • Leandro Yukio Mano
  • José Rodrigues Torres Neto
  • Jó Ueyama
Original Research


Current research in human–computer interaction reveals the importance of taking into account emotional aspects in the interaction with computer systems. The main objective of promoting this emotional recognition is to contribute to the enhanced coherence, consistency and credibility of the computer reactions and responses during human interaction through the human–computer interface. In that context, computer systems can be explored that can identify and classify the user’s emotional condition. In this study, computer techniques are used to identify and classify emotional aspects based on the users’ discourse, aiming to assess the emotional behavior in the daily reality of critical care professionals. Hence, in the study of computer techniques and psychological theories, both areas can be related to classify emotion through a framework for the acquisition and classification of users’ discourse. The system developed was applied as an additional evaluation method in the development of simulated scenarios in the context of Clinical Simulation and demonstrated its efficiency as a new approach in health assessment.


Human–computer interaction (HCI) Emotion Classification Cognitive computing 



Author would like to thank the support from Foundation for Research Support of the State of Sao Paulo - FAPESP (Grant Numbers 2016/14267-7, 2016/25865-2, 2017/21054-2 and 2017/23655-3) for funding the bulk of this research project. In addition, we would like to thank the University of Groningen and the University of Ottawa.


  1. AbdulGhaffar A, Mostafa SM, Alsaleh A, Sheltami T, Shakshuki EM (2019) Internet of things based multiple disease monitoring and health improvement system. J Ambient Intell Hum Comput. Google Scholar
  2. Ahmed W, Van der Werf G, Kuyper H, Minnaert A (2013) Emotions, self-regulated learning, and achievement in mathematics: a growth curve analysis. J Educ Psychol 105(1):150CrossRefGoogle Scholar
  3. Almeida RGdS, Mazzo A, Martins JCA, Baptista RCN, Girão FB, Mendes IAC (2015) Validation to portuguese of the scale of student satisfaction and self-confidence in learning. Revista latino-americana de enfermagem 23(6):1007–1013CrossRefGoogle Scholar
  4. Arigbabu OA, Mahmood S, Ahmad SMS, Arigbabu AA (2016) Smile detection using hybrid face representation. J Ambient Intell Hum Comput 7(3):415–426CrossRefGoogle Scholar
  5. Bailenson JN, Pontikakis ED, Mauss IB, Gross JJ, Jabon ME, Hutcherson CA, Nass C, John O (2008) Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int J Hum Comput Stud 66(5):303–317CrossRefGoogle Scholar
  6. Baptista RCN, Martins JCA, Pereira MFCR, Mazzo A (2014) Students’ satisfaction with simulated clinical experiences: validation of an assessment scale. Revista latino-americana de enfermagem 22(5):709–715CrossRefGoogle Scholar
  7. Becker ES, Keller MM, Goetz T, Frenzel AC, Taxer JL (2015) Antecedents of teachers’ emotions in the classroom: an intraindividual approach. Front Psychol 6:635CrossRefGoogle Scholar
  8. Brazeal KR, Brown TL, Couch BA (2016) Characterizing student perceptions of and buy-in toward common formative assessment techniques. CBE Life Sci Educ 15(4):ar73CrossRefGoogle Scholar
  9. Carneiro D, Pinheiro AP, Novais P (2017) Context acquisition in auditory emotional recognition studies. J Ambient Intell Hum Comput 8(2):191–203CrossRefGoogle Scholar
  10. Chang J, Scherer S (2017) Learning representations of emotional speech with deep convolutional generative adversarial networks. In: 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 2746–2750.
  11. Chouchani N, Abed M (2019) Enhance sentiment analysis on social networks with social influence analytics. J Ambient Intell Hum Comput. Google Scholar
  12. Ekman P, Friesen WV (1981) The repertoire of nonverbal behavior: categories, origins, usage, and coding. Nonverbal Commun Interact Gesture 1:57–106Google Scholar
  13. Ellsworth PC, Scherer KR (2003) Appraisal processes in emotion. Handb Affect Sci 572:V595Google Scholar
  14. Eyben F, Wöllmer M, Schuller B (2009) Openear—introducing the munich open-source emotion and affect recognition toolkit. In: Affective computing and intelligent interaction and workshops, 2009. ACII 2009. 3rd international conference on, IEEE, pp 1–6Google Scholar
  15. Fast LA, Funder DC (2008) Personality as manifest in word use: correlations with self-report, acquaintance report, and behavior. J Personal Soc Psychol 94(2):334CrossRefGoogle Scholar
  16. Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in Indonesia and The Netherlands. Cognit. Emot. 16(1):61–86CrossRefGoogle Scholar
  17. Frijda NH (1986) The emotions: Studies in emotion and social interaction. Paris: Maison de Sciences de l’HommeGoogle Scholar
  18. Fuentes C, Herskovic V, Rodríguez I, Gerea C, Marques M, Rossel PO (2017) A systematic literature review about technologies for self-reporting emotional information. J Ambient Intell Hum Comput 8(4):593–606CrossRefGoogle Scholar
  19. Gill R (2009) Breaking the silence: the hidden injuries of neo-liberal academia. Secr Silenc Res Process Fem Reflect 21:228–244Google Scholar
  20. Golbeck J, Robles C, Turner K (2011) Predicting personality with social media. In: CHI’11 extended abstracts on human factors in computing systems, ACM, pp 253–262Google Scholar
  21. Goy H, Pichora-Fuller MK, Singh G, Russo FA (2018) Hearing aids benefit recognition of words in emotional speech but not emotion identification. Trends Hearing 22:2331216518801736. CrossRefGoogle Scholar
  22. Halstead J, Green P, Speziale H et al (2005) Core competencies of nurse educators with task statements. National League for Nursing Publications, New YorkGoogle Scholar
  23. Harati S, Crowell A, Mayberg H, Nemati S (2018) Depression severity classification from speech emotion. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, pp 5763–5766.
  24. Hirsh JB, Peterson JB (2009) Personality and language use in self-narratives. J Res Pers 43(3):524–527CrossRefGoogle Scholar
  25. Huang KY, Wu CH, Su MH, Kuo YT (2018) Detecting unipolar and bipolar depressive disorders from elicited speech responses using latent affective structure model. IEEE Trans Affect Comput. Google Scholar
  26. IBM (2017) Cloud infrastructure, storage, security&, more - IBM bluemix. Accessed 13 Sep 2017
  27. IBM (2017) Documentation, IBM Watson developer cloud. Accessed 13 Sep 2017
  28. IBM (2017) Language translator, IBM Watson developer cloud. Accessed 13 Sept 2017
  29. IBM (2017) Speech to text, about speech to text, ibm watson developer cloud. Accessed 13 Sep 2017
  30. IBM (2017) Tone analyzer, about tone analyzer, IBM Watson developer cloud. Accessed 13 Sep 2017
  31. Lee CH, Kim K, Seo YS, Chung CK (2007) The relations between personality and language use. J Gen Psychol 134(4):405–413CrossRefGoogle Scholar
  32. Lichtenstein A, Oehme A, Kupschick S, Jürgensohn T (2008) Comparing two emotion models for deriving affective states from physiological data. In: Affect and emotion in human–computer interaction, Springer Berlin Heidelberg, Berlin, Heidelberg, pp 35–50.
  33. Lugović S, Dunđer I, Horvat M (2016) Techniques and applications of emotion recognition in speech. In: 2016 39th international convention on information and communication technology, electronics and microelectronics (MIPRO). IEEE, pp 1278–1283.
  34. Mano LY (2018) Emotional condition in the health smart homes environment: emotion recognition using ensemble of classifiers. In: 2018 Innovations in intelligent systems and applications (INISTA), IEEE, pp 1–8Google Scholar
  35. Mano LY, Faiçal BS, Nakamura LH, Gomes PH, Libralon GL, Meneguete RI, Filho GP, Giancristofaro GT, Pessin G, Krishnamachari B, Ueyama J (2016) Exploiting iot technologies for enhancing health smart homes through patient identification and emotion recognition. Comput Commun 89–90:178–190. CrossRefGoogle Scholar
  36. Mano LY, Giancristofaro GT, Faiçal BS, Libralon GL, Pessin G, Gomes PH, Ueyama J (2015) Exploiting the use of ensemble classifiers to enhance the precision of user’s emotion classification. In: Proceedings of the 16th international conference on engineering applications of neural networks (INNS), ACM, p 5Google Scholar
  37. Mano LY, Vasconcelos E, Ueyama J (2016) Identifying emotions in speech patterns: adopted approach and obtained results. IEEE Latin Am Trans 14(12):4775–4780CrossRefGoogle Scholar
  38. Mehl MR, Gosling SD, Pennebaker JW (2006) Personality in its natural habitat: manifestations and implicit folk theories of personality in daily life. J Pers Soc Psychol 90(5):862CrossRefGoogle Scholar
  39. Nardelli M, Valenza G, Greco A, Lanata A, Scilingo EP (2015) Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans Affect Comput 6(4):385–394CrossRefGoogle Scholar
  40. Neto JRT, Filho GP, Mano LY, Ueyama J (2018) Verbo: voice emotion recognition database in Portuguese language. J Comput Sci 14(11):1420–1430. CrossRefGoogle Scholar
  41. Oinas-Kukkonen H (2013) A foundation for the study of behavior change support systems. Pers Ubiquitous Comput 17(6):1223–1235CrossRefGoogle Scholar
  42. Parkinson B (1995) Ideas and realities of emotion. Psychology Press, LondonGoogle Scholar
  43. Patwardhan A, Knapp G (2016) Augmenting supervised emotion recognition with rule-based decision model. arXiv:1607.02660
  44. Pennebaker JW, Francis ME, Booth RJ (2001) Linguistic inquiry and word count: LIWC 2001. Lawrence Erlbaum Associates, Mahwah, NJGoogle Scholar
  45. Pennebaker JW, King LA (1999) Linguistic styles: language use as an individual difference. J Pers Soc Psychol 77(6):1296CrossRefGoogle Scholar
  46. Peter C, Urban B (2012) Emotion in human–computer interaction. In: Expanding the frontiers of visual analytics and visualization, Springer London, pp 239–262,
  47. Picard RW (2003) What does it mean for a computer to “have” emotions? In: Emotions in humans and artifacts, MIT Press, chap 7Google Scholar
  48. Scherer KR (2001) Appraisal considered as a process of multilevel sequential checking. Apprais Process Emot Theory Methods Res 92:120Google Scholar
  49. Scherer KR (2005) What are emotions? And how can they be measured? Soc Sci Inf 44(4):695–729CrossRefGoogle Scholar
  50. Stemmler G (2003) Methodological considerations in the psychophysiological study of emotion. Handb Affect Sci 37:225–255Google Scholar
  51. Tausczik YR, Pennebaker JW (2010) The psychological meaning of words: Liwc and computerized text analysis methods. J lang Soc Psychol 29(1):24–54CrossRefGoogle Scholar
  52. Verma P, Sood SK, Kalra S (2018) Cloud-centric iot based student healthcare monitoring framework. J Ambient Intell Hum Comput 9(5):1293–1309. CrossRefGoogle Scholar
  53. Xia R, Liu Y (2017) A multi-task learning framework for emotion recognition using 2d continuous space. IEEE Trans Affect Comput 8(1):3–14. CrossRefGoogle Scholar
  54. Yarkoni T (2010) Personality in 100,000 words: a large-scale analysis of personality and word use among bloggers. J Res Pers 44(3):363–373CrossRefGoogle Scholar
  55. Yogesh C, Hariharan M, Ngadiran R, Adom AH, Yaacob S, Berkai C, Polat K (2017) A new hybrid pso assisted biogeography-based optimization for emotion and stress recognition from speech signal. Expert Syst Appl 69:149–158. CrossRefGoogle Scholar
  56. Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Hum Comput Stud 69(12):801–819CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute of Mathematical and Computer SciencesUniversity of São PauloSão CarlosBrazil
  2. 2.Johann Bernoulli Institute for Mathematics and Computer ScienceUniversity of GroningenGroningenThe Netherlands
  3. 3.School of Eletrical Engineering and Computer ScienceUniversity of OttawaOttawaCanada

Personalised recommendations