Towards emotion recognition from contextual information using machine learning

Abstract

Emotions influence cognitive processes that underlie human behavior. Whereas experiencing negative emotions may lead to develop psychological disorders, experiencing positive emotions may improve creative thinking and promote cooperative behavior. The importance of human emotions has led to the development of automatic emotion recognition systems based on analysis of speech waveforms, facial expressions, and physiological signals as well as text data mining. However, emotions are associated with a context (in which emotions are actually experienced), hence, this work focuses on emotion recognition from contextual information. In this paper, we present a study aimed to assess the feasibility of automatically recognizing emotions from individuals’ contexts. In this study, 32 participants provided information using a mobile application about their emotions and the context (e.g., companions, activities, and locations) in which these emotions were experienced. We used machine learning techniques to build individual models, general models, and gender-specific models to automatically recognize emotions of participants. The empirical results show that individuals’ emotions are highly related to their context and that automatic recognition of emotions in real-world situations is feasible by using contextual data.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. Ahn H, Picard RW (2005) Affective-cognitive learning and decision making: a motivational reward framework for affective agents. In: Tao J, Tan T, Picard RW (eds) Affective computing and intelligent Interaction. ACII 2005. Lecture notes in computer science, vol 3784. Springer, Berlin, Heidelberg, pp 866–873

  2. Alegre U, Augusto JC, Clark T (2016) Engineering context-aware systems and applications: a survey. J Syst Softw 117:55–83

    Google Scholar 

  3. Ashkanasy NM, Daus CS (2002) Emotion in the workplace: the new challenge for managers. Acad Manage Exec 16(1):76–86

    Google Scholar 

  4. Becerra R, Preece D, Campitelli G, Scott-Pillow G (2019) The assessment of emotional reactivity across negative and positive emotions: development and validation of the perth emotional reactivity scale (pers). Assessment 26(5):867–879

    Google Scholar 

  5. Bechara A (2004) The role of emotion in decision-making: evidence from neurological patients with orbitofrontal damage. Brain Cogn 55(1):30–40

    Google Scholar 

  6. Bechara A, Damasio H, Damasio AR (2000) Emotion, decision making and the orbitofrontal cortex. Cereb Cortex 10(3):295–307

    Google Scholar 

  7. Bellavista P, Corradi A, Fanelli M, Foschini L (2012) A survey of context data distribution for mobile ubiquitous systems. ACM Comput Surv (CSUR) 44(4):1–45

    Google Scholar 

  8. Brave S, Nass C (2007) Emotion in human-computer interaction. In: Sears A, Jacko JA (eds) The human-computer interaction handbook: fundamentals, evolving technologies and emerging applications. CRC Press, Boca Raton, pp 103–118

    Google Scholar 

  9. Breazeal C (2003) Emotion and sociable humanoid robots. Int J Hum Comput Stud 59(1):119–155

    Google Scholar 

  10. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    MATH  Google Scholar 

  11. Cabanac M (1981) Physiological signals for thermal comfort. In: Cena K, Clark JA (eds) Bioengineering, thermal physiology and comfort. Elsevier, Amsterdam, pp 181–192

    Google Scholar 

  12. Calvo RA, D’Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1(1):18–37

    Google Scholar 

  13. Conati C (2002) Probabilistic assessment of user’s emotions in educational games. Appl Artif Intell 16(7–8):555–575

    Google Scholar 

  14. Costa H, Macedo L (2012) Affective computing. Tech Report, University of Coimbra, Coimbra, Portugal. https://eden.dei.uc.pt/~hpcosta/docs/papers/techReports/TAMC-stateOfTheArt.pdf

  15. Croy I, Olgun S, Joraschky P (2011) Basic emotions elicited by odors and pictures. Emotion 11(6):1331–1335

    Google Scholar 

  16. Deng J, Xu X, Zhang Z, Frühholz S, Schuller B (2017) Universum autoencoder-based domain adaptation for speech emotion recognition. IEEE Signal Process Lett 24(4):500–504

    Google Scholar 

  17. Dimotakis N, Scott BA, Koopman J (2011) An experience sampling investigation of workplace interactions, affective states, and employee well-being. J Organ Behav 32(4):572–588

    Google Scholar 

  18. Ekman P (2000) Basic emotions. In: Dalgleish T, Power M (eds) Handbook of cognition and emotion. Wiley, pp 45–60

  19. Ekman PE, Davidson RJ (1994) The nature of emotion: fundamental questions. Oxford University Press, Oxford

    Google Scholar 

  20. El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recogn 44(3):572–587

    MATH  Google Scholar 

  21. Farmer R, Sundberg ND (1986) Boredom proneness-the development and correlates of a new scale. J Pers Assess 50(1):4–17

    Google Scholar 

  22. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    MATH  Google Scholar 

  23. Fawcett T (2006) An introduction to roc analysis. Pattern Recogn Lett 27(8):861–874

    MathSciNet  Google Scholar 

  24. Forman G, Scholz M (2010) Apples-to-apples in cross-validation studies: pitfalls in classifier performance measurement. ACM SIGKDD Explor Newslett 12(1):49–57

    Google Scholar 

  25. Fredrickson BL (2001) The role of positive emotions in positive psychology: the broaden-and-build theory of positive emotions. Am Psychol 56(3):218–226

    Google Scholar 

  26. Giatsoglou M, Vozalis MG, Diamantaras K, Vakali A, Sarigiannidis G, Chatzisavvas KC (2017) Sentiment analysis leveraging emotions and word embeddings. Expert Syst Appl 69:214–224

    Google Scholar 

  27. Granat A, Gadassi R, Gilboa-Schechtman E, Feldman R (2017) Maternal depression and anxiety, social synchrony, and infant regulation of negative and positive emotions. Emotion 17(1):11–27

    Google Scholar 

  28. Grünerbl A, Muaremi A, Osmani V, Bahle G, Oehler S, Tröster G, Mayora O, Haring C, Lukowicz P (2015) Smartphone-based recognition of states and state changes in bipolar disorder patients. IEEE J Biomed Health Inform 19(1):140–148

    Google Scholar 

  29. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. ACM SIGKDD Explor Newslett 11(1):10–18

    Google Scholar 

  30. Hutchings CV, Shum KW, Gawkrodger DJ (2001) Occupational contact dermatitis has an appreciable impact on quality of life. Contact Dermat 45(1):17–20

    Google Scholar 

  31. John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the eleventh conference on uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc., San Francisco, pp 338–345

  32. Junot A, Paquet Y, Martin-Krumm C (2017) Passion for outdoor activities and environmental behaviors: a look at emotions related to passionate activities. J Environ Psychol 53:177–184

    Google Scholar 

  33. Kanjo E, Al-Husain L, Chamberlain A (2015) Emotions in context: examining pervasive affective sensing systems, applications, and analyses. Pers Ubiquit Comput 19(7):1197–1212

    Google Scholar 

  34. Keller MC, Fredrickson BL, Ybarra O, Côté S, Johnson K, Mikels J, Conway A, Wager T (2005) A warm heart and a clear head: the contingent effects of weather on mood and cognition. Psychol Sci 16(9):724–731

    Google Scholar 

  35. Kim HJ, Choi YS (2011) Emosens: affective entity scoring, a novel service recommendation framework for mobile platform. In: Proceedings of the 5th ACM conference on recommender system, pp 1–4

  36. Kim M, Chong SC, Chun C, Choi Y (2017) Effect of thermal sensation on emotional responses as measured through brain waves. Build Environ 118:32–39

    Google Scholar 

  37. Kołakowska A (2018) Usefulness of keystroke dynamics features in user authentication and emotion recognition. In: Hippe Z, Kulikowski J, Mroczek T (eds) Human-computer systems interaction. Advances in intelligent systems and computing, vol 551. Springer, Cham, pp 42–52

    Google Scholar 

  38. Kreibig SD (2010) Autonomic nervous system activity in emotion: a review. Biol Psychol 84(3):394–421

    Google Scholar 

  39. Le Cessie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Appl Stat 41(1):191–201

    MATH  Google Scholar 

  40. Lee H, Choi YS, Lee S, Park I (2012) Towards unobtrusive emotion recognition for affective social communication. In: IEEE consumer communications and networking conference, pp 260–264

  41. Lewis M (2008) The emergence of human emotions. In: Lewis M, Haviland-Jones JM, Feldman Barrett L (eds) Handbook of emotions, 3rd edn. Guilford Press, New York, pp 304–319

    Google Scholar 

  42. LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceedings of the 11th annual international conference on Mobile systems, applications, and services, Taipei, Taiwan, pp 389–402

  43. Liu KH, Huang DS, Li B (2007) Improving the performance of ICA based microarray data prediction models with genetic algorithm. In: 2007 IEEE congress on evolutionary computation, Singapore, pp 606–611

  44. Loewenstein G, Lerner JS (2003) The role of affect in decision making. In: Davidson RJ, Sherer KR, Goldsmith HH (eds) Handbook of affective science, Oxford University Press, pp 619–642

    Google Scholar 

  45. Martin-Krumm C, Fenouillet F, Csillik A, Kern L, Besançon M, Heutte J, Paquet Y, Delas Y, Trousselard M, Lecorre B et al (2018) Changes in emotions from childhood to young adulthood. Child Indic Res 11(2):541–561

    Google Scholar 

  46. Mesquita B, Boiger M, De Leersnyder J (2017) Doing emotions: the role of culture in everyday emotions. Eur Rev Soc Psychol 28(1):95–133

    Google Scholar 

  47. Morrison AS, Mateen MA, Brozovich FA, Zaki J, Goldin PR, Heimberg RG, Gross JJ (2016) Empathy for positive and negative emotions in social anxiety disorder. Behav Res Ther 87:232–242

    Google Scholar 

  48. Nalepa GJ, Kutt K, Bobek S (2019) Mobile platform for affective context-aware systems. Future Gener Comput Syst 92:490–503

    Google Scholar 

  49. Nass C, Takayama L, Brave S (2006) Socializing consistency: from technical homogeneity to human epitome. In: Zhang P, Galletta DF (eds) Human-computer interaction and management information systems: foundations. M. E. Sharpe, Armonk, NY, pp 373–391

    Google Scholar 

  50. Oh K, Park HS, Cho SB (2010) A mobile context sharing system using activity and emotion recognition with bayesian networks. In: 7th international conference on ubiquitous intelligence & computing and 7th international conference on autonomic & trusted computing, Xian, Shaanxi, pp 244–249

  51. Ortony A, Clore GL, Collins A (1990) The cognitive structure of emotions. Cambridge University Press, Cambridge

    Google Scholar 

  52. Panda R, Malheiro RM, Paiva RP (2019) Novel audio features for music emotion recognition. IEEE Trans Affect Comput 1:1–1

    Google Scholar 

  53. Pekrun R, Vogl E, Muis KR, Sinatra GM (2017) Measuring emotions during epistemic activities: the epistemically-related emotion scales. Cogn Emot 31(6):1268–1276

    Google Scholar 

  54. Phelps EA (2006) Emotion and cognition: insights from studies of the human amygdala. Annu Rev Psychol 57:27–53

    Google Scholar 

  55. Politou E, Alepis E, Patsakis C (2017) A survey on mobile affective computing. Comput Sci Rev 25:79–100

    Google Scholar 

  56. Ptaszynski M, Rzepka R, Araki K (2010) On the need for context processing in affective computing. Proc Fuzzy Syst Sympos Jpn Soc Fuzzy Theory Intell Inform 26:920–924

    Google Scholar 

  57. Rodríguez PM, Del Pino DA, Alvaredo RB (2011) De lo psicológico a lo fisiológico en la relación entre emociones y salud. Revista Psicología Científica 13(19):34–39

    Google Scholar 

  58. Royet JP, Zald D, Versace R, Costes N, Lavenne F, Koenig O, Gervais R (2000) Emotional responses to pleasant and unpleasant olfactory, visual, and auditory stimuli: a positron emission tomography study. J Neurosci 20(20):7752–7759

    Google Scholar 

  59. Salido Ortega MG, Rodriguez LF, Gutierrez-Garcia JO (2018) Energy-aware data collection from the internet of things for building emotional profiles. In: Third international conference on fog and mobile edge computing (FMEC), Barcelona, pp 234–239

  60. Sandstrom GM, Lathia N, Mascolo C, Rentfrow PJ (2017) Putting mood in context: using smartphones to examine how people feel in different locations. J Res Pers 69:96–101

    Google Scholar 

  61. Seo J, Laine TH, Sohn KA (2019) Machine learning approaches for boredom classification using eeg. J Ambient Intell Human Comput. https://doi.org/10.1007/s12652-019-01196-3

    Article  Google Scholar 

  62. Sheldon KM (1994) Emotionality differences between artists and scientists. J Res Pers 28(4):481–491

    Google Scholar 

  63. Soleimaninejadian P, Zhang M, Liu Y, Ma S (2018) Mood detection and prediction based on user daily activities. In: First Asian conference on affective computing and intelligent interaction (ACII Asia), Beijing, pp 1–6

  64. Sprenkle DH, Piercy FP (2005) Pluralism, diversity, and sophistication in family therapy research. Res Methods Fam Therapy 2:3–18

    Google Scholar 

  65. Stone AA, Schwartz JE, Schkade D, Schwarz N, Krueger A, Kahneman D (2006) A population approach to the study of emotion: diurnal rhythms of a working day examined with the day reconstruction method. Emotion 6(1):139

    Google Scholar 

  66. Swets JA (1988) Measuring the accuracy of diagnostic systems. Science 240(4857):1285–1293

    MathSciNet  MATH  Google Scholar 

  67. Tagar MR, Federico CM, Halperin E (2011) The positive effect of negative emotions in protracted conflict: the case of anger. J Exp Soc Psychol 47(1):157–164

    Google Scholar 

  68. Tangney JP, Wagner P, Fletcher C, Gramzow R (1992) Shamed into anger? the relation of shame and guilt to anger and self-reported aggression. J Pers Soc Psychol 62(4):669

    Google Scholar 

  69. Wegrzyn M, Vogt M, Kireclioglu B, Schneider J, Kissler J (2017) Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS One 12(5):1–15

    Google Scholar 

  70. Wharton AS, Erickson RI (1993) Managing emotions on the job and at home: understanding the consequences of multiple emotional roles. Acad Manag Rev 18(3):457–486

    Google Scholar 

  71. Wingenbach TS, Ashwin C, Brosnan M (2018) Sex differences in facial emotion recognition across varying expression intensity levels from videos. PLoS One 13(1):1–18

    Google Scholar 

  72. Witten IH, Frank E, Hall MA, Pal CJ (2016) Data mining: practical machine learning tools and techniques. Morgan Kaufmann, Burlington

    Google Scholar 

  73. Yin Z, Zhao M, Wang Y, Yang J, Zhang J (2017) Recognition of emotions using multimodal physiological signals and an ensemble deep learning model. Comput Methods Programs Biomed 140:93–110

    Google Scholar 

  74. Zhang Y, Tang J, Sun J, Chen Y, Rao J (2010) Moodcast: emotion prediction via dynamic continuous factor graph model. In: IEEE international conference on data mining, Sydney, NSW, pp 1193–1198

  75. Zhang X, Li W, Chen X, Lu S (2018) Moodexplorer: towards compound emotion detection via smartphone sensing. Proc ACM Interact Mobile Wear Ubiquitous Technol 1(4):176:1–176:30

    Google Scholar 

  76. Zualkernan I, Aloul F, Shapsough S, Hesham A, El-Khorzaty Y (2017) Emotion recognition using mobile phones. Comput Electr Eng 60:1–13

    Google Scholar 

Download references

Acknowledgements

J. O. Gutierrez-Garcia gratefully acknowledges the financial support from the Asociación Mexicana de Cultura, A.C. This work was supported by PFCE 2019.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Luis-Felipe Rodríguez.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Salido Ortega, M., Rodríguez, L. & Gutierrez-Garcia, J.O. Towards emotion recognition from contextual information using machine learning. J Ambient Intell Human Comput 11, 3187–3207 (2020). https://doi.org/10.1007/s12652-019-01485-x

Download citation

Keywords

  • Emotion recognition
  • Context-aware applications
  • Smart devices
  • Affective computing