Advertisement

Emotion detection from text and speech: a survey

  • Kashfia Sailunaz
  • Manmeet Dhaliwal
  • Jon Rokne
  • Reda Alhajj
Original Article

Abstract

Emotion recognition has emerged as an important research area which may reveal some valuable input to a variety of purposes. People express their emotions directly or indirectly through their speech, facial expressions, gestures or writings. Many different sources of information, such as speech, text and visual can be used to analyze emotions. Nowadays, writings take many forms of social media posts, micro-blogs, news articles, etc., and the content of these posts can be useful resource for text mining to discover and unhide various aspects, including emotions. Extracting emotions behind these postings is an immense and complicated task. To tackle this problem, researchers from diverse fields are trying to find an efficient way to more precisely detect human emotions from various sources, including text and speech. In this sense, different word-based and sentence-based techniques, machine learning, natural language processing methods, etc., have been used to achieve better accuracy. Analyzing emotions can be helpful in many different domains. One such domain is human computer interaction. With the help of emotion recognition, computers can make better decisions to help users. With the increase in popularity of robotic research, emotion recognition will also help making human–robot interaction more natural. This survey covers existing emotion detection research efforts, emotion models, emotion datasets, emotion detection techniques, their features, limitations and some possible future directions. We focus on reviewing research efforts analyzing emotions based on text and speech. We investigated different feature sets that have been used in existing methodologies. We summarize basic achievements in the field and highlight possible extensions for better outcome.

Keywords

Emotion Text Emotion models Emotion recognition Emotion analysis Speech Classifiers 

References

  1. Abak FS, Evrim V (2016) HONET-ICT. IEEE, pp 154–158Google Scholar
  2. Agrawal A, An A (2012) Unsupervised emotion detection from text using semantic and syntactic relations. In: Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology, pp 346–353Google Scholar
  3. Akaike H (1998) Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. Springer, New York, pp 199–213Google Scholar
  4. Anagnostopoulos C-N, Iliou T, Giannoukos I (2015) Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artif Intell Rev 43(2):155–177CrossRefGoogle Scholar
  5. An Y, Sun S, Wang S (2017) Naive Bayes classifiers for music emotion classification based on lyrics. In: IEEE/ACIS international conference on computer and information science (ICIS), pp 635–638Google Scholar
  6. Ayadi ME, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Pattern Recognit 44:572–587CrossRefzbMATHGoogle Scholar
  7. Bell C (1824) Essays on the anatomy and philosophy of expression. John Murray, LondonGoogle Scholar
  8. Bellegarda J (2013) Large-scale personal assistant technology deployment: the Siri experience. ISCA INTERSPEECH, pp 2029–2033Google Scholar
  9. Binali H, Wu C, Potdar V (2010) Computational approaches for emotion detection in text. In: IEEE international conference on digital ecosystems and technologies (DEST), pp 172–177Google Scholar
  10. Borod JC (2000) The neuropsychology of emotion. Oxford University Press, OxfordGoogle Scholar
  11. Broad CD (1954) Emotion and sentiment. J Aesthet Art Crit 13(2):203–214CrossRefGoogle Scholar
  12. Busso C, Deng Z, Yildirim S, Bulut M, Lee C, Kazemzadeh A, Lee S, Neumann U, Narayanan S (2004) Analysis of emotion recognition using facial expression, speech and multimodal information. ACM Multimodal Interfaces 6:205–211Google Scholar
  13. Calvo RA, Kim SM (2013) Emotions in text: dimensional and categorical models. Comput Intell 29(3):527–543MathSciNetCrossRefGoogle Scholar
  14. Calvo R, Kim S (2013) Emotions in text: dimensional and categorical models. Comput Intell 29(3):527–543MathSciNetCrossRefGoogle Scholar
  15. Canales L, Martinez-Barco P (2014) Emotion detection from text: a survey. In: Processing in the 5th Information Systems Research Working Days (JISIC 2014), pp 37–43Google Scholar
  16. Chen LS, Tao H, Huang TS, Miyasato T, Nakatsu R (1998) Emotion recognition from audiovisual information. In: IEEE second workshop on multimedia, signal processing, pp 83–88Google Scholar
  17. Chopade CR (2015) Text based emotion recognition: a survey. Int J Sci Res (IJSR) 4(6):409–414Google Scholar
  18. Cohen I, Garg A, Huang T (2000) Emotion recognition from facial expression using multilevel HMM. In: Neural information processing systemGoogle Scholar
  19. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human–computer interaction. IEEE Signal Process 18(1):32–80CrossRefGoogle Scholar
  20. Daga D, Hudait A, Tripathy HK, Das MN (2016) Automatic emotion detection model from facial expression. In: International conference on advanced communication control and computing technologies, pp 77–85Google Scholar
  21. Darwin C (1998) The expression of the emotions in man and animals. Oxford University Press, OxfordGoogle Scholar
  22. Desmet B, Hoste V (2013) Emotion detection in suicide notes. Expert Syst Appl 40(16):6351–6358CrossRefGoogle Scholar
  23. Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice-Hall, LondonzbMATHGoogle Scholar
  24. Dini L, Bittar A (May 2016) Emotion analysis on twitter: the hidden challenge. In: Language Resources and Evaluation Conference (LREC)Google Scholar
  25. Ekman P (1992) An argument for basic emotions. Cogn Emot 6(3–4):169–200CrossRefGoogle Scholar
  26. Ghazi D, Inkpen D, Szpakowicz S (2014) Prior and contextual emotion of words in sentential context. Comput Speech Lang 28(1):76–92CrossRefGoogle Scholar
  27. Grover S, Verma A (2016) Design for emotion detection of punjabi text using hybrid approach. In: International Conference on Inventive Computation Technologies (ICICT), vol 2, pp 1–6Google Scholar
  28. Gupta U, Chatterjee A, Srikanth R, Agrawal P (2017) A sentiment-and-semantics-based approach for emotion detection in textual conversations. Neu-IR: Workshop on Neural Information Retrieval, SIGIR 2017, ACM, arXiv preprint arXiv:1707.06996
  29. Hajar M (2016) Using youtube comments for text-based emotion recognition. Procedia Comput Sci 83:292–299CrossRefGoogle Scholar
  30. Han K, Yu D, Tashev I (2014) Speech emotion recognition using deep neural network and extreme learning machine. In: InterspeechGoogle Scholar
  31. Hasan M, Rundensteiner E, Agu E (2014) Emotex: detecting emotions in twitter messages. In: 2014 ASE BIGDATA/SOCIALCOM/CYBERSECURITY ConferenceGoogle Scholar
  32. Hasan M, Rundensteiner E, Kong X, Agu E (2017) Using social sensing to discover trends in public emotion. In: IEEE 11th International Conference on Semantic Computing (ICSC), pp 172–179Google Scholar
  33. Hayden M (1998) The ensemble system. Cornell UniversityGoogle Scholar
  34. Hu H, Xu M, Wu W (2007) GMM supervector based SVM with spectral features for speech emotion recognition. In: ICASSP ’07, pp 4-413–4-416Google Scholar
  35. Jain VK, Kumar S, Fernandes SL (2017) Extraction of emotions from multilingual text using intelligent text processing and computational linguistics. J Comput Sci 21:316–326CrossRefGoogle Scholar
  36. Joshi A, Tripathi V, Soni R, Bhattacharyya P, Carman MJ (2016) Emogram: an open-source time sequence-based emotion tracker and its innovative applications. In: Knowledge extraction from text, AAAI WorkshopGoogle Scholar
  37. Kahou SE, Bouthillier X, Lamblin P, Gulcehre C, Michalski V, Konda K, Jean S, Froumenty P, Dauphin Y, Boulanger-Lewandowski N, Ferrari RC, Mirza M, Warde-Farley D, Courville A, Vincent P, Memisevic R, Pal C, Bengio Y (2016) Emonets: multimodal deep learning approaches for emotion recognition in video. J Multimodal User Interfaces 10(2):99–111CrossRefGoogle Scholar
  38. Kanger N, Bathla G (2017) Recognizing emotion in text using neural network and fuzzy logic. Indian J Sci Technol 10(12).  https://doi.org/10.17485/ijst/2017/v10i12/100526
  39. Kang X, Ren F, Wu Y (2017) Exploring latent semantic information for textual emotion recognition in blog articles. IEEE/CAA J Autom Sin 5(1):204–216CrossRefGoogle Scholar
  40. Kao EC-C, Liu C-C, Yang T-H, Hsieh C-T, Soo V-W (2009) Towards text-based emotion detection a survey and possible improvements. In: IEEE international conference on information management and engineering (ICIME), pp 70–74Google Scholar
  41. Kessous L, Castellano G, Caridakis G (2010) Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. J Multimodal User Interface 3(1–2):33–48CrossRefGoogle Scholar
  42. Kim J, Wagner J, Vogt T, Andre E, Jung F, Rehm M (2009) Emotional sensitivity in human–computer interaction. Inf Technol Methods Appl Inform Inf Technol 51(6):325–328Google Scholar
  43. Kjeldskov J, Graham C (2003) A review of Mobile HCI Research Methods. Springer Human–Computer Interaction with Mobile Devices and Services, Mobile HCI, pp 317–335Google Scholar
  44. Kononenko I (1994) Estimating attributes: analysis and extensions of RELIEF. In: European conference on machine learning. Springer, Berlin, pp 171–182Google Scholar
  45. Kudiri KM, Said AM, Nayan MY (2016) Human emotion detection through speech and facial expressions. In: IEEE international conference on computer and information sciences (ICCOINS), pp 351–356Google Scholar
  46. Kwon O, Chan K, Hao J, Lee T (2003) Emotion recognition by speech signals. In: EUROSPEECH Speech Communication and Technology, p 8Google Scholar
  47. Lalitha S, Madhavan A, Bhushan B, Saketh S (2014) Speech emotion recognition. In: ICAECC, pp 1–4Google Scholar
  48. Lalitha S, Patnaik S, Arvind TH, Madhusudhan V, Tripathi S (2015) Emotion recognition through speech signal for human–computer interaction. In: IEEE fifth international symposium on electronic system design, vol 5, pp 217–218Google Scholar
  49. LeDoux JE (1984) Cognition and emotion. Handbook of cognitive neuroscience. Springer, Berlin, pp 357–368CrossRefGoogle Scholar
  50. Lee CM, Narayanan S, Pieraccini R (2001) Recognition of negative emotions from the speech signal. In: IEEE workshop on automatic speech recognition and understanding. ASRU’01, pp 240–243Google Scholar
  51. Li W, Xu H (2014) Text-based emotion classification using emotion cause extraction. Expert Syst Appl 41(4):1742–1749CrossRefGoogle Scholar
  52. Lin Y, Wei G (2005) Speech emotion recognition based on HMM and SVM. In: International conference on machine learning and cybernetics, vol 8, pp 4898–4901Google Scholar
  53. Li X, Pang J, Mo B, Rao Y (2016) Hybrid neural networks for social emotion detection over short text. In: International Joint Conference on Neural Networks (IJCNN), pp 537–544Google Scholar
  54. Lovheim H (2012) A new three-dimensional model for emotions and monoamine neurotransmitters. Med Hypotheses 78(2):341–348CrossRefGoogle Scholar
  55. Maedche A, Morana S, Schacht S, Werth D, Krumeich J (2016) Advanced user assistance systems. Bus Inf Syst Eng 58(5):367–370CrossRefGoogle Scholar
  56. Manser AR (1963) Sketch for a theory of the emotions. Anal Philos 4(1):27–28Google Scholar
  57. Mao X, Zhang B, Luo Y (2007) Speech emotion recognition based on a hybrid of HMM/ANN. In: Proceedings of the 7th conference on 7th WSEAS international conference on applied informatics and communications, vol 7, pp 367–370Google Scholar
  58. Mohammad SM, Bravo-Marquez F (2017) Emotion intensities in tweets. In: Proceedings of the sixth Joint Conference on Lexical and Computational Semantics (*Sem)Google Scholar
  59. Mohammad SM, Kiritchenko S (2015) Using hashtags to capture fine emotion categories from tweets. Comput Intell 31(2):301–326MathSciNetCrossRefGoogle Scholar
  60. Munezero MD, Montero CS, Sutinen E, Pajunen J (2014) Are they different? Affect, feeling, emotion, sentiment, and opinion detection in text. IEEE Trans Affect Comput 5(2):101–111CrossRefGoogle Scholar
  61. Nahin NH, Alam JM, Mahmud H, Hasan K (2014) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inf Technol 33(9):987–996CrossRefGoogle Scholar
  62. Nahin N, Alam J, Mahmud H, Hasan K (2015) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inf Technol 33(9):301–326Google Scholar
  63. Neiberg D, Elenius K, Laskowski K (2006) Emotion recognition in spontaneous speech using GMMs. In: Proceedings of Interspeech, pp 809–812Google Scholar
  64. Oatley K, Johnson-Laird PN (1987) Towards a cognitive theory of emotions. Cogn Emot 1(1):29–50CrossRefGoogle Scholar
  65. Ortony A, Clore GL, Collins A (1988) The cognitive structure of emotions. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  66. Pane J, Myers B, Miller L (2002) Using HCI techniques to design a more usable programming, system. IEEE Human Centric Computing Language and Environment, HCC, pp 198–206Google Scholar
  67. Pan Y, Shen P, Shen L (2012) Speech emotion recognition using support vector machine. Int J Smart Home 6(2):101–108Google Scholar
  68. Perikos I, Hatzilygeroudis I (2016) Recognizing emotions in text using ensemble of classifiers. Eng Appl Artif Intell 51:191–201CrossRefGoogle Scholar
  69. Perikos I, Hatzilygeroudis I (2016) Recognizing emotions in text using ensemble of classifiers. Eng Appl Artif Intell 51:191–201CrossRefGoogle Scholar
  70. Petrushin VA (2000) Emotion recognition in speech signal: experimental study, development, and application. In: Proceedings of ICSLP, pp 222–225Google Scholar
  71. Plutchik R (1980) Emotion: a psychoevolutionary synthesis. Harper and Row, New YorkGoogle Scholar
  72. Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59CrossRefGoogle Scholar
  73. Poria S, Chaturvedi I, Cambria E, Hussain A (2016) Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: IEEE 16th international conference on data mining (ICDM), pp 439–448Google Scholar
  74. Preotiuc-Pietro et al (2016) The valence and arousal facebook posts. In: Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment and social media analysis, San Diego, California, pp 9–15Google Scholar
  75. Rosenblum M, Yacoob Y, Davis L (1994) Human emotion recognition from motion using a radial basis function network architecture. In: IEEE motion of non-rigid and articulated objects, pp 43–49Google Scholar
  76. Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39(6):1161–1178CrossRefGoogle Scholar
  77. Schmitt JJ, Hartje W, Willmes K (1997) Hemispheric asymmetry in the recognition of emotional attitude conveyed by facial expression, prosody and propositional speech. Cortex 33(1):65–81CrossRefGoogle Scholar
  78. Schuller B, Rigoll G, Lang M (2004) Speech emotion recognition combining acoustic features and linguistic information in a hybrid support vector machine-belief network architecture. In: ICASSP, vol 1, pp I-577–I-580Google Scholar
  79. Seehapoch T, Wongthanavasu S (2013) Speech emotion recognition using support vector machines. IEEE Knowl Smart Technol 5:86–91Google Scholar
  80. Semwal N, Kumar A, Narayanan S (2017) Automatic speech emotion detection system using multi-domain acoustic feature selection and classification models. In: IEEE international conference on identity, security and behavior analysis (ISBA), pp 1–6Google Scholar
  81. Sen A, Sinha M, Mannarswamy S, Roy S (2017) Multi-task representation learning for enhanced emotion categorization in short text. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, Cham, pp 324–336Google Scholar
  82. Shaver P, Schwartz J, Kirson D, O’connor C (1987) Emotion knowledge: further exploration of a prototype approach. J Pers Soc Psychol 52(6):1061–1086CrossRefGoogle Scholar
  83. Shivhare SN, Khethawat S (2012) Emotion detection from text. arXiv:1205.4944 [cs.HC]
  84. Silva LC, Ng PC (2000) Bimodal emotion recognition. In: Fourth IEEE international conference on automatic face and gesture recognition, pp 332–335Google Scholar
  85. Soleymani M, Asghari-Esfeden S, Fu Y, Pantic M (2016) Analysis of eeg signals and facial expressions for continuous emotion detection. IEEE Trans Affect Comput 7(1):17–28CrossRefGoogle Scholar
  86. Sreeja PS, Mahalakshmi GS (2017) Emotion models: a review. Int J Control Theory Appl (IJCTA) 10(8):651–657Google Scholar
  87. Stearns S (1976) On selecting features for pattern classifiers. In: 3-d international conference on pattern recognition, Coronado, CA, pp 71–75Google Scholar
  88. Strapparava C, Mihalcea R (2008) Learning to identify emotions in text. In: Proceedings of ACM symposium on applied computing, pp 1556–1560Google Scholar
  89. Summa A, Resch B, GIS Geoinformatics-Z, Strube M (2016) Microblog emotion classification by computing similarity in text, time, and space. In: Proceedings of the workshop on computational modeling of people’s opinions, personality, and emotions in social media, pp 153–162Google Scholar
  90. Tiwari SP, Raju MV, Phonsa G, Deepu DK (2016) A novel approach for detecting emotion in text. Indian J Sci Technol 9(29):1–5Google Scholar
  91. Tripathi V, Joshi A, Bhattacharyya P Emotion analysis from text: a survey. http://www.cfilt.iitb.ac.in/resources/surveys/emotion-analysis-survey-2016-vaibhav.pdf
  92. Vlassis N, Likas A (2002) A greedy EM algorithm for Gaussian mixture learning. Neural Process Lett 15(1):77–87CrossRefzbMATHGoogle Scholar
  93. Yadollahi A, Shahraki AG, Zaiane OR (2017) Current state of text sentiment analysis from opinion to emotion mining. ACM Comput Surv (CSUR) 50(2):25:1–25:33CrossRefGoogle Scholar
  94. Yat S, Lee M, Chen Y, Huang C-R (2010) A text-driven rule-based system for emotion cause detection. In: Proceedings of the NAACL HLT 2010 workshop on computational approaches to analysis and generation of emotion in text. Association for Computational Linguistics, pp 45–53Google Scholar

Copyright information

© Springer-Verlag GmbH Austria, part of Springer Nature 2018

Authors and Affiliations

  • Kashfia Sailunaz
    • 1
  • Manmeet Dhaliwal
    • 1
  • Jon Rokne
    • 1
  • Reda Alhajj
    • 1
  1. 1.Department of Computer ScienceUniversity of CalgaryCalgaryCanada

Personalised recommendations