Multimedia Tools and Applications

, Volume 62, Issue 2, pp 319–332 | Cite as

Actor level emotion magnitude prediction in text and speech

  • Ricardo A. Calix
  • Gerald M. KnappEmail author


The digital universe is expanding at very high rates. New ways of retrieving and enriching text and audio content are required. In this work, a methodology for actor level emotion magnitude prediction in text and speech is proposed. A model is trained to predict emotion magnitudes per actor at any point in a story using previous emotion magnitudes plus current text and speech features which act on the actor’s emotional state. The methodology compares linear and non-linear regression techniques to determine the optimal model that fits the data. Results of the analysis show that non-linear regression models based on Support Vector Regression (SVR) using a Radial Basis Function (RBF) kernel provide the most accurate prediction model. An analysis of the contribution of the features for emotion magnitude prediction is performed.


Artificial intelligence Natural language processing Machine learning Multimedia semantic analysis Affect detection Speech processing 


  1. 1.
    Akenine-Moller T, Haines E, Hoffman N (2008) Real-Time Rendering. A K Peters, Wellesley, MassachusettsCrossRefGoogle Scholar
  2. 2.
    Alm CO, Roth D, Sproat R (2005) Emotions from text: machine learning for text-based emotion prediction. In Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 579–586Google Scholar
  3. 3.
    Alm CO (2011) Affect data. Accessed 30 March 2011
  4. 4.
    Alm CO (2008) Affect in Text and Speech. Dissertation, University of Illinois at Urbana-ChampaignGoogle Scholar
  5. 5.
    Bird S, Klein E, Loper E (2009) Natural Language Processing with Python. 1st ed., O’Reilly MediaGoogle Scholar
  6. 6.
    Boersma P, Weenink D (2011) Praat: doing phonetics by computer. Version 5.2.21. Accessed 30 March 2011
  7. 7.
    Burns B, Morrison C (2003) Temporal Abstraction in Bayesian Networks. In Working Notes of Association for the Advancement of Artificial Intelligence (AAAI), Spring Symposium Workshop: Foundation and Applications of Spatio-Temporal Reasoning, AAAI,Technical Report SS-03-03, 2003Google Scholar
  8. 8.
    Busso C, Lee S, Narayanan S (2009) Analysis of emotionally salient aspects of fundamental frequency for emotion detection. IEEE Transactions on Audio, Speech, and Language Processing Vol. 17, No. 4Google Scholar
  9. 9.
    Calix RA, Mallepudi S, Chen B, Knapp GM (2010) Emotion recognition in text for 3D facial expression rendering. IEEE Transactions on Multimedia, Special Issue on Multimodal Affective Interaction 12(6):544–551CrossRefGoogle Scholar
  10. 10.
    Calix RA, Knapp GM (2011) Affect Corpus 2.0: An extension of a corpus for actor level emotion magnitude detection. In Proceedings of the 2nd ACM Multimedia Systems (MMSys) conference, Feb. 2011, San Jose, California, U.S.A.Google Scholar
  11. 11.
    Chang CC, Lin C (2001) LIBSVM: a library for support vector machines. Accessed 30 March 2011
  12. 12.
    El-Nasr M, Loerger T, Yen J (1999) PETEEI: A pet with evolving emotional intelligence. Proceedings of the third annual conference on autonomous agents, Seattle, Washington, USA, pp. 9–15Google Scholar
  13. 13.
    Gantz J, Reinsel D (2010) The digital universe decade—Are you ready? IDC Report. Accessed 30 March 2011
  14. 14.
    Grimm M, Kroschel K, Narayanan S (2007) Support Vector Regression for automatic recognition of spontaneous emotions in speech. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2007), Vol. 4, pp. IV-1085-IV-1088Google Scholar
  15. 15.
    Jurafsky D, Martin J (2008) Speech and Language Processing, 2nd edn. Prentice Hall, New JerseyGoogle Scholar
  16. 16.
    Lu CY, Hong J, Cruz-Lara S (2006) Emotion detection in textual information by semantic role labeling and web mining techniques. Third Taiwanese-French Conference on Information Technology—TFITGoogle Scholar
  17. 17.
    Luengo I, Navas E, Hernaez I, (2010) Feature analysis and evaluation for automatic emotion identification in speech. IEEE Transactions on Multimedia, Vol. 12, No. 6Google Scholar
  18. 18.
    Mao Y, Lebanon G (2006) Sequential models for sentiment prediction. In Proceedings of the International Conference on Machine Learning (ICML), Workshop on Learning in Structured Output Spaces, Pittsburg, PAGoogle Scholar
  19. 19.
    Moilanen K, Pulman S (2007) Sentiment Composition. In Proceedings of Recent Advances in Natural Language Processing (RANLP 2007), September 27–29, Borovets, Bulgaria, pp. 378–382Google Scholar
  20. 20.
    Neviarouskaya A, Prendinger H, Ishizuka M (2009) Semantically distinct verb classes involved in sentiment analysis. In Proceedings IADIS international conference on applied computing, AC 1:27–35Google Scholar
  21. 21.
    Nuance (2011) Naturally speaking software. Accessed 30 March 2011
  22. 22.
    Pang B, Lee L (2008) Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1–2):1–135CrossRefGoogle Scholar
  23. 23.
    Smola A, Scholkopf B (2004) A tutorial on Support Vector Regression. Stat Comput 14:199–222MathSciNetCrossRefGoogle Scholar
  24. 24.
    Tokuhisa R, Inui K, Matsumoto Y (2008) Emotion classification using massive examples extracted from the web. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), Manchester, pp. 881–888Google Scholar
  25. 25.
    Witten I, Frank E (2005) Data Mining: Practical Machine Learning Tools and Techniques, 2 dth edn. Morgan Kaufmann Publishers Inc., San FranciscozbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.College of EngineeringLouisiana State UniversityBaton RougeUSA

Personalised recommendations