Mutual Information-Based Emotion Recognition

  • Yue Cui
  • Suhuai Luo
  • Qi Tian
  • Shiliang Zhang
  • Yu Peng
  • Lei Jiang
  • Jesse S. Jin
Conference paper


Emotions that arise in viewers in response to videos play an essential role in content-based indexing and retrieval. However, the emotional gap between low-level features and high-level semantic meanings is not well understood. This paper proposes a general scheme for video emotion identification using mutual information-based feature selection followed by regression. Continuous arousal and valence values are used to measure video affective content in dimensional arousal-valence space. Firstly, rich audio-visual features are extracted from video clips. The minimum redundancy and maximum relevance feature selection is then used to select most representative feature subsets for arousal and valence modelling. Finally support vector regression is employed to model arousal and valence estimation functions. As evaluated via tenfold cross-validation, the estimation results achieved by our scheme for arousal and valence are: mean absolute error, 0.1358 and 0.1479, variance of absolute error, 0.1074 and 0.1175, respectively. Encouraging results demonstrate the effectiveness of our proposed method.

Key words

Affective content analysis Mutual information-based feature selection Support vector regression 



This work was supported by a CSC-Newcastle scholarship.


  1. 1.
    Hanjalic, A., Xu, L.Q.: Affective video content representation and modeling. IEEE Transactions on Multimedia, vol. 7, pp. 143–154 (2005)CrossRefGoogle Scholar
  2. 2.
    Kang, H.B.: Affective content detection using HMMs. Proceedings of the 11th ACM International Conference on Multimedia (MM), Berkeley, California, USA, pp. 259–262 (2003)Google Scholar
  3. 3.
    Xu, M., Jin, J.S., Luo, S., Duan, L.: Hierarchical movie affective content analysis based on arousal and valence features. Proceeding of the 16th ACM International Conference on Multimedia (MM), Vancouver, Canada, pp. 677–680 (2008)Google Scholar
  4. 4.
    Xu, M., Chia, L.T., Jin, J.: Affective content analysis in comedy and horror videos by audio emotional event detection. In: IEEE International Conference on Multimedia and Expo (ICME), pp. 621–625. IEEE Press, New York (2005)Google Scholar
  5. 5.
    Zhang, S., Huang, Q., Jiang, S., Gao, W., Tian, Q.: Affective visualization and retrieval for music video. IEEE Transactions on Multimedia, vol. 12, pp. 510–522 (2010)CrossRefGoogle Scholar
  6. 6.
    Zhang, S.L., Tian, Q., Huang, Q.M., Gao, W., Li, S.P.: Utilizing Affective Analysis for Efficient Movie Browsing. 16th IEEE International Conference on Image Processing, Vols 1-6, pp. 1833–1836 (2009)Google Scholar
  7. 7.
    Arifin, S., Cheung, P.Y.K.: Affective level video segmentation by utilizing the pleasure-arousal-dominance information. IEEE Transactions on Multimedia, vol. 10, pp. 1325–1341 (2008)CrossRefGoogle Scholar
  8. 8.
    Zhang, S., Xu, Y.J., Jia, J., Cai, L.H.: Analysis and Modeling of Affective Audio Visual Speech Based on PAD Emotion Space. 6th International Symposium on Chinese Spoken Language Processing, pp. 281–284 (2008)Google Scholar
  9. 9.
    Li, D.G., Sethi, I.K., Dimitrova, N., McGee, T.: Classification of general audio data for content-based retrieval. Pattern Recognition Letters 22, pp. 533–544 (2001)zbMATHCrossRefGoogle Scholar
  10. 10.
    Lu, L., Liu, D., Zhang, H.J.: Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio Speech and Language Processing 14, pp. 5–18 (2006)CrossRefGoogle Scholar
  11. 11.
    Lu, L., Zhang, H.J., Li, S.Z.: Content-based audio classification and segmentation by using support vector machines. Multimedia Systems 8, pp. 482–491 (2003)CrossRefGoogle Scholar
  12. 12.
    Valdez, P., Mehrabian, A.: Effects of Color on Emotions. Journal of Experimental Psychology-General 123, pp. 394–409 (1994)CrossRefGoogle Scholar
  13. 13.
    Wang, H.L., Cheong, L.F.: Affective understanding in film. IEEE Transactions on Circuits and Systems for Video Technology 16, pp. 689–704 (2006)CrossRefGoogle Scholar
  14. 14.
    Guyon, I., Elisseeff, A.: An introduction to variable and feature selection. Journal of Machine Learning Research 3, pp. 1157–1182 (2003)zbMATHGoogle Scholar
  15. 15.
    Ding, C., Peng, H.: Minimum redundancy feature selection from microarray gene expression data. Journal of Bioinformatics and Computational Biology 3, pp. 185–205 (2005)CrossRefGoogle Scholar
  16. 16.
    Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, pp. 1226–1238 (2005)CrossRefGoogle Scholar
  17. 17.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 27:21–27:27 (2011)Google Scholar
  18. 18.
    Hsu, C.-W., Chang, C.-C., Lin, C.-J.: A Practical Guide to Support Vector Classification. Department of Computer Science and Information Engineering, National Taiwan University, Taipei (2003)Google Scholar
  19. 19.
    Cui, Y., Lin, J.S., Zhang, S.L., Luo, S.H., Tian, Q.: Correlation-Based Feature Selection and Regression. Advances in Multimedia Information Processing-PCM, Pt I 6297, pp. 25–35 (2010)Google Scholar
  20. 20.
    Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.: Predicting human brain activity associated with the meanings of nouns. Science 320, pp. 1191–1195 (2008)CrossRefGoogle Scholar
  21. 21.
    Hu, X., Deng, F., Li, K., Zhang, T., Chen, H., Jiang, X., Lv, J., Zhu, D., Faraco, C., Zhang, D., Mesbah, A., Han, J., Hua, X., Xie, L., Miller, S., Lei, G., Liu, T.: Bridging low-level features and high-level semantics via fMRI brain imaging for video classification. Proceedings of the International Conference on Multimedia (MM), Firenze, Italy, pp. 451–460 (2010)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2013

Authors and Affiliations

  • Yue Cui
    • 1
  • Suhuai Luo
    • 1
  • Qi Tian
    • 2
  • Shiliang Zhang
    • 3
  • Yu Peng
    • 1
  • Lei Jiang
    • 1
  • Jesse S. Jin
    • 1
  1. 1.School of Design, Communication and Information TechnologyUniversity of NewcastleNewcastle upon TyneAustralia
  2. 2.Department of Computer ScienceUniversity of Texas at San AntonioSan AntonioUSA
  3. 3.Key Laboratory of Intelligent Information Processing, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina

Personalised recommendations