Associating Textual Features with Visual Ones to Improve Affective Image Classification

  • Ningning Liu
  • Emmanuel Dellandréa
  • Bruno Tellez
  • Liming Chen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6974)


Many images carry a strong emotional semantic. These last years, some investigations have been driven to automatically identify induced emotions that may arise in viewers when looking at images, based on low-level image properties. Since these features can only catch the image atmosphere, they may fail when the emotional semantic is carried by objects. Therefore additional information is needed, and we propose in this paper to make use of textual information describing the image, such as tags. Thus, we have developed two textual features to catch the text emotional meaning: one is based on the semantic distance matrix between the text and an emotional dictionary, and the other one carries the valence and arousal meanings of words. Experiments have been driven on two datasets to evaluate visual and textual features and their fusion. The results have shown that our textual features can improve the classification accuracy of affective images.


textual feature visual feature affective image classification fusion 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Smeulders, A.W.M., et al.: Content-based Image Retrieval: the end of the early years. IEEE Trans. PAMI 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  2. 2.
    Zeng, Z., et al.: A survey of affect recognition methods: audio, visual and spontaneous expressions. IEEE Transactions PAMI 31(1), 39–58 (2009)CrossRefGoogle Scholar
  3. 3.
    Wang, W., He, Q.: A survey on emotional semantic image retrieval. In: ICIP, pp. 117–120 (2008)Google Scholar
  4. 4.
    Wang, S., Wang, X.: Emotion semantics image retrieval: a brief overview. In: Tao, J., Tan, T., Picard, R.W. (eds.) ACII 2005. LNCS, vol. 3784, pp. 490–497. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  5. 5.
    Al-Ani, A., Deriche, M.: A new technique for combing multiple classifiers using the Dempster Shafer theory of evidence. J. Artif. Intell. Res. 17, 333–361 (2002)zbMATHGoogle Scholar
  6. 6.
    Columbo, C., Del Bimbo, A., Pala, P.: Semantics in visual information retrieval. IEEE Multimedia 6(3), 38–53 (1999)CrossRefGoogle Scholar
  7. 7.
    Itten, J.: The art of colour. Otto Maier Verlab, Ravensburg, Germany (1961)Google Scholar
  8. 8.
    Dellandréa, E., Liu, N., Chen, L.: Classification of affective semantics in images based on discrete and dimensional models of emotions. In: CBMI, pp. 99–104 (2010)Google Scholar
  9. 9.
    Yanulevskaya, V., et al.: Emotional valence categorization using holistic image features. In: ICIP, pp. 101–104 (2008)Google Scholar
  10. 10.
    Weining, W., Yinlin, Y., Shengming, J.: Image retrieval by emotional semantics: A study of emotional space and feature extraction. ICSMC 4, 3534–3539 (2006)Google Scholar
  11. 11.
    Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. ACM Multimedia (2010)Google Scholar
  12. 12.
    Wang, G., Hoiem, D., Forsyth, D.: Building text features for object image classification. In: CVPR, pp. 1367–1374 (2009)Google Scholar
  13. 13.
    Hevner, K.: Experimental studies of the elements of expression in music. American Journal of Psychology 48(2), 246–268 (1936)CrossRefGoogle Scholar
  14. 14.
    Natural language toolkit,
  15. 15.
    Bradley, M.M., Lang, P.J.: Affective norms for English words (ANEW). Tech. Rep C-1, GCR in Psychophysiology, University of Florida (1999)Google Scholar
  16. 16.
    Tamura, H., Mori, S., Yamawaki, T.: Textural features corresponding to visual perception. IEEE Transactions on SMC 8(6), 460–473 (1978)Google Scholar
  17. 17.
    Liu, N., Dellandréa, E., Tellez, B., Chen, L.: Evaluation of Features and Combination Approaches for the Classification of Emotional Semantics in Images. VISAPP (2011)Google Scholar
  18. 18.
    Datta, R., Li, J., Wang, J.Z.: Content-based image retrieval: approaches and trends of the new age. In: ACM Workshop MIR (2005)Google Scholar
  19. 19.
    Ke, Y., Tang, X., Jing, F.: The Design of High-Level Features for Photo Quality Assessment. In: CVPR (2006)Google Scholar
  20. 20.
    Dunker, P., Nowak, S., Begau, A., Lanz, C.: Content-based mood classification for photos and music. In: ACM MIR, pp. 97–104 (2008)Google Scholar
  21. 21.
    Lang, P.J., Bradley, M.M., Cuthbert, B.N.: The IAPS: Technical manual and affective ratings. Tech. Rep A-8., GCR in Psychophysiology, Unv. of Florida (2008)Google Scholar
  22. 22.
    Huiskes, M.J., Lew, M.S.: The MIR Flickr Retrieval Evaluation. In: ACM Multimedia Information Retrieval, MIR 2008 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Ningning Liu
    • 1
  • Emmanuel Dellandréa
    • 1
  • Bruno Tellez
    • 2
  • Liming Chen
    • 1
  1. 1.Ecole Centrale de Lyon, LIRIS, UMR5205, Université de Lyon, CNRSFrance
  2. 2.Université Lyon 1, LIRIS, UMR5205France

Personalised recommendations