A Multimodal Approach to Image Sentiment Analysis

  • António GasparEmail author
  • Luís A. Alexandre
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11871)


Multimodal sentiment analysis is a process for the classification of the content of composite comments in social media at the sentiment level that takes into consideration not just the textual content but also the accompanying images. A composite comment is normally represented by the union of text and image. Multimodal sentiment analysis has a great dependency on text to obtain its classification, because image analysis can be very subjective according to the context where the image is inserted. In this paper we propose a method that reduces the text analysis dependency on this kind of classification giving more importance to the image content. Our method is divided into three main parts: a text analysis method that was adapted to the task, an image classifier tuned with the dataset that we use, and a method that analyses the class content of an image and checks the probability that it belongs to one of the possible classes. Finally a weighted sum takes the results of these methods into account to classify content according to its sentiment class. We improved the accuracy on the dataset used by more than 9%.


Multimodal sentiment analysis Image Text Deep learning 


  1. 1.
    Pretrained Models GitHub pretrained models for pytorch github. Accessed 17 Jun 2019
  2. 2.
    Pretrained Models pretrained models for pytorch. Accessed 17 Jun 2019
  3. 3.
    TextBlob. Accessed 17 Jun 2019
  4. 4.
    Bonasoli, W., Dorini, L., Minetto, R., Silva, T.: Sentiment analysis in outdoor images using deep learning, pp. 181–188 (2018).
  5. 5.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR 2009 (2009)Google Scholar
  6. 6.
    Hovy, E.H.: What are sentiment, affect, and emotion? applying the methodology of Michael Zock to sentiment analysis. In: Gala, N., Rapp, R., Bel-Enguix, G. (eds.) Language Production, Cognition, and the Lexicon. TSLT, vol. 48, pp. 13–24. Springer, Cham (2015). Scholar
  7. 7.
    Hutto, C., Gilbert, E.: Vader: a parsimonious rule-based model for sentiment analysis of social media text (2015)Google Scholar
  8. 8.
    Paszke, A., et al.: Automatic differentiation in PyTorch (2017)Google Scholar
  9. 9.
    Pawar, A.B., Jawale, M.A., Kyatanavar, D.N.: Fundamentals of sentiment analysis: concepts and methodology. In: Pedrycz, W., Chen, S.-M. (eds.) Sentiment Analysis and Ontology Engineering. SCI, vol. 639, pp. 25–48. Springer, Cham (2016). Scholar
  10. 10.
    Vadicamo, L., et al.: Cross-media learning for image sentiment analysis in the wild. In: 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 308–317, October 2017.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Instituto de TelecomunicaçõesUniversidade da Beira InteriorCovilhãPortugal

Personalised recommendations