Visual and Textual Sentiment Analysis of Brand-Related Social Media Pictures Using Deep Convolutional Neural Networks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10484)


Social media pictures represent a rich source of knowledge for companies to understand consumers’ opinions, as they are available in real time and at low costs and represent an active feedback which is of importance not only for companies developing products, but also to their rivals and potential consumers. In order to estimate the overall sentiment of a picture, it is essential to not only judge the sentiment of the visual elements but also to understand the meaning of the included text. This paper introduces an approach to estimate the overall sentiment of brand-related pictures from social media based on both visual and textual clues. In contrast to existing papers, we do not consider text accompanying a picture, but text embedded in a picture, which is more challenging since the text has to be detected and recognized first, before its sentiment can be identified. Based on visual and textual features extracted from two trained Deep Convolutional Neural Networks (DCNNs), the sentiment of a picture is identified by a machine learning classifier. The approach was applied and tested on a newly collected dataset, “GfK Verein Dataset” and several machine learning algorithms are compared. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the proposed approach.



This work was funded by GfK Verein ( The authors would like to thank Lara Enzingmüller and Regina Schreder for their help with data preparation.


  1. 1.
    Carolin Kaiser, R.W.: Gaining marketing-relevant knowledge from social media photos - a picture is worth a thousand words. In: Proceedings of the 2016 ESOMAR Congress, New Orleans (2016)Google Scholar
  2. 2.
    Yang, Y., Jia, J., Zhang, S., Wu, B., Chen, Q., Li, J., Xing, C., Tang, J.: How do your friends on social media disclose your emotions? In: AAAI, vol. 14, pp. 1–7 (2014)Google Scholar
  3. 3.
    You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. arXiv preprint arXiv:1509.06041 (2015)
  4. 4.
    Yi, J., Nasukawa, T., Bunescu, R., Niblack, W.: Sentiment analyzer: extracting sentiments about a given topic using natural language processing techniques. In: Third IEEE International Conference on Data Mining, ICDM 2003, pp. 427–434. IEEE (2003)Google Scholar
  5. 5.
    Mukherjee, S., Bhattacharyya, P.: Feature specific sentiment analysis for product reviews. In: Gelbukh, A. (ed.) CICLing 2012. LNCS, vol. 7181, pp. 475–487. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-28604-9_39 CrossRefGoogle Scholar
  6. 6.
    Liciotti, D., Paolanti, M., Frontoni, E., Mancini, A., Zingaretti, P.: Person re-identification dataset with RGB-D camera in a top-view configuration. In: Nasrollahi, K., Distante, C., Hua, G., Cavallaro, A., Moeslund, T.B., Battiato, S., Ji, Q. (eds.) FFER/VAAM -2016. LNCS, vol. 10165, pp. 1–11. Springer, Cham (2017). doi: 10.1007/978-3-319-56687-0_1 CrossRefGoogle Scholar
  7. 7.
    Naspetti, S., Pierdicca, R., Mandolesi, S., Paolanti, M., Frontoni, E., Zanoli, R.: Automatic analysis of eye-tracking data for augmented reality applications: a prospective outlook. In: De Paolis, L.T., Mongelli, A. (eds.) AVR 2016. LNCS, vol. 9769, pp. 217–230. Springer, Cham (2016). doi: 10.1007/978-3-319-40651-0_17 Google Scholar
  8. 8.
    Xu, C., Cetintas, S., Lee, K.C., Li, L.J.: Visual sentiment prediction with deep convolutional neural networks. arXiv preprint arXiv:1411.5731 (2014)
  9. 9.
    Pang, B., Lee, L., et al.: Opinion mining and sentiment analysis. Found. Trends® Inf. Retrieval 2(1–2), 1–135 (2008)Google Scholar
  10. 10.
    Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., Kappas, A.: Sentiment strength detection in short informal text. J. Am. Soc. Inform. Sci. Technol. 61(12), 2544–2558 (2010)CrossRefGoogle Scholar
  11. 11.
    Yuan, J., Mcdonough, S., You, Q., Luo, J.: Sentribute: image sentiment analysis from a mid-level perspective. In: Proceedings of the Second International Workshop on Issues of Sentiment Discovery and Opinion Mining, p. 10. ACM (2013)Google Scholar
  12. 12.
    Chang, Y., Tang, L., Inagaki, Y., Liu, Y.: What is tumblr: a statistical overview and comparison. ACM SIGKDD Explor. Newsl. 16(1), 21–29 (2014)CrossRefGoogle Scholar
  13. 13.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  14. 14.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  15. 15.
    Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: Advances in Neural Information Processing Systems, pp. 649–657 (2015)Google Scholar
  16. 16.
    Bø, T.H., Dysvik, B., Jonassen, I.: LSimpute: accurate estimation of missing values in microarray data with least squares methods. Nucleic Acids Res. 32(3), e34 (2004)CrossRefGoogle Scholar
  17. 17.
    Troyanskaya, O., Cantor, M., Sherlock, G., Brown, P., Hastie, T., Tibshirani, R., Botstein, D., Altman, R.B.: Missing value estimation methods for dna microarrays. Bioinformatics 17(6), 520–525 (2001)CrossRefGoogle Scholar
  18. 18.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)zbMATHGoogle Scholar
  19. 19.
    Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986)Google Scholar
  20. 20.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  21. 21.
    Rish, I.: An empirical study of the naive Bayes classifier. In: IJCAI 2001 Workshop on Empirical Methods in Artificial Intelligence, vol. 3, pp. 41–46. IBM, New York (2001)Google Scholar
  22. 22.
    Lippmann, R.: An introduction to computing with neural nets. IEEE Assp Mag. 4(2), 4–22 (1987)CrossRefGoogle Scholar
  23. 23.
    Paolanti, M., Frontoni, E., Mancini, A., Pierdicca, R., Zingaretti, P.: Automatic classification for anti mixup events in advanced manufacturing system. In: ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, p. V009T07A061. American Society of Mechanical Engineers (2015)Google Scholar
  24. 24.
    Cambria, E.: Affective computing and sentiment analysis. IEEE Intell. Syst. 31(2), 102–107 (2016)CrossRefGoogle Scholar
  25. 25.
    Strapparava, C., Valitutti, A., et al.: Wordnet affect: an affective extension of wordnet. In: LREC, vol. 4, pp. 1083–1086. Citeseer (2004)Google Scholar
  26. 26.
    Esuli, A.: Sentiwordnet: a publicly available lexical resource for opinion mining. In: Proceedings of Language Resources And Evaluation (LREC), Genoa, Italy, pp. 24–26 (2006)Google Scholar
  27. 27.
    Cambria, E., White, B.: Jumping NLP curves: a review of natural language processing research [review article]. IEEE Comput. Intell. Mag. 9(2), 48–57 (2014)CrossRefGoogle Scholar
  28. 28.
    Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
  29. 29.
    Mesnil, G., Mikolov, T., Ranzato, M., Bengio, Y.: Ensemble of generative and discriminative techniques for sentiment analysis of movie reviews. arXiv preprint arXiv:1412.5335 (2014)
  30. 30.
    Chen, T., Borth, D., Darrell, T., Chang, S.F.: Deepsentibank: visual sentiment concept classification with deep convolutional neural networks. arXiv preprint arXiv:1410.8586 (2014)
  31. 31.
    Cambria, E., Poria, S., Bisio, F., Bajpai, R., Chaturvedi, I.: The CLSA model: a novel framework for concept-level sentiment analysis. In: Gelbukh, A. (ed.) CICLing 2015. LNCS, vol. 9042, pp. 3–22. Springer, Cham (2015). doi: 10.1007/978-3-319-18117-2_1 Google Scholar
  32. 32.
    You, Q., Luo, J., Jin, H., Yang, J.: Joint visual-textual sentiment analysis with deep neural networks. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1071–1074. ACM (2015)Google Scholar
  33. 33.
    Yu, Y., Lin, H., Meng, J., Zhao, Z.: Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9(2), 41 (2016)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Liao, M., Shi, B., Bai, X., Wang, X., Liu, W.: Textboxes: a fast text detector with a single deep neural network. arXiv preprint arXiv:1611.06779 (2016)
  35. 35.
    Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. Int. J. Comput. Vis. 116(1), 1–20 (2016)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Bhowmick, P.K., Mitra, P., Basu, A.: An agreement measure for determining inter-annotator reliability of human judgements on affective text. In: Proceedings of the Workshop on Human Judgements in Computational Linguistics, pp. 58–65. Association for Computational Linguistics (2008)Google Scholar
  37. 37.
    Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Information EngineeringUniversità Politecnica delle MarcheAnconaItaly
  2. 2.GfK VereinNürnbergGermany

Personalised recommendations