Multimedia Tools and Applications

, Volume 77, Issue 20, pp 27123–27142 | Cite as

Automated curation of brand-related social media images with deep learning

  • Ruben TousEmail author
  • Mauro Gomez
  • Jonatan Poveda
  • Leonel Cruz
  • Otto Wust
  • Mouna Makni
  • Eduard Ayguadé


This paper presents a work consisting in using deep convolutional neural networks (CNNs) to facilitate the curation of brand-related social media images. The final goal is to facilitate searching and discovering user-generated content (UGC) with potential value for digital marketing tasks. The images are captured in real time and automatically annotated with multiple CNNs. Some of the CNNs perform generic object recognition tasks while others perform what we call visual brand identity recognition. When appropriate, we also apply object detection, usually to discover images containing logos. We report experiments with 5 real brands in which more than 1 million real images were analyzed. In order to speed-up the training of custom CNNs we applied a transfer learning strategy. We examine the impact of different configurations and derive conclusions aiming to pave the way towards systematic and optimized methodologies for automatic UGC curation.


Social media Instagram Twitter User generated content Deep learning Marketing 



This work is partially supported by the Spanish Ministry of Economy and Competitivity under contract TIN2015-65316-P and by the SGR programme (2014-SGR-1051) of the Catalan Government.


  1. 1.
    Bakhshi S, Shamma D A, Kennedy L, Gilbert E (2015) Why we filter our photos and how it impacts engagement. In: Proceedings of the ninth international conference on web and social media, ICWSM 2015. University of Oxford, Oxford, pp 12–21Google Scholar
  2. 2.
    Chute Enterprise ugc. Accessed 6 June (2017)
  3. 3.
    Clarke T, Costall A (2008) The emotional connotations of color: a qualitative investigation. Color Res Appl 33(5):406–410CrossRefGoogle Scholar
  4. 4.
    Clark M, Black H G, Judson K (2017) Brand community integration and satisfaction with social media sites: a comparative study. J Res Interact Market 11 (1):39–55CrossRefGoogle Scholar
  5. 5.
    Curalate Accessed 6 June (2017)
  6. 6.
    Denton E, Weston J, Paluri M, Bourdev L, Fergus R (2015) User conditional hashtag prediction for images. In: Proceedings of the 21th ACM SIGKDD International conference on knowledge discovery and data mining, KDD ’15. ACM, pp 1731–1740Google Scholar
  7. 7.
    Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: Proceedings of the 31th international conference on machine learning, ICML 2014. Beijing, China, 21-26 June 2014, pp 647–655Google Scholar
  8. 8.
    Gao Y, Wang F, Luan H, Chua T (2014) Brand data gathering from live social media streams. In: Proceedings of the international conference on multimedia retrieval, ICMR 2014. Glasgow, United Kingdom - April 01 - 04, 2014, p 169Google Scholar
  9. 9.
    Gao Y, Zhao S, Yang Y, Chua T (2015) Multimedia social event detection in microblog. In: MultiMedia Modeling - 21st International conference, MMM 2015. Sydney, NSW, Australia, January 5-7, 2015, Proceedings, Part I, pp 269–281Google Scholar
  10. 10.
    Gao Y, Zhen Y, Li H, Chua T (2016) Filtering of brand-related microblogs using social-smooth multiview embedding. IEEE Trans Multimed 18(10):2115–2126CrossRefGoogle Scholar
  11. 11.
    Nguyen DT, Alam F, Ofli F, Imran M (2017) Automatic image filtering on social networks using deep learning and perceptual hashing during crises. CoRR arXiv:1704.02602
  12. 12.
    Olapic Earned content platform. Accessed 6 June (2017)
  13. 13.
    Park M, Li H, Kim J (2016) HARRISON: a benchmark on hashtag recommendation for real-world images in social networks. CoRR arXiv:1605.05054
  14. 14.
    Redmon J, Divvala S K, Girshick R B, Farhadi A (2016) You only look once: unified, real-time object detection. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp 779–788Google Scholar
  15. 15.
    Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, Lecun Y (2014) Overfeat: integrated recognition, localization and detection using convolutional networks. In: Proceedings of the international conference on learning representations (ICLR2014), CBLS, April 2014Google Scholar
  16. 16.
    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the 2016 IEEE conference on computer vision and pattern recognition, CVPR 2016. Las Vegas, NV, USA, June 27-30, 2016, pp 2818–2826Google Scholar
  17. 17.
    Tous R, Torres J, Ayguadé E (2015) Multimedia big data computing for in-depth event analysis. In: Proceedings of the 2015 IEEE International conference on multimedia big data (BigMM), April 20-22, 2015, Beijing, China. IEEE, pp 144–147Google Scholar
  18. 18.
    Tous R, Wu̇st O, Gomez M, Poveda J, Elena M, Torres J, Makni M, Ayguadė E (2016) User-generated content curation with deep convolutional neural networks. In: Proceedings of the 2016 IEEE international conference on big data, BigData 2016. Washington DC, USA, December 5-8, 2016, pp 2535– 2540Google Scholar
  19. 19.
    Wang F, Qi S, Gao G, Zhao S, Wang X (2016) Logo information recognition in large-scale social media data. Multimed Syst 22(1):63–73CrossRefGoogle Scholar
  20. 20.
    Zhao S, Yao H, Gao Y, Ji R, Xie W, Jiang X, Chua T (2016) Predicting personalized emotion perceptions of social images. In: Proceedings of the 2016 ACM conference on multimedia conference, MM 2016. Amsterdam, The Netherlands, October 15-19, 2016, pp 1385–1394Google Scholar
  21. 21.
    Zhao S, Yao H, Zhao S, Jiang X, Jiang X (2016) Multi-modal microblog classification via multi-task learning. Multimedia Tools Appl 75(15):8921–8938CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Universitat Politècnica de Catalunya (UPC)BarcelonaSpain
  2. 2.AdsmuraiBarcelonaSpain
  3. 3.Barcelona Supercomputing Center (BSC)BarcelonaSpain

Personalised recommendations