Skip to main content
SpringerLink
Log in

Visual sentiment analysis using data-augmented deep transfer learning techniques

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

There has been a growing trend among users of social media platforms to express their emotions using visual content. Visual sentiment analysis is the process of understanding the emotional polarity of images or videos and is still considered a challenging problem in artificial intelligence. Most of the existing models are based on robust machine learning or deep learning techniques. The idea of using deep transfer learning techniques for visual sentiment analysis is fairly new. In this paper, we propose a new approach using data-augmented-transfer learning architecture consisting of a pre-trained VGG16 model that is fine-tuned using SVM with augmented training data. For fine-tuning and evaluation, we initially use two Twitter image datasets. We further validated the proposed model on a third dataset. The commonly used geometric augmentation methods such as rotation, zoom range, width shift, height shift, shear range and horizontal flip were are used. We compare our proposed VGG16-SVM model with 3 other state-of-the-art deep models commonly used for transfer learning and 4 machine learning models (besides SVM) used for fine-tuning. The results show that VGG16-SVM produces the overall best accuracy (94%) and recall (96%) among all transfer learning and machine learning pairs. We also show that our proposed model outperforms all previous studies that use the same dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Availability of data and material

No dataset generated in this study.

References

  1. Ahsan U, De Choudhury M, Essa I (2017) Towards using visual attributes to infer image sentiment of social events. IEEE, pp 1372–1379

  2. Ain QT et al (2017) Sentiment analysis using deep learning techniques: a review. Int J Adv Comput Sci Appl 8(6):424

    Google Scholar 

  3. Borth D, Chen T, Ji R, Chang S-F (2013) Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. pp 459–460

  4. Borth D, Ji R, Chen T, Breuel T, Chang S-F (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. pp 223–232

  5. Cai G, Lv G (2017) Heterogeneous transfer with deep latent correlation for sentiment analysis, vol 2. IEEE, pp 252–256

  6. Cambria E, Livingstone A, Hussain A (2012) In: The hourglass of emotions. Springer, pp 144–157

  7. Campos V, Jou B, Giro-i Nieto X (2017) From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction. Image Vis Comput 65:15–22

    Article  Google Scholar 

  8. Chandrasekaran G, Antoanela N, Andrei G, Monica C, Hemanth J (2022) Visual sentiment analysis using deep learning models with social media data. Appl Sci 12(3):1030

    Article  Google Scholar 

  9. Chen S, Yang J, Feng J, Gu Y (2017) Image sentiment analysis using supervised collective matrix factorization. IEEE, pp 1033–1038

  10. Fan S et al (2017) The role of visual attention in sentiment prediction. pp 217–225

  11. Fengjiao W, Aono M (2018) Visual sentiment prediction by merging hand-craft and CNN features. IEEE, pp 66–71

  12. Giancristofaro GT, Panangadan A (2016) Predicting sentiment toward transportation in social media using visual and textual features. IEEE, pp 2113–2118

  13. Göring S, Brand K, Raake A (2018) Extended features using machine learning techniques for photo liking prediction. IEEE, pp 1–6

  14. Hassan SZ et al (2020) Visual sentiment analysis from disaster images in social media. Preprint at http://arxiv.org/abs/2009.03051

  15. Huang C-C, Wu Y-L, Tang C-Y (2019) Human face sentiment classification using synthetic sentiment images with deep convolutional neural networks. IEEE, pp 1–5

  16. Islam J, Zhang Y (2016) Visual sentiment analysis for social images using transfer learning approach. IEEE, pp 124–130

  17. Ji R, Cao D, Zhou Y, Chen F (2016) Survey of visual sentiment prediction for social media analysis. Front Comp Sci 10(4):602–611

    Article  Google Scholar 

  18. Li Z et al (2021) Visual sentiment analysis based on image caption and adjective–noun–pair description. Soft Computing 1–13

  19. Li Z, Fan Y, Jiang B, Lei T, Liu W (2019) A survey on sentiment analysis and opinion mining for social multimedia. Multimed Tools Appl 78(6):6939–6967

    Article  Google Scholar 

  20. Liu W, Qiu J-L, Zheng W-L, Lu B-L (2019) Multimodal emotion recognition using deep canonical correlation analysis. Preprint at http://arxiv.org/abs/1908.05349

  21. McDuff D, El Kaliouby R, Cohn JF, Picard RW (2014) Predicting ad liking and purchase intent: large-scale analysis of facial responses to ads. IEEE Trans Affect Comput 6(3):223–235

    Article  Google Scholar 

  22. Mittal N, Sharma D, Joshi ML (2018) Image sentiment analysis using deep learning. IEEE, pp 684–687

  23. Poria S et al (2017) Context-dependent sentiment analysis in user-generated videos. pp 873–883

  24. Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230

    Article  Google Scholar 

  25. Siersdorfer S, Minack E, Deng F, Hare J (2010) Analyzing and predicting sentiment of images on the social web. pp 715–718

  26. Sun M, Yang J, Wang K, Shen H (2016) Discovering affective regions in deep convolutional neural networks for visual sentiment prediction. IEEE, pp 1–6

  27. Vadicamo L et al (2017) Cross-media learning for image sentiment analysis in the wild. pp 308–317

  28. Wang Y, Li B (2015) Sentiment analysis for social media images. IEEE, pp 1584–1591

  29. Wang C, Yang J, Xie L, Yuan J (2019) Kervolutional neural networks. pp 31–40

  30. Wu L, Zhang H, Deng S, Shi G, Liu X (2021) Discovering sentimental interaction via graph convolutional network for visual sentiment prediction. Appl Sci 11(4):1404

    Article  Google Scholar 

  31. Wu L, Zhang H, Shi G, Deng S (2021) Weakly supervised interaction discovery network for image sentiment analysis. Tech. Rep, EasyChair

    Google Scholar 

  32. Yadav A, Vishwakarma DK (2020) Sentiment analysis using deep learning architectures: a review. Artif Intell Rev 53(6):4335–4385

    Article  Google Scholar 

  33. Yang J et al (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimedia 20(9):2513–2525

    Article  Google Scholar 

  34. Yang L, Song Q, Wang Z, Jiang M (2019) Parsing r-CNN for instance-level human analysis. pp 364–373

  35. Yazdavar AH et al (2020) Multimodal mental health analysis in social media. PLoS ONE 15(4):e0226248

    Article  Google Scholar 

  36. You Q, Jin H, Luo J (2017) Visual sentiment analysis by attending on local image regions

  37. You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks

  38. You Q, Luo J, Jin H, Yang J (2015) Robust image sentiment analysis using progressively trained and domain transferred deep networks

  39. Yuan J, Mcdonough S, You Q, Luo J (2013) Sentribute: image sentiment analysis from a mid-level perspective. pp 1–8

  40. Zhang K, Zhu Y, Zhang W, Zhu Y (2021) Cross-modal image sentiment analysis via deep correlation of textual semantic. Knowl-Based Syst 216:106803

    Article  Google Scholar 

  41. Zisad SN, Chowdhury E, Hossain MS, Islam RU, Andersson K (2021) An integrated deep learning and belief rule-based expert system for visual sentiment analysis under uncertainty. Algorithms 14(7):213

    Article  Google Scholar 

  42. Zisad SN, Chowdhury E, Hossain MS, Islam RU, Andersson K (2021) An integrated deep learning and belief rule-based expert system for visual sentiment analysis under uncertainty. Algorithms 14(7):213

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Technology, Anhui Vocational College of Defense Technology, Lu’an, China

    Zhiguo Jiang

  2. Department of Computer Science, National University of Computer and Emerging Sciences, Lahore, Pakistan

    Waneeza Zaheer, Aamir Wali & S. A. M. Gilani

Authors
  1. Zhiguo Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Waneeza Zaheer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aamir Wali
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. S. A. M. Gilani
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All author contributed equally.

Corresponding author

Correspondence to Aamir Wali.

Ethics declarations

Compliance with ethical standards

This statement is to certify that the author list is correct. The Authors also confirm that this research has not been published previously and that it is not under consideration for publication elsewhere. On behalf of all Co-Authors, the Corresponding Author shall bear full responsibility for the submission. There is no conflict of interest. This research did not involve any human participants and/or animals.

Conflict of interest

The authors declare that they have no relevant financial or non-financial interests to disclose. There is no personal relationship that could influence the work reported in this paper. No funding was received for conducting this study. The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Zaheer, W., Wali, A. et al. Visual sentiment analysis using data-augmented deep transfer learning techniques. Multimed Tools Appl 83, 17233–17249 (2024). https://doi.org/10.1007/s11042-023-16262-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16262-4

Keywords

Search

Navigation