Skip to main content

Sentiment analysis of multimodal twitter data

Multimedia Tools and Applications volume 78, pages 24103–24119 (2019)Cite this article

Abstract

Text-driven sentiment analysis has been widely studied in the past decade, on both random and benchmark textual Twitter datasets. Few pertinent studies have also reported visual analysis of images to predict sentiment, but much of the work has analyzed a single modality data, that is either text or image or GIF video. More recently, as the images, memes and GIFs dominate the social feeds; typographic/infographic visual content has become a non-trivial element of social media. This multimodal text combines both text and image defining a novel visual language which needs to be analyzed as it has the potential to modify, confirm or grade the polarity of the sentiment. We propose a multimodal sentiment analysis model to determine the sentiment polarity and score for any incoming tweet, i.e., textual, image or info-graphic and typographic. Image sentiment scoring is done using SentiBank and SentiStrength scoring for Regions with convolution neural network (R-CNN). Text sentiment scoring is done using a novel context-aware hybrid (lexicon and machine learning) technique. Multimodal sentiment scoring is done by separating text from image using an optical character recognizer and then aggregating the independently processed image and text sentiment scores. High performance accuracy of 91.32% is observed for the random multimodal tweet dataset used to evaluate the proposed model. The research further demonstrates that combining both textual and image features outperforms separate models that rely exclusively on either images or text analysis.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Notes

  1. www.wjh.harvard.edu/~inquirer

References

  1. Aftab, K., Hartley, R. and Trumpf, J., 2015. Generalized weiszfeld algorithms for lq optimization

  2. Bird S, Loper E (2004) NLTK: the natural language toolkit. In: Proceedings of the ACL 2004 on Interactive poster and demonstration sessions (p. 31). Association for Computational Linguistics

  3. Bollen J, Mao H, Zeng X-J (2011) Twitter mood predicts the stock market. J Comput Sci 2(2011):1–8

    Article  Google Scholar 

  4. Borth D, Chen T, Ji R, Chang SF (2013) Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the 21st ACM international conference on Multimedia (pp. 459–460). ACM

  5. Borth D, Ji R, Chen T, Breuel T, Chang S-F (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM international conference on Multimedia, pp. 223–232. ACM

  6. Cai G, Xia B (2015) Convolutional neural networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Computing (pp. 159–167). Springer, Cham

  7. Chen T, Salah Eldeen HM, He X, Kan MY, Lu D (2017) VELDA: Relating an Image Tweet's Text and Images. In AAAI 2015 Jan 25 (pp. 30–36)

  8. Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach. In: European Conference on Computer Vision (pp. 288–301). Springer, Berlin

  9. Dave K, Lawrence S, Pennock DM (2003) Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. Proceedings of the 12th international conference on World Wide Web. ACM. 519–528

  10. Esuli A, Sebastiani F (2007) SentiWordNet: a high-coverage lexical resource for opinion mining. Evaluation. 17:1–26

    Google Scholar 

  11. Gajarla V, Gupta A (2015) Emotion detection and sentiment analysis of images. Georgia Institute of Technology, Atlanta

    Google Scholar 

  12. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587

  13. Go A, Bhayani R, Huang L (2009) Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1(12)

  14. Golder SA, Macy MW (2011) Diurnal and seasonal mood vary with work, sleep, and day length across diverse cultures. Science 333(6051):1878–1881

    Article  Google Scholar 

  15. Hao T, Rusanov A, Boland MR, Weng C (2014) Clustering clinical trials with similar eligibility criteria features. J Biomed Inform 52:112–120

    Article  Google Scholar 

  16. Hare JS, Samangooei S, Dupplaw DP, Lewis PH (2013) Twitter's visual pulse. In: Proceedings of the 3rd ACM conference on International conference on multimedia retrieval (pp. 297–298). ACM

  17. Hodosh M, Young P, Hockenmaier J (2013) Framing image description as a ranking task: Data, models and evaluation metrics. J Artif Intell Res 47:853–899

    MathSciNet  MATH  Article  Google Scholar 

  18. Jia J, Wu S, Wang X, Hu P, Cai L, Tang J (2012) Can we understand van gogh's mood?: learning to infer affects from images in social networks. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 857–860

  19. Katsurai M, Satoh SI (2016) Image sentiment analysis using latent correlations among visual, textual, and sentiment views. In: Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on (pp. 2837–2841). IEEE

  20. Kouloumpis E, Wilson T, Moore JD (2011) Twitter sentiment analysis: The good the bad and the omg! ICWSM 11(538–541):164

    Google Scholar 

  21. Kumar A, Dogra P, Dabas V (2015) Emotion analysis of Twitter using opinion mining. In: Contemporary Computing (IC3), 2015 Eighth International Conference on, IEEE, pp. 285–290

  22. Kumar A, Jaiswal A (2017) Empirical study of twitter and tumblr for sentiment analysis using soft computing techniques. Proceedings of the World Congress on Engineering and Computer Science 1:1–5

    Google Scholar 

  23. Kumar A, Jaiswal A (2017) Image sentiment analysis using convolutional neural network. In: International Conference on Intelligent Systems Design and Applications (pp. 464–473). Springer, Cham

  24. Kumar A, Jaiswal A (2019) Systematic literature review of sentiment analysis on Twitter using soft computing techniques. Concurrency and Computation: Practice and Experience: e5107

  25. Kumar A, Jaiswal A, Garg S, Verma S, Kumar S (2019) Sentiment Analysis Using Cuckoo Search for Optimized Feature Selection on Kaggle Tweets. International Journal of Information Retrieval Research (IJIRR) 9(1):1–15

    Article  Google Scholar 

  26. Kumar, Akshi, Renu Khorwal, and Shweta Chaudhary (2016) A survey on sentiment analysis using swarm intelligence." Indian Journal of Science and Technology 9, no. 39

  27. Kumar A, Sebastian TM (2012) Sentiment analysis: A perspective on its past, present and future. International Journal of Intelligent Systems and Applications 4(10):1–4

    Article  Google Scholar 

  28. Kumar A, Sebastian TM (2012) Sentiment analysis on Twitter. IJCSI International Journal of Computer Science Issues 9(3):372–378

    Google Scholar 

  29. Kumar A, Sebastian TM (2012) Machine learning assisted sentiment analysis. In: Proceedings of International Conference on Computer Science & Engineering (ICCSE’2012), pp. 123–130

  30. Kumar A, Sharma A (2017) Systematic literature review on opinion mining of big data for government intelligence. Webology 14(2)

  31. Kumar A, Sharma A (2018) Socio-sentic framework for sustainable agricultural governance. Sustainable Computing: Informatics and Systems

  32. Kumar A, Sharma A (2019) Opinion mining of Saubhagya Yojna for Digital India. In: International Conference on Innovative Computing and Communications, pp. 375–386. Springer, Singapore

  33. Li B, Feng S, Xiong W, Hu W (2012) Scaring or pleasing: exploit emotional impact of an image. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 1365–1366

  34. Li B, Xiong W, Hu W, Ding X (2012) Context-aware affective images classification based on bilayer sparse representation. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 721–724

  35. Lu X, Suryanarayan P, Adams Jr RB, Li J, Newman MG, Wang JZ (2012) On shape and the computability of emotions. In Proceedings of the 20th ACM international conference on Multimedia (pp. 229–238). ACM

  36. Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp. 83-92

  37. Mandhyani J, Khatri L, Ludhrani V, Nagdev R, Sahu S (2017) Image Sentiment Analysis. International Journal of Engineering Science 4566

  38. Marchesotti L, Perronnin F, Larlus D, Csurka G (2011) Assessing the aesthetic quality of photographs using generic image descriptors. In: Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, pp. 1784–1791

  39. Neethu MS, Rajasree R (2013) Sentiment analysis in twitter using machine learning techniques. In: Computing, Communications and Networking Technologies (ICCCNT), 2013 Fourth International Conference on, (pp. 1–5). IEEE

  40. Pak A, Paroubek P (2010) Twitter as a corpus for sentiment analysis and opinion mining. In: LREC, vol. 10, pp. 1320–1326

  41. Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59

    Article  Google Scholar 

  42. Porter MF (1980) An algorithm for suffix stripping. Program. 14(3):130–137

    Article  Google Scholar 

  43. Saif H, Fernandez M, He Y, Alani H (2013) Evaluation datasets for Twitter sentiment analysis: a survey and a new dataset, the STS-Gold

  44. Saif H, Fernandez M, He Y, Alani H (2014) Senticircles for contextual and conceptual semantic sentiment analysis of twitter. In: European Semantic Web Conference (pp. 83–98). Springer, Cham

  45. Saif H, He Y, Fernandez M, Alani H (2016) Contextual semantics for sentiment analysis of Twitter. Inf Process Manag 52(1):5–19

    Article  Google Scholar 

  46. Siersdorfer S, Minack E, Deng F, Hare J (2010) Analyzing and predicting sentiment of images on the social web. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp. 715–718

  47. Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14

    Article  Google Scholar 

  48. Vonikakis V, Winkler S (2012) Emotion-based sequence of family photos. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 1371–1372

  49. Wang Y, Wang S, Tang J, Liu H, Li B (2015) Unsupervised sentiment analysis for social media images. In: IJCAI. pp. 2378–2379

  50. Yang Y, Cui P, Zhu W, Zhao HV, Shi Y, Yang S (2014) Emotionally representative image discovery for social events. In: Proceedings of International Conference on Multimedia Retrieval. ACM, p. 177

  51. Yanulevskaya V, Uijlings J, Bruni E, Sartori A, Zamboni E, Bacci F, Melcher D, Sebe N (2012) In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 349–358

  52. Yanulevskaya V, van Gemert JC, Roth K, Herbold AK, Sebe N, Geusebroek JM (2008) Emotional valence categorization using holistic image features. In: ICIP. pp. 101–104

  53. You Q, Luo J (2013) Towards social imagematics: sentiment analysis in social multimedia. In: Proceedings of the Thirteenth International Workshop on Multimedia Data Mining, ACM, p. 3

  54. Zhao S, Gao Y, Jiang X, Yao H, Chua TS, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the 22nd ACM international conference on Multimedia (pp. 47–56). ACM

  55. Zhao S, Yao H, Wang F, Jiang X, Zhang W (2014) Emotion based image musicalization. In: Multimedia and Expo Workshops (ICMEW), 2014 IEEE International Conference on (pp. 1–6). IEEE

  56. Zhao S, Yao H, Yang Y, Zhang Y (2014) Affective image retrieval via multi-graph learning. In: Proceedings of the 22nd ACM international conference on Multimedia (pp. 1025–1028). ACM

  57. Zhou X, Yao C, Wen H, Wang Y, Zhou S, He W, Liang J (2017) EAST: an efficient and accurate scene text detector. In Proc. CVPR, pp. 2642–2651

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Delhi Technological University, Delhi, India

    Akshi Kumar & Geetanjali Garg

Authors
  1. Akshi Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geetanjali Garg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Akshi Kumar.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kumar, A., Garg, G. Sentiment analysis of multimodal twitter data. Multimed Tools Appl 78, 24103–24119 (2019). https://doi.org/10.1007/s11042-019-7390-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7390-1

Keywords

  • Multimodal text
  • Twitter
  • Sentiment
  • Context-aware
  • Optical character recognition