Sentiment analysis of multimodal twitter data

  • Akshi KumarEmail author
  • Geetanjali Garg


Text-driven sentiment analysis has been widely studied in the past decade, on both random and benchmark textual Twitter datasets. Few pertinent studies have also reported visual analysis of images to predict sentiment, but much of the work has analyzed a single modality data, that is either text or image or GIF video. More recently, as the images, memes and GIFs dominate the social feeds; typographic/infographic visual content has become a non-trivial element of social media. This multimodal text combines both text and image defining a novel visual language which needs to be analyzed as it has the potential to modify, confirm or grade the polarity of the sentiment. We propose a multimodal sentiment analysis model to determine the sentiment polarity and score for any incoming tweet, i.e., textual, image or info-graphic and typographic. Image sentiment scoring is done using SentiBank and SentiStrength scoring for Regions with convolution neural network (R-CNN). Text sentiment scoring is done using a novel context-aware hybrid (lexicon and machine learning) technique. Multimodal sentiment scoring is done by separating text from image using an optical character recognizer and then aggregating the independently processed image and text sentiment scores. High performance accuracy of 91.32% is observed for the random multimodal tweet dataset used to evaluate the proposed model. The research further demonstrates that combining both textual and image features outperforms separate models that rely exclusively on either images or text analysis.


Multimodal text Twitter Sentiment Context-aware Optical character recognition 



  1. 1.
    Aftab, K., Hartley, R. and Trumpf, J., 2015. Generalized weiszfeld algorithms for lq optimizationGoogle Scholar
  2. 2.
    Bird S, Loper E (2004) NLTK: the natural language toolkit. In: Proceedings of the ACL 2004 on Interactive poster and demonstration sessions (p. 31). Association for Computational LinguisticsGoogle Scholar
  3. 3.
    Bollen J, Mao H, Zeng X-J (2011) Twitter mood predicts the stock market. J Comput Sci 2(2011):1–8CrossRefGoogle Scholar
  4. 4.
    Borth D, Chen T, Ji R, Chang SF (2013) Sentibank: large-scale ontology and classifiers for detecting sentiment and emotions in visual content. In: Proceedings of the 21st ACM international conference on Multimedia (pp. 459–460). ACMGoogle Scholar
  5. 5.
    Borth D, Ji R, Chen T, Breuel T, Chang S-F (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: Proceedings of the 21st ACM international conference on Multimedia, pp. 223–232. ACMGoogle Scholar
  6. 6.
    Cai G, Xia B (2015) Convolutional neural networks for multimedia sentiment analysis. In Natural Language Processing and Chinese Computing (pp. 159–167). Springer, ChamGoogle Scholar
  7. 7.
    Chen T, Salah Eldeen HM, He X, Kan MY, Lu D (2017) VELDA: Relating an Image Tweet's Text and Images. In AAAI 2015 Jan 25 (pp. 30–36)Google Scholar
  8. 8.
    Datta R, Joshi D, Li J, Wang JZ (2006) Studying aesthetics in photographic images using a computational approach. In: European Conference on Computer Vision (pp. 288–301). Springer, BerlinGoogle Scholar
  9. 9.
    Dave K, Lawrence S, Pennock DM (2003) Mining the peanut gallery: Opinion extraction and semantic classification of product reviews. Proceedings of the 12th international conference on World Wide Web. ACM. 519–528Google Scholar
  10. 10.
    Esuli A, Sebastiani F (2007) SentiWordNet: a high-coverage lexical resource for opinion mining. Evaluation. 17:1–26Google Scholar
  11. 11.
    Gajarla V, Gupta A (2015) Emotion detection and sentiment analysis of images. Georgia Institute of Technology, AtlantaGoogle Scholar
  12. 12.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587Google Scholar
  13. 13.
    Go A, Bhayani R, Huang L (2009) Twitter sentiment classification using distant supervision. CS224N Project Report, Stanford 1(12)Google Scholar
  14. 14.
    Golder SA, Macy MW (2011) Diurnal and seasonal mood vary with work, sleep, and day length across diverse cultures. Science 333(6051):1878–1881CrossRefGoogle Scholar
  15. 15.
    Hao T, Rusanov A, Boland MR, Weng C (2014) Clustering clinical trials with similar eligibility criteria features. J Biomed Inform 52:112–120CrossRefGoogle Scholar
  16. 16.
    Hare JS, Samangooei S, Dupplaw DP, Lewis PH (2013) Twitter's visual pulse. In: Proceedings of the 3rd ACM conference on International conference on multimedia retrieval (pp. 297–298). ACMGoogle Scholar
  17. 17.
    Hodosh M, Young P, Hockenmaier J (2013) Framing image description as a ranking task: Data, models and evaluation metrics. J Artif Intell Res 47:853–899MathSciNetCrossRefGoogle Scholar
  18. 18.
    Jia J, Wu S, Wang X, Hu P, Cai L, Tang J (2012) Can we understand van gogh's mood?: learning to infer affects from images in social networks. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 857–860Google Scholar
  19. 19.
    Katsurai M, Satoh SI (2016) Image sentiment analysis using latent correlations among visual, textual, and sentiment views. In: Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on (pp. 2837–2841). IEEEGoogle Scholar
  20. 20.
    Kouloumpis E, Wilson T, Moore JD (2011) Twitter sentiment analysis: The good the bad and the omg! ICWSM 11(538–541):164Google Scholar
  21. 21.
    Kumar A, Dogra P, Dabas V (2015) Emotion analysis of Twitter using opinion mining. In: Contemporary Computing (IC3), 2015 Eighth International Conference on, IEEE, pp. 285–290Google Scholar
  22. 22.
    Kumar A, Jaiswal A (2017) Empirical study of twitter and tumblr for sentiment analysis using soft computing techniques. Proceedings of the World Congress on Engineering and Computer Science 1:1–5Google Scholar
  23. 23.
    Kumar A, Jaiswal A (2017) Image sentiment analysis using convolutional neural network. In: International Conference on Intelligent Systems Design and Applications (pp. 464–473). Springer, ChamGoogle Scholar
  24. 24.
    Kumar A, Jaiswal A (2019) Systematic literature review of sentiment analysis on Twitter using soft computing techniques. Concurrency and Computation: Practice and Experience: e5107Google Scholar
  25. 25.
    Kumar A, Jaiswal A, Garg S, Verma S, Kumar S (2019) Sentiment Analysis Using Cuckoo Search for Optimized Feature Selection on Kaggle Tweets. International Journal of Information Retrieval Research (IJIRR) 9(1):1–15CrossRefGoogle Scholar
  26. 26.
    Kumar, Akshi, Renu Khorwal, and Shweta Chaudhary (2016) A survey on sentiment analysis using swarm intelligence." Indian Journal of Science and Technology 9, no. 39Google Scholar
  27. 27.
    Kumar A, Sebastian TM (2012) Sentiment analysis: A perspective on its past, present and future. International Journal of Intelligent Systems and Applications 4(10):1–4CrossRefGoogle Scholar
  28. 28.
    Kumar A, Sebastian TM (2012) Sentiment analysis on Twitter. IJCSI International Journal of Computer Science Issues 9(3):372–378Google Scholar
  29. 29.
    Kumar A, Sebastian TM (2012) Machine learning assisted sentiment analysis. In: Proceedings of International Conference on Computer Science & Engineering (ICCSE’2012), pp. 123–130Google Scholar
  30. 30.
    Kumar A, Sharma A (2017) Systematic literature review on opinion mining of big data for government intelligence. Webology 14(2)Google Scholar
  31. 31.
    Kumar A, Sharma A (2018) Socio-sentic framework for sustainable agricultural governance. Sustainable Computing: Informatics and SystemsGoogle Scholar
  32. 32.
    Kumar A, Sharma A (2019) Opinion mining of Saubhagya Yojna for Digital India. In: International Conference on Innovative Computing and Communications, pp. 375–386. Springer, SingaporeGoogle Scholar
  33. 33.
    Li B, Feng S, Xiong W, Hu W (2012) Scaring or pleasing: exploit emotional impact of an image. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 1365–1366Google Scholar
  34. 34.
    Li B, Xiong W, Hu W, Ding X (2012) Context-aware affective images classification based on bilayer sparse representation. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 721–724Google Scholar
  35. 35.
    Lu X, Suryanarayan P, Adams Jr RB, Li J, Newman MG, Wang JZ (2012) On shape and the computability of emotions. In Proceedings of the 20th ACM international conference on Multimedia (pp. 229–238). ACMGoogle Scholar
  36. 36.
    Machajdik J, Hanbury A (2010) Affective image classification using features inspired by psychology and art theory. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp. 83-92Google Scholar
  37. 37.
    Mandhyani J, Khatri L, Ludhrani V, Nagdev R, Sahu S (2017) Image Sentiment Analysis. International Journal of Engineering Science 4566Google Scholar
  38. 38.
    Marchesotti L, Perronnin F, Larlus D, Csurka G (2011) Assessing the aesthetic quality of photographs using generic image descriptors. In: Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, pp. 1784–1791Google Scholar
  39. 39.
    Neethu MS, Rajasree R (2013) Sentiment analysis in twitter using machine learning techniques. In: Computing, Communications and Networking Technologies (ICCCNT), 2013 Fourth International Conference on, (pp. 1–5). IEEEGoogle Scholar
  40. 40.
    Pak A, Paroubek P (2010) Twitter as a corpus for sentiment analysis and opinion mining. In: LREC, vol. 10, pp. 1320–1326Google Scholar
  41. 41.
    Poria S, Cambria E, Howard N, Huang GB, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing 174:50–59CrossRefGoogle Scholar
  42. 42.
    Porter MF (1980) An algorithm for suffix stripping. Program. 14(3):130–137CrossRefGoogle Scholar
  43. 43.
    Saif H, Fernandez M, He Y, Alani H (2013) Evaluation datasets for Twitter sentiment analysis: a survey and a new dataset, the STS-GoldGoogle Scholar
  44. 44.
    Saif H, Fernandez M, He Y, Alani H (2014) Senticircles for contextual and conceptual semantic sentiment analysis of twitter. In: European Semantic Web Conference (pp. 83–98). Springer, ChamGoogle Scholar
  45. 45.
    Saif H, He Y, Fernandez M, Alani H (2016) Contextual semantics for sentiment analysis of Twitter. Inf Process Manag 52(1):5–19CrossRefGoogle Scholar
  46. 46.
    Siersdorfer S, Minack E, Deng F, Hare J (2010) Analyzing and predicting sentiment of images on the social web. In: Proceedings of the 18th ACM international conference on Multimedia. ACM, pp. 715–718Google Scholar
  47. 47.
    Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14CrossRefGoogle Scholar
  48. 48.
    Vonikakis V, Winkler S (2012) Emotion-based sequence of family photos. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 1371–1372Google Scholar
  49. 49.
    Wang Y, Wang S, Tang J, Liu H, Li B (2015) Unsupervised sentiment analysis for social media images. In: IJCAI. pp. 2378–2379Google Scholar
  50. 50.
    Yang Y, Cui P, Zhu W, Zhao HV, Shi Y, Yang S (2014) Emotionally representative image discovery for social events. In: Proceedings of International Conference on Multimedia Retrieval. ACM, p. 177Google Scholar
  51. 51.
    Yanulevskaya V, Uijlings J, Bruni E, Sartori A, Zamboni E, Bacci F, Melcher D, Sebe N (2012) In the eye of the beholder: employing statistical analysis and eye tracking for analyzing abstract paintings. In: Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 349–358Google Scholar
  52. 52.
    Yanulevskaya V, van Gemert JC, Roth K, Herbold AK, Sebe N, Geusebroek JM (2008) Emotional valence categorization using holistic image features. In: ICIP. pp. 101–104Google Scholar
  53. 53.
    You Q, Luo J (2013) Towards social imagematics: sentiment analysis in social multimedia. In: Proceedings of the Thirteenth International Workshop on Multimedia Data Mining, ACM, p. 3Google Scholar
  54. 54.
    Zhao S, Gao Y, Jiang X, Yao H, Chua TS, Sun X (2014) Exploring principles-of-art features for image emotion recognition. In: Proceedings of the 22nd ACM international conference on Multimedia (pp. 47–56). ACMGoogle Scholar
  55. 55.
    Zhao S, Yao H, Wang F, Jiang X, Zhang W (2014) Emotion based image musicalization. In: Multimedia and Expo Workshops (ICMEW), 2014 IEEE International Conference on (pp. 1–6). IEEEGoogle Scholar
  56. 56.
    Zhao S, Yao H, Yang Y, Zhang Y (2014) Affective image retrieval via multi-graph learning. In: Proceedings of the 22nd ACM international conference on Multimedia (pp. 1025–1028). ACMGoogle Scholar
  57. 57.
    Zhou X, Yao C, Wen H, Wang Y, Zhou S, He W, Liang J (2017) EAST: an efficient and accurate scene text detector. In Proc. CVPR, pp. 2642–2651Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer Science & EngineeringDelhi Technological UniversityDelhiIndia

Personalised recommendations