Skip to main content
SpringerLink
Log in

Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Social networking platforms have witnessed tremendous growth of textual, visual, audio, and mix-mode contents for expressing the views or opinions. Henceforth, Sentiment Analysis (SA) and Emotion Detection (ED) of various social networking posts, blogs, and conversation are very useful and informative for mining the right opinions on different issues, entities, or aspects. The various statistical and probabilistic models based on lexical and machine learning approaches have been employed for these tasks. The emphasis was given to the improvement in the contemporary tools, techniques, models, and approaches, are reflected in majority of the literature. With the recent developments in deep neural networks, various deep learning models are being heavily experimented for the accuracy enhancement in the aforementioned tasks. Recurrent Neural Network (RNN) and its architectural variants such as Long Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) comprise an important category of deep neural networks, basically adapted for features extraction in the temporal and sequential inputs. Input to SA and related tasks may be visual, textual, audio, or any combination of these, consisting of an inherent sequentially, we critically investigate the role of sequential deep neural networks in sentiment analysis of multimodal data. Specifically, we present an extensive review over the applicability, challenges, issues, and approaches for textual, visual, and multimodal SA using RNN and its architectural variants.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Abburi H, Gangashetty SV, Shrivastava M, Mamidi R Audio and Text based Multimodal Sentiment Analysis using Features Extracted from Selective Regions and Deep Neural Networks (Doctoral dissertation, International Institute of Information Technology Hyderabad).

  2. Alayba AM, Palade V, England M, Iqbal R (2018) A combined cnn and lstm model for arabic sentiment analysis. In: International cross-domain conference for machine learning and knowledge extraction. Springer, Cham, pp 179–191

    Google Scholar 

  3. Al-Moslmi T, Omar N, Abdullah S, Albared M (2017) Approaches to cross-domain sentiment analysis: a systematic literature review. IEEE Access 5:16173–16192

    Google Scholar 

  4. Araque O, Barbado R, Sánchez-Rada JF, Iglesias CA (2017) Applying recurrent neural networks to sentiment analysis of spanish tweets Proc TASS 1896.

  5. Atzeni M, Recupero DR (2019) Multi-domain sentiment analysis with mimicked and polarized word embeddings for humanrobot interaction. Futur Gener Comput Syst 110:984–999

    Google Scholar 

  6. Baecchi C, Uricchio T, Bertini M, Del Bimbo A (2016) A multimodal feature learning approach for sentiment analysis of social network multimedia. Multimed Tools Appl 75(5):2507–2525

    Google Scholar 

  7. Bai X (2011) Predicting consumer sentiments from online text. Decis Support Syst 50(4):732–742

    Google Scholar 

  8. Balamurali AR, Joshi A, Bhattacharyya P (2011) Robust sense-based sentiment classification. In: Proceedings of the 2nd workshop on computational approaches to subjectivity and sentiment analysis, pp 132-138.

  9. Beel J, Gipp B, Langer S, Breitinger C (2016) Paper recommender systems: a literature survey. Int J Digit Libr 17(4):305–338

    Google Scholar 

  10. Bengio Y, Courville AC, Vincent P (2012) Unsupervised feature learning and deep learning: a review and new perspectives. CoRR, abs/1206.5538, 1(2012).

  11. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828

    Google Scholar 

  12. Borth D, Ji R, Chen T, Breuel T, Chang SF (2013) Large-scale visual sentiment ontology and detectors using adjective noun pairs. In: proceedings of the 21st ACM international conference on multimedia, pp 223-232.

  13. Britz D (2015) Recurrent neural networks tutorial, part 1 – introduction to RNNs. http://www.wildml.Com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns.

  14. Budiharto W, Meiliana M (2018) Prediction and analysis of Indonesia presidential election from twitter using sentiment analysis. J Big Data 5(1):51

    Google Scholar 

  15. Bullas J (2014) (22) social media facts and statistics you should know in 2014. Jeffbullas.com. https://www.jeffbullas.com/20-social-media-factsand-statistics-you-should-know-in-2014/. Accessed 13 Apr 2020

  16. Busso C, Bulut M, Lee CC, Kazemzadeh A, Mower E, Kim S, Chang JN, Lee S, Narayanan SS (2008) IEMOCAP: interactive emotional dyadic motion capture database. Lang Resour Eval 42(4):335–359

    Google Scholar 

  17. Caschera MC, Ferri F, Grifoni P (2016) Sentiment analysis from textual to multimodal features in digital environments. In: proceedings of the 8th international conference on Management of Digital EcoSystems, pp 137-144.

  18. Chaturvedi I, Satapathy R, Cavallari S, Cambria E (2019) Fuzzy commonsense reasoning for multimodal sentiment analysis. Pattern Recogn Lett 125:264–270

    Google Scholar 

  19. Che Z, Purushotham S, Cho K, Sontag D, Liu Y (2018) Recurrent neural networks for multivariate time series with missing values. Sci Rep 8(1):1–12

    Google Scholar 

  20. Chen PC, Pavlidis T (1979) Segmentation by texture using a co-occurrence matrix and a split-and-merge algorithm. Comput Graphics Image Process 10(2):172–182

    Google Scholar 

  21. Chen T, Yu FX, Chen J, Cui Y, Chen YY, Chang SF (2014) Object-based visual sentiment concept analysis and application. In: Proceedings of the 22nd ACM international conference on multimedia, pp 367-376.

  22. Chen R, Zhou Y, Zhang L, Duan X (2019) Word-level sentiment analysis with reinforcement learning. In: IOP conference series: materials science and engineering, pp 490(6).

  23. Cheng J, Li P, Ding Z, Zhang S, Wang H (2016) Sentiment classification of chinese microblogging texts with global RNN. In: 2016 IEEE first international conference on data science in cyberspace – DSC’16, pp 653-657.

  24. Cheng J, Zhang X, Li P, Zhang S, Ding Z, Wang H (2016) Exploring sentiment parsing of microblogging texts for opinion polling on chinese public figures. Appl Intell 45(2):429–442

    Google Scholar 

  25. Chikersal P, Poria S, Cambria E (2015) SeNTU: sentiment analysis of tweets by combining a rule-based classifier with supervised learning. In: Proceedings of the 9th international workshop on semantic evaluation- SemEval’15, pp 647-651.

  26. Choi M, Tani J (2017) Predictive coding for dynamic vision: development of functional hierarchy in a multiple spatio-temporal scales RNN model. In: 2017 international joint conference on neural networks – IJCNN’17, pp 657-664.

  27. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.

  28. Dai AM, Le QV (2015) Semi-supervised sequence learning. In: Advances in neural information processing systems, pp. 3079–3087.

  29. Dang NC, Moreno-García MN, De la Prieta F (2020) Sentiment analysis based on deep learning: a comparative study. Electronics 9(3):483

    Google Scholar 

  30. Day MY, Lin YD (2017) Deep learning for sentiment analysis on google play consumer review. In: 2017 IEEE international conference on information reuse and integration – IRI’17, pp 382-388.

  31. Deng L (2014) A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans Signal Inf Process 3.

  32. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248-255.

  33. Dermouche M, Velcin J, Khouas L, Loudcher S (2014) A joint model for topic-sentiment evolution over time. In: 2014 IEEE international conference on data mining, pp 773-778.

  34. Devaraj M, Piryani R, Singh VK (2016) Lexicon ensemble and lexicon pooling for sentiment polarity detection. IETE Tech Rev 33(3):332–340

    Google Scholar 

  35. Devika MD, Sunitha C, Ganesh A (2016) Sentiment analysis: a comparative study on different approaches. Procedia Comput Sci 87:44–49

    Google Scholar 

  36. Donahue J, Anne-Hendricks L, Guadarrama S, Rohrbach M, Venugopalan S, Saenko K, Darrell T (2015) Long-term recurrent convolutional networks for visual recognition and description. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2625–2634.

  37. Donaldson M (2017) Plitchik’s Wheel of emotions–2017 Update. https://www.designwizard.com/wp-content/uploads/2017_old/09/plutchiks-modelof-emotions. Accessed 13 Apr 2020

  38. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 2: short papers), pp 49-54.

  39. Eyben F, Wullmer M, Schuller BO (2018) The Munich versatile and fast open-source audio feature extractor. In: Proceedings of ACM multimedia, pp 1459-1462.

  40. Fang X, Zhan J (2015) Sentiment analysis using product review data. J Big Data 2(1):5

    Google Scholar 

  41. Feng S, Wang Y, Liu L, Wang D, Yu G (2019) Attention based hierarchical LSTM network for context-aware microblog sentiment classification. World Wide Web 22(1):59–81

    Google Scholar 

  42. Ghosal D, Akhtar MS, Chauhan D, Poria S, Ekbal A, Bhattacharyya P (2018) Contextual inter-modal attention for multi-modal sentiment analysis. In: proceedings of the 2018 conference on empirical methods in natural language processing, pp 3454-3466.

  43. Go A, Bhayani R, Huang L (2009) Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12).

  44. Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18(5–6):602–610

    Google Scholar 

  45. Graves A, Schmidhuber J (2009) Offline handwriting recognition with multidimensional recurrent neural networks. In: Advances in neural information processing systems, pp. 545–552.

  46. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Chen T (2018) Recent advances in convolutional neural networks. Pattern Recogn 77:354–377

    Google Scholar 

  47. Guo S, Höhn S, Xu F, Schommer C (2018) PERSEUS: a personalization framework for sentiment categorization with recurrent neural network. In: International Conference on Agents and Artificial Intelligence, Funchal, pp. 94–102.

  48. Hammou BA, Lahcen AA, Mouline S (2020) Towards a real-time processing framework based on improved distributed recurrent neural network variants with fastText for social big data analytics. Inf Process Manag 57(1):102122

    Google Scholar 

  49. Hao W, Zhang Z, Guan H (2018) Integrating both visual and audio cues for enhanced video caption. arXiv, arXiv-1711.

  50. Hassan A, Mahmood A (2018) Convolutional recurrent deep learning model for sentence classification. IEEE Access 6:13949–13957

    Google Scholar 

  51. Hatua A, Nguyen TT, Sung AH (2017) Information diffusion on twitter: pattern recognition and prediction of volume, sentiment, and influence. In: Proceedings of the Fourth IEEE/ACM International Conference on Big Data Computing, Applications and Technologies, pp. 157–167.

  52. Hemmatian F, Sohrabi MK (2017) A survey on classification techniques for opinion mining and sentiment analysis. Artif Intell Rev 1–51.

  53. Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    MathSciNet  MATH  Google Scholar 

  54. Ho SL, Xie M, Goh TN (2002) A comparative study of neural network and box-Jenkins ARIMA modeling in time series prediction. Comput Ind Eng 42(2–4):371–375

    Google Scholar 

  55. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Google Scholar 

  56. Hu A, Flaxman S (2019) Multimodal sentiment analysis to explore the structure of emotions. In: proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining, pp 350-358.

  57. Hu S, Kumar A, Al-Turjman F, Gupta S, Seth S (2020) Reviewer credibility and sentiment analysis based user profile Modelling for online product recommendation. IEEE Access 8:26172–26189

    Google Scholar 

  58. Hu F, Li L, Zhang ZL, Wang JY, Xu XF (2017) Emphasizing essential words for sentiment classification based on recurrent neural networks. J Comput Sci Technol 32(4):785–795

    Google Scholar 

  59. Huang F, Zhang X, Zhao Z, Xu J, Li Z (2019) Image–text sentiment analysis via deep multimodal attentive fusion. Knowl-Based Syst 167:26–37

    Google Scholar 

  60. Hubert RB, Estevez E, Maguitman A, Janowski T (2018) Examining government-citizen interactions on twitter using visual and sentiment analysis. In: Proceedings of the 19th annual international conference on digital government research: governance in the data age, pp 1-10.

  61. Hussein DMEDM (2018) A survey on sentiment analysis challenges. Journal of King Saud Univ Eng Sci 30(4):330–338

    Google Scholar 

  62. Imdb.com Traffic, Demographics and Competitors – Alexa (2018). Alexa Internet. Accessed 1 October 2018.

  63. Jabreel M, Hassan F, Moreno A (2018) Target-dependent sentiment analysis of tweets using bidirectional gated recurrent neural networks. In: Advances in Hybridization of Intelligent Methods, pp. 39–55.

  64. Ji S, Xu W, Yang M, Yu K (2012) 3D convolutional neural networks for human action recognition. IEEE Trans Pattern Anal Mach Intell 35(1):221–231

    Google Scholar 

  65. Jiang K, Calix R, Gupta M (2016) Construction of a personal experience tweet corpus for health surveillance. In: Proceedings of the 15th workshop on biomedical natural language processing, pp 128-135.

  66. Jiang K, Feng S, Song Q, Calix RA, Gupta M, Bernard GR (2018) Identifying tweets of personal health experience through word embedding and LSTM neural network. BMC Bioinf 19(8):210

    Google Scholar 

  67. Johnson R, Zhang T (2015) Semi-supervised convolutional neural networks for text categorization via region embedding. In: Advances in neural information processing systems, pp. 919–927.

  68. Johnson R, Zhang T (2017) Deep pyramid convolutional neural networks for text categorization. In: Proceedings of the 55th annual meeting of the Association for Computational Linguistics (volume 1: long papers), pp 562-570.

  69. Kaladevi P, Thyagarajah K (2019) Integrated CNN-and LSTM-DNN-based sentiment analysis over big social data for opinion mining. Behav Inform Technol 1-9.

  70. Kalchbrenner N, Danihelka I, Graves A (2015) Grid long short-term memory. arXiv preprint arXiv:1507.01526.

  71. Kamel NS, Sayeed S, Ellis GA (2008) Glove-based approach to online signature verification. IEEE Trans Pattern Anal Mach Intell 30(6):1109–1113

    Google Scholar 

  72. Kansal N, Goel L, Gupta S (2020) A literature review on cross domain sentiment analysis using machine learning. Int J Artif Intell Mach Learn 10(2):43–56

    Google Scholar 

  73. Katsurai M, Satoh SI (2016) Image sentiment analysis using latent correlations among visual, textual, and sentiment views. In: 2016 IEEE international conference on acoustics, speech and signal processing - ICASSP’16, pp 2837-2841.

  74. Kauffmann E, Peral J, Gil D, Ferrández A, Sellers R, Mora H (2019) Managing marketing decision-making with sentiment analysis: an evaluation of the Main product features using text data mining. Sustainability 11(15):4235. https://doi.org/10.3390/su11154235

    Article  Google Scholar 

  75. Kaur R, Kautish S (2019) Multimodal sentiment analysis: a survey and comparison. Int J Serv Sci Manag Eng Technol 10:38–58

    Google Scholar 

  76. Khan W, Malik U, Ghazanfar MA, Azam MA, Alyoubi KH, Alfakeeh AS (2019) Predicting stock market trends using machine learning algorithms via public sentiment and political situation analysis. Soft Comput 1–25.

  77. Kim J, Kim J, Thu HLT, Kim H (2016) Long short term memory recurrent neural network classifier for intrusion detection. In: 2016 international conference on platform technology and service - PlatCon’16, pp 1-5.

  78. Kleenankandy J, Nazeer KA (2020) An enhanced tree-LSTM architecture for sentence semantic modeling using typed dependencies. arXiv preprint arXiv:2002.07775.

  79. Kotzias D, Denil M, De Freitas N, Smyth P (2015) From group to individual labels using deep features. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp 597-606.

  80. Lai S, Xu L, Liu K, Zhao J (2015) Recurrent convolutional neural networks for text classification. In: Twenty-ninth AAAI conference on artificial intelligence.

  81. Land WH, Schaffer JD (2020) The support vector machine. In: The art and science of machine intelligence. Springer, Cham, pp 45–76

    Google Scholar 

  82. Lauren P, Qu G, Yang J, Watta P, Huang GB, Lendasse A (2018) Generating word embeddings from an extreme learning machine for sentiment analysis and sequence labeling tasks. Cogn Comput 10(4):625–638

    Google Scholar 

  83. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International conference on machine learning, pp. 1188–1196.

  84. Lee G, Jeong J, Seo S, Kim C, Kang P (2018) Sentiment classification with word localization based on weakly supervised learning with a convolutional neural network. Knowl-Based Syst 152:70–82

    Google Scholar 

  85. Li L, Zhu X, Hao Y, Wang S, Gao X, Huang Q (2019) A hierarchical CNN-RNN approach for visual emotion classification. ACM Trans Multimed Comput Commun Appl 15(3s):1–7

    Google Scholar 

  86. Liang M, Hu X (2015) Recurrent convolutional neural network for object recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3367–3375.

  87. Liao S, Wang J, Yu R, Sato K, Cheng Z (2017) CNN for situations understanding based on sentiment analysis of twitter data. Procedia Comput Sci 111:376–381

    Google Scholar 

  88. Liu M, Chen L, Liu B, Wang X (2015) VRCA: a clustering algorithm for massive amount of texts. In: Twenty-Fourth International Joint Conference on Artificial Intelligence, pp. 2355–2361.

  89. Liu Y, Qin Z, Li P, Wan T (2017) Stock volatility prediction using recurrent neural networks with sentiment analysis. In: International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pp. 192–201.

  90. Liu Z, Shen Y, Lakshminarasimhan VB, Liang PP, Zadeh A, Morency LP (2018) Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064.

  91. Liu T, Yu S, Xu B, Yin H (2018) Recurrent networks with attention and convolutional networks for sentence representation and classification. Appl Intell 48(10):3797–3806

    Google Scholar 

  92. Lu Y, Hu X, Wang F, Kumar S, Liu H, Maciejewski R (2015) Visualizing social media sentiment in disaster scenarios. In: Proceedings of the 24th international conference on world wide web, pp 1211-1215.

  93. Lu K, Wu J (2019) Sentiment analysis of film review texts based on sentiment dictionary and SVM. In: Proceedings of the 2019 3rd international conference on innovation in artificial intelligence, pp 73-77.

  94. Majumder N (2017) Multimodal sentiment analysis in social media using deep learning with convolutional neural networks. CIC, Instituto Politécnico Nacional.

  95. Majumder N, Poria S, Peng H, Chhaya N, Cambria E, Gelbukh A (2019) Sentiment and sarcasm classification with multitask learning. IEEE Intell Syst 34(3):38–43

    Google Scholar 

  96. Markoff J (2012) Scientists see promise in deep-learning programs, NY Times. http://nyti.ms/sgcVec. https://www.nytimes.com/2012/11/24/science/scientists-see-advances-in-deep-learning-a-part-of-artificial-intelligence.html. Accessed 13 Apr 2020

  97. Mathews AP, Xie L, He X (2016) Senticap: generating image descriptions with sentiments. arXiv preprint arXiv:1510.01431.

  98. McGurk Z, Nowak A, Hall JC (2019) Stock returns and investor sentiment: textual analysis and social media. J Econ Financ 1–28.

  99. Medhat W, Hassan A, Korashy H (2014) Sentiment analysis algorithms and applications: a survey. Ain Shams Eng J 5(4):1093–1113

    Google Scholar 

  100. Mezaal MR, Pradhan B, Sameen MI, Shafri M, Zulhaidi H, Yusoff ZM (2017) Optimized neural architecture for automatic landslide detection from high-resolution airborne laser scanning data. Appl Sci 7(7):730

    Google Scholar 

  101. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

  102. Miyato T, Dai AM, Goodfellow I (2016) Adversarial training methods for semi-supervised text classification. arXiv preprint arXiv:1605.07725.

  103. Newberry C (28) Twitter statistics all marketers need to know in 2018. Hootsuite Blog, Retrieved October, 14, 2018.

  104. Ning Y, Muthiah S, Rangwala H, Ramakrishnan N (2016) Modeling precursors for event forecasting via nested multi-instance learning. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1095-1104.

  105. Nio L, Murakami K (2018) Japanese sentiment classification using bidirectional long short-term memory recurrent neural network. In: Proceedings of the 24th annual meeting Association for Natural Language Processing, pp 1119-1122.

  106. Palangi H, Deng L, Shen Y, Gao J, He X, Chen J, Ward R (2016) Deep sentence embedding using long short-term memory networks: analysis and application to information retrieval. IEEE Trans Audio Speech Lang Process (TASLP) 24(4):694–707

    Google Scholar 

  107. Pathak AR, Pandey M, Rautaray S (2019) Empirical evaluation of deep learning models for sentiment analysis. J Stats Manag Syst 22(4):741–752

    Google Scholar 

  108. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. arXiv preprint arXiv:1802.05365.

  109. Pham H, Manzini T, Liang PP, Poczos B (2018) Seq2seq2sentiment: multimodal sequence to sequence models for sentiment analysis. arXiv preprint arXiv:1807.03915.

  110. Poria S, Cambria E, Gelbukh A (2015) Deep convolutional neural network textual features and multiple kernel learning for utterance-level multimodal sentiment analysis. In: proceedings of the 2015 conference on empirical methods in natural language processing, pp 2539-2544.

  111. Poria S, Chaturvedi I, Cambria E, Hussain A (2016) Convolutional MKL based multimodal emotion recognition and sentiment analysis. In: 2016 IEEE 16th international conference on data mining – ICDM’16, pp 439-448).

  112. Qiu J, Wang B, Zhou C (2020) Forecasting stock prices with long-short term memory neural network based on attention mechanism. PLoS One 15(1):e0227222. https://doi.org/10.1371/journal.pone.0227222

    Article  Google Scholar 

  113. Rather AM, Agarwal A, Sastry VN (2015) Recurrent neural network and a hybrid model for prediction of stock returns. Expert Syst Appl 42(6):3234–3241

    Google Scholar 

  114. Ren R, Wu DD, Liu T (2018) Forecasting stock market movement direction using sentiment analysis and support vector machine. IEEE Syst J 13(1):760–770

    Google Scholar 

  115. Rojas-Barahona LM (2016) Deep learning for sentiment analysis. Lang Ling Compass 10(12):701–719

    Google Scholar 

  116. Rong X (2014) Word2vec parameter learning explained. arXiv preprint arXiv:1411.2738.

  117. Rong W, Peng B, Ouyang Y, Li C, Xiong Z (2015) Structural information aware deep semi-supervised recurrent neural network for sentiment analysis. Front Comp Sci 9(2):171–184

    MathSciNet  Google Scholar 

  118. Sachan DS, Zaheer M, Salakhutdinov R (2019) Revisiting LSTM networks for semi-supervised text classification via mixed objective function. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 33:6940–6948.

  119. Sak H, Senior A, Rao K, Beaufays F (2015) Fast and accurate recurrent neural network acoustic models for speech recognition. arXiv preprint arXiv:1507.06947.

  120. Salehinejad H, Sankar S, Barfett J, Colak E, Valaee S (2017) Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078.

  121. Sheikh I, Illina I, Fohr D (2017) Segmentation and classification of opinions with recurrent neural networks.

  122. Shenoy A, Sardana A (2020) Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation arXiv preprint arXiv:2002.08267.

  123. Sheoran A, Kanojia D, Joshi A, Bhattacharyya P (2020) Recommendation chart of domains for cross-domain sentiment analysis: findings of a 20 domain study. arXiv preprint arXiv:2004.04478.

  124. Sigurdsson GA, Chen X, Gupta A (2016) Learning visual storylines with skipping recurrent neural networks. In: European Conference on Computer Vision, pp. 71–88.

  125. Soleymani M, Garcia D, Jou B, Schuller B, Chang SF, Pantic M (2017) A survey of multimodal sentiment analysis. Image Vis Comput 65:3–14

    Google Scholar 

  126. Steyn DH, Greyling T, Rossouw S, Mwamba JM (2020) Sentiment, emotions and stock market predictability in developed and emerging markets (no. 502). GLO discussion paper

  127. Summarizing different types of sequence processing tasks (n.d.), https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789536089/5/ch05lvl1sec86/summarizing-different-types-of-sequence-processing-tasks

  128. Tang D, Qin B, Liu T (2015) Deep learning for sentiment analysis: successful approaches and future challenges. Wiley Interdiscip Rev: Data Min Knowl Disc 5(6):292–303

    Google Scholar 

  129. Tani HL (2016) Characteristics of visual categorization of long-concatenated and object-directed human actions by a multiple spatio-temporal scales recurrent neural network model. arXiv preprint arXiv:1602.01921.

  130. Tarasov DS (2015) Deep recurrent neural networks for multiple language aspect-based sentiment analysis of user reviews. In: Proceedings of the 21st international conference on computational linguistics dialog, pp 2:77-88.

  131. Tomihira T, Otsuka A, Yamashita A, Satoh T (2018) What does your tweet emotion mean? Neural emoji prediction for sentiment analysis. In: proceedings of the 20th international conference on information integration and web-based applications & services, pp 289-296.

  132. Vadicamo L, Carrara F, Cimino A, Cresci S, Dell’Orletta F, Falchi F, Tesconi M (2017) Cross-media learning for image sentiment analysis in the wild. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 308–317.

  133. Wang Y, Shen Y, Liu Z, Liang PP, Zadeh A, Morency LP (2019) Words can shift: dynamically adjusting word representations using nonverbal behaviors. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 33:7216–7223.

  134. Wang Y, Sun A, Han J, Liu Y, Zhu X (2018) Sentiment analysis by capsules. In: Proceedings of the 2018 world wide web conference, pp 1165-1174.

  135. Wang J, Zhang L, Chen Y, Yi Z (2018) A new delay connection for long short-term memory networks. Int J Neural Syst 28(06):1750061

    Google Scholar 

  136. Wei D, Wang B, Lin G, Liu D, Dong Z, Liu H, Liu Y (2017) Research on unstructured text data mining and fault classification based on RNN-LSTM with malfunction inspection report. Energies 10(3):406

    Google Scholar 

  137. Wen Y, Xu A, Liu W, Chen L (2018) A wide residual network for sentiment classification. In: proceedings of the 2nd international conference on deep learning technologies, pp 7-11.

  138. Werbos PJ (1990) Backpropagation through time: what it does and how to do it. Proc IEEE 78(10):1550–1560

    Google Scholar 

  139. Wissner-Gross A (2016) Datasets over algorithms. Edge.com. Accessed 8 January 2016.

  140. Wu D, Chi M (2017) Long short-term memory with quadratic connections in recursive neural networks for representing compositional semantics. IEEE Access 5:16077–16083

    Google Scholar 

  141. Wu DD, Zheng L, Olson DL (2014) A decision support approach for online stock forum sentiment analysis. IEEE Trans Syst Man Cybern Syst 44(8):1077–1087

    Google Scholar 

  142. Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y (2019) Visual-textual sentiment classification with bi-directional multi-level attention networks. Knowl-Based Syst 178:61–73

    Google Scholar 

  143. Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y (2019) Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput 80:387–399

    Google Scholar 

  144. Xu J, Li H, Zhou S (2015) An overview of deep generative models. IETE Tech Rev 32(2):131–139

    Google Scholar 

  145. Xu N, Liu AA, Wong Y, Zhang Y, Nie W, Su Y, Kankanhalli M (2018) Dual-stream recurrent neural network for video captioning. IEEE Trans Circuits Syst Video Technol 29(8):2482–2493

    Google Scholar 

  146. Xu N, Mao W, Chen G (2019) Multi-interactive memory network for aspect based multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp 33:371–378.

  147. Yadav A, Vishwakarma DK (2019) Sentiment analysis using deep learning architectures: a review. Artif Intell Rev 1–51.

  148. Yang L, Li Y, Wang J, Sherratt RS (2020) Sentiment analysis for E-commerce product reviews in Chinese based on sentiment lexicon and deep learning. IEEE Access 8:23522–23530

    Google Scholar 

  149. You Q, Cao L, Jin H, Luo J (2016) Robust visual-textual sentiment analysis: when attention meets tree-structured recursive neural networks. In: proceedings of the 24th ACM international conference on multimedia, pp 1008-1017.

  150. You Q, Luo J, Jin H, Yang J (2016) Cross-modality consistent regression for joint visual-textual sentiment analysis of social multimedia. In: Proceedings of the Ninth ACM international conference on Web search and data mining, pp. 13–22.

  151. Yu Y, Lin H, Meng J, Zhao Z (2016) Visual and textual sentiment analysis of a microblog using deep convolutional neural networks. Algorithms 9(2):41

    MathSciNet  MATH  Google Scholar 

  152. Yu LC, Wu JL, Chang PC, Chu HS (2013) Using a contextual entropy model to expand emotion words and their intensity for the sentiment classification of stock market news. Knowl-Based Syst 41:89–97

    Google Scholar 

  153. Yue B, Fu J, Liang J (2018) Residual recurrent neural networks for learning sequential representations. Information 9(3):56

    Google Scholar 

  154. Zadeh A, Chen M, Poria S, Cambria E, Morency LP (2017) Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250.

  155. Zadeh A, Liang PP, Mazumder N, Poria S, Cambria E, Morency LP (2018) Memory fusion network for multi-view sequential learning. arXiv preprint arXiv:1802.00927.

  156. Zadeh A, Liang PP, Poria S, Vij P, Cambria E, Morency LP (2018) Multi-attention recurrent network for human communication comprehension. In: Thirty-Second AAAI Conference on Artificial Intelligence. (vol. 2018, pp 5642). NIH Public Access.

  157. Zadeh A, Zellers R, Pincus E, Morency LP (2016) Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages. IEEE Intell Syst 31(6):82–88

    Google Scholar 

  158. Zaytar MA, El Amrani C (2016) Sequence to sequence weather forecasting with long short-term memory recurrent neural networks. Int J Comput Appl 143(11):7–11

    Google Scholar 

  159. Zhang Y, Jiang Y, Tong Y (2016) Study of sentiment classification for Chinese microblog based on recurrent neural network. Chin J Electron 25(4):601–607

    Google Scholar 

  160. Zhang Y, Liu Q, Song L (2018) Sentence-state LSTM for text representation. arXiv preprint arXiv:1805.02474.

  161. Zhang Y, Song D, Li X, Zhang P, Wang P, Rong L, Yu G, Wang B (2020) A quantum-like multimodal network framework for modeling interaction dynamics in multiparty conversational sentiment analysis. Inform Fusion 62:14–31

    Google Scholar 

  162. Zhang L, Wang S, Liu B (2018) Deep learning for sentiment analysis: a survey. Wiley Interdiscip Rev: Data Min Knowl Disc 8(4):e1253

    Google Scholar 

  163. Zhang XY, Yin F, Zhang YM, Liu CL, Bengio Y (2017) Drawing and recognizing chinese characters with recurrent neural network. IEEE Trans Pattern Anal Mach Intell 40(4):849–862

    Google Scholar 

  164. Zhang X, Zhao J, LeCun Y (2015) Character-level convolutional networks for text classification. In: Advances in neural information processing systems, pp. 649–657.

  165. Zhao W, Guan Z, Chen L, He X, Cai D, Wang B, Wang Q (2017) Weakly-supervised deep embedding for product review sentiment analysis. IEEE Trans Knowl Data Eng 30(1):185–197

    Google Scholar 

  166. Zhao C, Wang S, Li D (2020) Multi-source domain adaptation with joint learning for cross-domain sentiment classification. Knowl-Based Syst 191:105254

    Google Scholar 

  167. Zheng J, Guo Y, Feng C, Chen H (2018) A hierarchical neural-network-based document representation approach for text classification. Math Probl Eng

  168. Zhu X, Li L, Zhang W, Rao T, Xu M, Huang Q, Xu D (2019) Dependency exploitation: a unified CNN-RNN approach for visual emotion recognition. In: proceedings of the 26th international joint conference on artificial intelligence, pp 3595-3601.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Indian Institute of Information Technology, Nagpur, India

    Jitendra V. Tembhurne & Tausif Diwan

Authors
  1. Jitendra V. Tembhurne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tausif Diwan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jitendra V. Tembhurne.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tembhurne, J.V., Diwan, T. Sentiment analysis in textual, visual and multimodal inputs using recurrent neural networks. Multimed Tools Appl 80, 6871–6910 (2021). https://doi.org/10.1007/s11042-020-10037-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10037-x

Keywords

Search

Navigation