Skip to main content
SpringerLink
Log in

Target-oriented multimodal sentiment classification by using topic model and gating mechanism

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Multimodality sentiment classification of social media attracts increasing attention, whose main purpose is to predict the sentiment of the target mentioned in the posts. Current research mainly focuses on integrating the multimodal data, but fails to consider the impacts on the target. In this work, we tend to propose a target-oriented multimodal sentiment classification model. Specifically, our model starts with exploiting the target-oriented topic within the text. Then, a multi-head attention network is established to learn the multimodal interaction among textual, visual and topic information, based on which the target-oriented representations of the topic, the text and the image are obtained. Moreover, a gating unit to fuse the multimodal information is also built up. On the task of target-oriented multimodal sentiment classification, experiments on multimodal samples are carried out on manually annotated the dataset. Experimental results reveal that our method significantly reduces the gap over each given target, which sets a foundation to achieve the state-of-arts sentiment classification results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. https://www.omnicoreagency.com/twitter-statistics/. Accessed 10 Aug 2021

  2. Saha T, Upadhyaya A, Saha S, Bhattacharyya P (2022) A multitask multimodal ensemble model for sentiment- and emotion-aided tweet act classificatio. IEEE transactions on computational social systems 9(2):508–517. https://doi.org/10.1109/TCSS.2021.3088714

  3. M. E. Lewis, J. M. Haviland, Handbook of emotions

  4. Wang Y, Huang M, Zhu X, Zhao L (2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing pp. 606–615

  5. Schouten K, Frasincar F (2015) Survey on aspect-level sentiment analysis. IEEE Trans Knowl Data Eng 28(3):813–830

    Article  Google Scholar 

  6. Bakshi RK, Kaur N, Kaur R, Kaur G (2016) Opinion mining and sentiment analysis. In: 2016 3rd International Conference on computing for sustainable global development (INDIACom), IEEE, Vol.2, 2016, pp 452–455

  7. Xu N, Mao W, Chen G (2019) Multi-interactive memory network for aspect based multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp 371–378

  8. Xu J, Li Z, Huang F, Li C, Philip SY (2020) Social image sentiment analysis by exploiting multimodal content and heterogeneous relations. IEEE Trans Industr Inf 17(4):2974–2982

    Article  Google Scholar 

  9. Zhang K, Zhu Y, Zhang W, Zhang W, Zhu Y (2020) Transfer correlation between textual content to images for sentiment analysis. IEEE Access 8:35276–35289

    Article  Google Scholar 

  10. Yu Y, Jiang J (2019) Adapting bert for target-oriented multimodal sentiment classification. In: IJCAI

  11. Yu J, Jiang J, Xia R (2019) Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE/ACM Trans Audio Speech Lang Process 28:429–439

    Article  Google Scholar 

  12. Cai Y, Cai H, Wan X (2019) Multi-modal sarcasm detection in twitter with hierarchical fusion model. In: Proceedings of the 57th Annual Meeting of the ACL, pp 2506–2515

  13. Linmei H, Yang T, Shi C, Ji H, Li X (2019) Heterogeneous graph attention networks for semi-supervised short text classification. EMNLP-IJCNLP 39:4821–4830

  14. Yan Jiang HS, Gao J, Cheng X Incorporating topic information and bert embedding for stance detection in twitter text. Chin Inf Process

  15. Wang J, Gu D, Yang C, Xue Y, Song Z, Zhao H, Xiao L Targeted aspect based multimodal sentiment analysis: an attention capsule extraction and multi-head fusion network, arXiv preprint arXiv:2103.07659

  16. Ren L, Lin H, Xu B, Yang L, Zhang D (2021) Learning to capture contrast in sarcasm with contextual dual-view attention network. Int J Mach Learn Cybern, 401:1–9

  17. Huang Y, Chen J, Zheng S, Xue Y, Hu X (2021) Hierarchical multi-attention networks for document classification. Int J Mach Learn Cybern 12(6):1639–1647

    Article  Google Scholar 

  18. Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for computational linguistics (volume 2: Short papers) pp 49–54

  19. Tang D, Qin B, Feng X, Liu T Effective lstms for target-dependent sentiment classification, arXiv preprint arXiv:1512.01100

  20. Xue W, Li T Aspect based sentiment analysis with gated convolutional networks, arXiv preprint arXiv:1805.07043

  21. Yang X, Feng S, Wang D, Zhang Y (2021) Image-Text Multimodal Emotion Classification via Multi-View Attentional Network, In: IEEE Transactions on Multimedia 23:4014–4026. https://doi.org/10.1109/TMM.2020.3035277

  22. Pathak AR, Pandey M, Rautaray S (2021) Topic level sentiment analysis of social media data using deep learning. Appl Soft Comput 108:107440

    Article  Google Scholar 

  23. Dahal B, Kumar SA, Li Z (2019) Topic modeling and sentiment analysis of global climate change tweets. Soc Netw Anal Min 9(1):1–20

    Article  Google Scholar 

  24. Fu X, Sun X, Wu H, Cui L, Huang JZ (2018) Weakly supervised topic sentiment joint model with word embeddings. Knowl Based Syst 147:43–54

    Article  Google Scholar 

  25. Ma D, Li S, Zhang X, Wang H Interactive attention networks for aspect-level sentiment classification, arXiv preprint arXiv:1709.00893

  26. Liao W, Zeng B, Liu J, Wei P, Cheng X, Zhang W (2021) Multi-level graph neural network for text sentiment analysis. Comput Electr Eng 92:107096

    Article  Google Scholar 

  27. F. Alzazah, X. Cheng and X. Gao ( 2022) Predict Market Movements Based on the Sentiment of Financial Video News Sites, 2022 IEEE 16th International Conference on Semantic Computing (ICSC) 103–110. https://doi.org/10.1109/ICSC52841.2022.00022.

  28. Lu D, Neves L, Carvalho V, Zhang N, Ji H (2018) Visual attention model for name tagging in multimodal social media. In Proc Annu Meet Assoc Comput Linguist 1:1990–1999

  29. Zhang Q, Fu J, Liu X, Huang X (2018) Adaptive co-attention network for named entity recognition in tweets. In: Proc. AAAI Conf. Artif. Intell. 5674–5681

  30. Adversarial Incomplete Multi-view Clustering, IJCAI 2019

  31. Multiview concept learning via deep matrix factorization, TNNLS 2021

  32. Multimodal gesture recognition based on the resc3d network, ICCV 2017

  33. Cr-net: A deep classification-regression network for multimodal apparent personality analysis, IJCV 2020

Download references

Funding

This work was supported by the National Statistical Science Research Project of China under Grant No.2016LY98, the Characteristic Innovation Projects of Guangdong Colleges and Universities (Nos. 2018KTSCX049), the Science and Technology Plan Project of Guangzhou under Grant Nos. 202102080258 and 201903010013.

Author information

Authors and Affiliations

  1. Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum, Materials, School of Physics and Telecommunication Engineering, South China Normal University, Guangzhou, 510006, China

    Zhengxin Song, Yun Xue & Donghong Gu

  2. NIT, Zhejiang University, Zhejiang, 310058, China

    Haolan Zhang

  3. School of Information Science and Technology, Nantong University, Nantong, Nantong, 226019, China

    Weiping Ding

Authors
  1. Zhengxin Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Donghong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haolan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weiping Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yun Xue.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, Z., Xue, Y., Gu, D. et al. Target-oriented multimodal sentiment classification by using topic model and gating mechanism. Int. J. Mach. Learn. & Cyber. 14, 2289–2299 (2023). https://doi.org/10.1007/s13042-022-01757-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-022-01757-7

Keywords

Search

Navigation