Abstract
Multimodality sentiment classification of social media attracts increasing attention, whose main purpose is to predict the sentiment of the target mentioned in the posts. Current research mainly focuses on integrating the multimodal data, but fails to consider the impacts on the target. In this work, we tend to propose a target-oriented multimodal sentiment classification model. Specifically, our model starts with exploiting the target-oriented topic within the text. Then, a multi-head attention network is established to learn the multimodal interaction among textual, visual and topic information, based on which the target-oriented representations of the topic, the text and the image are obtained. Moreover, a gating unit to fuse the multimodal information is also built up. On the task of target-oriented multimodal sentiment classification, experiments on multimodal samples are carried out on manually annotated the dataset. Experimental results reveal that our method significantly reduces the gap over each given target, which sets a foundation to achieve the state-of-arts sentiment classification results.
Similar content being viewed by others
Data availability
The datasets generated and analyzed during the current study are available from the corresponding author on reasonable request.
References
https://www.omnicoreagency.com/twitter-statistics/. Accessed 10 Aug 2021
Saha T, Upadhyaya A, Saha S, Bhattacharyya P (2022) A multitask multimodal ensemble model for sentiment- and emotion-aided tweet act classificatio. IEEE transactions on computational social systems 9(2):508–517. https://doi.org/10.1109/TCSS.2021.3088714
M. E. Lewis, J. M. Haviland, Handbook of emotions
Wang Y, Huang M, Zhu X, Zhao L (2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing pp. 606–615
Schouten K, Frasincar F (2015) Survey on aspect-level sentiment analysis. IEEE Trans Knowl Data Eng 28(3):813–830
Bakshi RK, Kaur N, Kaur R, Kaur G (2016) Opinion mining and sentiment analysis. In: 2016 3rd International Conference on computing for sustainable global development (INDIACom), IEEE, Vol.2, 2016, pp 452–455
Xu N, Mao W, Chen G (2019) Multi-interactive memory network for aspect based multimodal sentiment analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33, 2019, pp 371–378
Xu J, Li Z, Huang F, Li C, Philip SY (2020) Social image sentiment analysis by exploiting multimodal content and heterogeneous relations. IEEE Trans Industr Inf 17(4):2974–2982
Zhang K, Zhu Y, Zhang W, Zhang W, Zhu Y (2020) Transfer correlation between textual content to images for sentiment analysis. IEEE Access 8:35276–35289
Yu Y, Jiang J (2019) Adapting bert for target-oriented multimodal sentiment classification. In: IJCAI
Yu J, Jiang J, Xia R (2019) Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification. IEEE/ACM Trans Audio Speech Lang Process 28:429–439
Cai Y, Cai H, Wan X (2019) Multi-modal sarcasm detection in twitter with hierarchical fusion model. In: Proceedings of the 57th Annual Meeting of the ACL, pp 2506–2515
Linmei H, Yang T, Shi C, Ji H, Li X (2019) Heterogeneous graph attention networks for semi-supervised short text classification. EMNLP-IJCNLP 39:4821–4830
Yan Jiang HS, Gao J, Cheng X Incorporating topic information and bert embedding for stance detection in twitter text. Chin Inf Process
Wang J, Gu D, Yang C, Xue Y, Song Z, Zhao H, Xiao L Targeted aspect based multimodal sentiment analysis: an attention capsule extraction and multi-head fusion network, arXiv preprint arXiv:2103.07659
Ren L, Lin H, Xu B, Yang L, Zhang D (2021) Learning to capture contrast in sarcasm with contextual dual-view attention network. Int J Mach Learn Cybern, 401:1–9
Huang Y, Chen J, Zheng S, Xue Y, Hu X (2021) Hierarchical multi-attention networks for document classification. Int J Mach Learn Cybern 12(6):1639–1647
Dong L, Wei F, Tan C, Tang D, Zhou M, Xu K (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for computational linguistics (volume 2: Short papers) pp 49–54
Tang D, Qin B, Feng X, Liu T Effective lstms for target-dependent sentiment classification, arXiv preprint arXiv:1512.01100
Xue W, Li T Aspect based sentiment analysis with gated convolutional networks, arXiv preprint arXiv:1805.07043
Yang X, Feng S, Wang D, Zhang Y (2021) Image-Text Multimodal Emotion Classification via Multi-View Attentional Network, In: IEEE Transactions on Multimedia 23:4014–4026. https://doi.org/10.1109/TMM.2020.3035277
Pathak AR, Pandey M, Rautaray S (2021) Topic level sentiment analysis of social media data using deep learning. Appl Soft Comput 108:107440
Dahal B, Kumar SA, Li Z (2019) Topic modeling and sentiment analysis of global climate change tweets. Soc Netw Anal Min 9(1):1–20
Fu X, Sun X, Wu H, Cui L, Huang JZ (2018) Weakly supervised topic sentiment joint model with word embeddings. Knowl Based Syst 147:43–54
Ma D, Li S, Zhang X, Wang H Interactive attention networks for aspect-level sentiment classification, arXiv preprint arXiv:1709.00893
Liao W, Zeng B, Liu J, Wei P, Cheng X, Zhang W (2021) Multi-level graph neural network for text sentiment analysis. Comput Electr Eng 92:107096
F. Alzazah, X. Cheng and X. Gao ( 2022) Predict Market Movements Based on the Sentiment of Financial Video News Sites, 2022 IEEE 16th International Conference on Semantic Computing (ICSC) 103–110. https://doi.org/10.1109/ICSC52841.2022.00022.
Lu D, Neves L, Carvalho V, Zhang N, Ji H (2018) Visual attention model for name tagging in multimodal social media. In Proc Annu Meet Assoc Comput Linguist 1:1990–1999
Zhang Q, Fu J, Liu X, Huang X (2018) Adaptive co-attention network for named entity recognition in tweets. In: Proc. AAAI Conf. Artif. Intell. 5674–5681
Adversarial Incomplete Multi-view Clustering, IJCAI 2019
Multiview concept learning via deep matrix factorization, TNNLS 2021
Multimodal gesture recognition based on the resc3d network, ICCV 2017
Cr-net: A deep classification-regression network for multimodal apparent personality analysis, IJCV 2020
Funding
This work was supported by the National Statistical Science Research Project of China under Grant No.2016LY98, the Characteristic Innovation Projects of Guangdong Colleges and Universities (Nos. 2018KTSCX049), the Science and Technology Plan Project of Guangzhou under Grant Nos. 202102080258 and 201903010013.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Song, Z., Xue, Y., Gu, D. et al. Target-oriented multimodal sentiment classification by using topic model and gating mechanism. Int. J. Mach. Learn. & Cyber. 14, 2289–2299 (2023). https://doi.org/10.1007/s13042-022-01757-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-022-01757-7