Skip to main content
SpringerLink
Log in

A mutual attention based multimodal fusion for fake news detection on social network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

As the advance of social networks, the emergency of fake news has been the major threat for information security, privacy, and trustworthiness. The fake news can leverage multimedia contents to fabricate evidences or mislead readers, which damages a lot in machine learning and network systems. In this work, we explored the task of multimodal fake news detection. The major challenge of fake news detection stems from the modality fusion by abundant information. Overcoming the limitations of the current models, we tackle the challenge of learning corrections between modalities in news, and substantially proposed a mutual attention neural network (MANN) that can learn the relationship between each different modality. Our model consists of four components: multimodal feature extractor, mutual attention fusion, fake news detector and irrelevant event discriminator. The performance of our proposed architecture is evaluated on Weibo dataset, which indicates the MANN model outperforms the state-of-the-arts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick C L, Parikh D (2015) Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision (pp. 2425–2433)

  2. Boididou C, Andreadou K, Papadopoulos S, Dang-Nguyen D T, Boato G, Riegler M, Kompatsiaris Y, et al. (2015) Verifying multimedia use at mediaeval 2015. MediaEval 3(3):7

    Google Scholar 

  3. Castillo C, Mendoza M, Poblete B (2011) Information credibility on twitter. In: Proceedings of the 20th international conference on World wide web, pp 675–684

  4. Conroy N K, Rubin V L, Chen Y (2015) Automatic deception detection: methods for finding fake news. Proceedings of the association for information science and technology 52(1):1–4

    Article  Google Scholar 

  5. Du X, Zhu R, Zhao F, Zhao F, Han P, Zhu Z (2020) A deceptive detection model based on topic, sentiment, and sentence structure information. Appl Intell 50(11):3868–3881

    Article  Google Scholar 

  6. Feng S, Banerjee R, Choi Y (2012) Syntactic stylometry for deception detection. In: Proceedings of the 50th annual meeting of the association for computational linguistics (volume 2: short papers), pp 171–175

  7. Gupta A, Lamba H, Kumaraguru P, Joshi A (2013) Faking sandy: characterizing and identifying fake images on twitter during hurricane sandy. In: Proceedings of the 22nd international conference on World Wide Web, pp 729–736

  8. Gupta A, Kumaraguru P, Castillo C, Meier P (2014) Tweetcred: real-time credibility assessment of content on twitter. In: International conference on social informatics, Springer, pp 228–243

  9. Gupta M, Zhao P, Han J (2012) Evaluating event credibility on twitter. In: Proceedings of the SIAM international conference on data mining. SIAM, pp 153–164

  10. Hao M, Xu B, Liang JY, Zhang BW, Yin XC (2020) Chinese short text classification with mutual-attention convolutional neural networks. ACM Trans Asian Low Resour Lang Inf Process (TALLIP) 19(5):1–13

    Article  Google Scholar 

  11. Jain P, Singh V (2016) Credrank: evaluating tweet credibility during high impact events. In: 2016 2nd international conference on contemporary computing and informatics (IC3I). IEEE, pp 553–557

  12. Ji W, Guo J, Li Y (2020) Multi-head mutual-attention cyclegan for unpaired image-to-image translation. IET Image Process 14(11):2395–2402

    Article  Google Scholar 

  13. Jiang N, Tian F, Li J, Yuan X, Zheng J (2020) Man: mutual attention neural networks model for aspect-level sentiment classification in siot. IEEE Internet Things J 7(4):2901–2913. https://doi.org/10.1109/JIOT.2020.2963927

    Article  Google Scholar 

  14. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimed 19(3):598–608

    Article  Google Scholar 

  15. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017, October) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In Proceedings of the 25th ACM international conference on Multimedia (pp. 795–816)

  16. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia, association for computing machinery. New York, MM ’17, p 795–816, DOI https://doi.org/10.1145/3123266.3123454, (to appear in print)

  17. Karpathy A, Fei-Fei L (2016) Deep visual-semantic alignments for generating image descriptions. IEEE Transactions on Pattern Analysis & Machine Intelligence, pp 664–676

  18. Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: multimodal variational autoencoder for fake news detection. In: The world wide web conference, pp 2915–2921

  19. Kwon S, Cha M, Jung K, Chen W, Wang Y (2013) Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th International conference on data mining. IEEE, pp 1103–1108

  20. Liu N, Zhang N, Han J (2020) Learning selective self-mutual attention for rgb-d saliency detection, pp 13753–13762. https://doi.org/10.1109/CVPR42600.2020.01377

  21. Liu Y, Wu YF (2018) Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  22. Ma J, Gao W, Mitra P, Kwon S, Jansen B J, Wong KF, Cha M (2016) Detecting rumors from microblogs with recurrent neural networks

  23. Ma J, Gao W, Wong KF (2018) Rumor detection on twitter with tree-structured recursive neural networks. Association for Computational Linguistics

  24. Ma Q, Yu L, Tian S, Chen E, Ng WWY (2019) Global-local mutual attention model for text classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27(12):2127–2139. https://doi.org/10.1109/TASLP.2019.2942160

    Article  Google Scholar 

  25. Mihalcea R, Strapparava C (2009) The lie detector: explorations in the automatic recognition of deceptive language. In: Proceedings of the ACL-IJCNLP 2009 conference short papers, pp 309–312

  26. Popat K, Mukherjee S, Strötgen J, Weikum G (2016) Credibility assessment of textual claims on the web. In: Proceedings of the 25th ACM international on conference on information and knowledge management, pp 2173–2178

  27. Raj C, Meel P (2021) Convnet frameworks for multi-modal fake news detection. Appl Intell, pp 1–17

  28. Rashkin H, Choi E, Jang JY, Volkova S, Choi Y (2017) Truth of varying shades: analyzing language in fake news and political fact-checking. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp 2931–2937

  29. Shu K, Sliva A, Wang S, Tang J, Liu H (2017) Fake news detection on social media: a data mining perspective. ACM SIGKDD Explorations Newsl 19(1):22–36

    Article  Google Scholar 

  30. Singhal S, Shah RR, Chakraborty T, Kumaraguru P, Satoh S (2019) Spotfake: a multi-modal framework for fake news detection. In: 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, pp 39–47

  31. Song C, Yang C, Chen H, Tu C, Liu Z, Sun M (2019) Ced: credible early detection of social media rumors. IEEE Transactions on Knowledge and Data Engineering

  32. Sun S, Liu H, He J, Du X (2013) Detecting event rumors on sina weibo automatically. In: Asia-Pacific web conference. Springer, pp 120–131

  33. Tuan NMD, Minh PQN (2021) Multimodal fusion with bert and attention mechanism for fake news detection. In: 2021 RIVF international conference on computing and communication technologies (RIVF). IEEE, pp 1–6

  34. Vinyals O, Toshev A, Bengio S, Erhan D (2015) Show and tell: a neural image caption generator

  35. Wang Y, Ma F, Jin Z, Yuan Y, Jha K (2018) Eann: event adversarial neural networks for multi-modal fake news detection. In: Acm sigkdd international conference

  36. Wu K, Yang S, Zhu KQ (2015) False rumors detection on sina weibo by propagation structures. In: 2015 IEEE 31st international conference on data engineering. IEEE, pp 651–662

  37. Wu L, Liu H (2018) Tracing fake-news footprints: characterizing social media messages by how they propagate. In: Proceedings of the eleventh ACM international conference on web search and data mining, pp 637–645

  38. You Q, Cao L, Jin H, Luo J (2016) Robust visual-textual sentiment analysis: when attention meets tree-structured recursive neural networks. In: the 2016 ACM

  39. Yu F, Liu Q, Wu S, Wang L, Tan T et al (2017) A convolutional approach for misinformation identification

  40. Yu F, Liu Q, Wu S, Wang L, Tan T (2019) Attention-based convolutional approach for misinformation identification from massive and noisy microblog posts. Comput Secur 83:106–121

    Article  Google Scholar 

Download references

Acknowledgements

This research work is funded by the Science Foundation of North China University of Technology, R&D Program of Beijing Municipal Education Commission (KM202210009001) and Beijing Social Science Foundation (21XCCC013).

Author information

Authors and Affiliations

  1. Department of Computer Science, North China University of Technology, No. 5 Jinyuanzhuang Road, Shijingshan District, Beijing, 100144, People’s Republic of China

    Ying Guo

Authors
  1. Ying Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Guo.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, Y. A mutual attention based multimodal fusion for fake news detection on social network. Appl Intell 53, 15311–15320 (2023). https://doi.org/10.1007/s10489-022-04266-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04266-w

Keywords

Search

Navigation