Skip to main content
SpringerLink
Log in

An effective strategy for multi-modal fake news detection

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

News plays an indispensable role in the development of human society. With the emergence of new media, fake news including multi-modal content such as text and images has greater social harm. Therefore how to identify multi-modal fake news has been a challenge. The traditional methods of multi-modal fake news detection are to simply fuse the different modality information, such as concatenation and element-wise product, without considering the different impacts of the different modalities, which leads to the low accuracy of fake news detection. To address this issue, we design a new multi-modal attention adversarial fusion method built on the pre-training language model BERT, which consists of two important components: the attention mechanism and the adversarial mechanism. The attention mechanism is used to capture the differences in different modalities. The adversarial mechanism is to capture the correlation between different modalities. Experiments on a fake news Chinese public dataset indicate that our proposed new method achieves 5% higher in terms of F1.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://github.com/1210882202/data

References

  1. Agrawal A, Lu J, Antol S, Mitchell M, Zitnick CL, Parikh D, Batra D (2015) VQA: Visual Question Answering. International Journalof Computer Vision 123(1), 4

  2. Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick CL, Parikh D (2015) .. In: Proceedings of the IEEE international conference on computer vision, pp 2425–2433

  3. Bhatt G, Sharma A, Sharma S, Nagpal A, Raman B, Mittal A (2018) Combining Neural, Statistical and External Features for Fake News Stance Identification. In: Companion proceedings of the the web conference 2018, pp 1353–1357

  4. Cadene R, Ben-Younes H, Cord M, Thome N (2019) MUREL: Multimodal Relational Reasoning for Visual Question Answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1989–1998

  5. Cao J, Sheng Q, Qi P (2020) Progress and Prospects of Internet False Information Detection. Newsletter of the chinese computer society (3) 52

  6. Cui L, Wang S, Lee D (2019) SAME: sentiment-aware multi-modal embedding for detecting fake news. In: Proceedings of the 2019 IEEE/ACM international conference on advances in social networks analysis and mining, pp 41–48

  7. Devlin J, Chang M W, Lee K, Toutanova K (2018) arXiv:1810.04805

  8. Durier F, Vieira R, Garcia AC (2019) Can Machines Learn To Detect Fake News? A Survey Focused on Social Media

  9. Fukui A, Park DH, Yang D, Rohrbach A, Darrell T, Rohrbach M (2016) Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding

  10. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  11. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708

  12. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Transactions on Multimedia 19(3):598

    Article  Google Scholar 

  13. Kafle K, Kanan C (2016) .. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR)

  14. Karpathy A, Fei-Fei L. (2015) Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3128–3137

  15. Ke L, Zou C, Bu S, Yun L, Gong M (2017) Multi-modal Feature Fusion for Geographic Image Annotation. Pattern recognition 73

  16. Khattar D, Singh J, Gupta M, Varma V (2019) MVAE: Multimodal Variational Autoencoder for Fake News Detection. pp. 2915–2921. https://doi.org/10.1145/3308558.3313552

  17. Ma J, Gao W, Mitra P, Kwon S, Jansen BJ, Wong KF, Cha M (2016) .

  18. McCann B, Bradbury J, Xiong C, Socher R (2017) Learned in translation: Contextualized word vectors. In: Advances in neural information processing systems, pp 6294–6305

  19. Mohtarami M, Baly R, Glass J, Nakov P, Màrquez L, Moschitti A (2018) arXiv:1804.07581

  20. Parikh SB, Atrey PK (2018) Media-rich fake news detection: A survey. In: 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR) (IEEE, 2018), pp 436–441

  21. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  22. Qian F, Gong C, Sharma K, Liu Y (2018) .. In: Twenty-Seventh international joint conference on artificial intelligence IJCAI-18

  23. Rashkin H, Choi E, Jin Y J, Volkova S, Choi Y. (2017) Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

  24. Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99

  25. Shah P, Kobti Z (2020) Multimodal fake news detection using a Cultural Algorithm with situational and normative knowledge. In: 2020 IEEE Congress on Evolutionary Computation (CEC), pp 1–7. https://doi.org/10.1109/CEC48606.2020.9185643

  26. Shu K, Sliva A, Wang S, Tang J, Liu H (2017) . ACM SIGKDD Explorations Newsletter 19(1):22

    Article  Google Scholar 

  27. Simonyan K, Zisserman A (2014) Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556

  28. Simonyan K, Zisserman A (2015) Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv:1409.1556

  29. Singhal S, Shah RR, Chakraborty T, Kumaraguru P, Kabra A, Sharma M (2020) .. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20), pp 13,915–13,916

  30. Singhal S, Shah RR, Chakraborty T, Kumaraguru P, Satoh S (2019) SpotFake: A Multi-modal Framework for Fake News Detection. In: 2019 IEEE fifth international conference on multimedia big data (BigMM), pp 39–47. https://doi.org/10.1109/BigMM.2019.00-44

  31. Song C, Ning N, Zhang Y, Wu B (2020) A Multimodal Fake News Detection Model Based on Crossmodal Attention Residual and Multichannel Convolutional Neural Networks. Information Processing and Management 58. https://doi.org/10.1016/j.ipm.2020.102437

  32. Sun C, Qiu X, Xu Y, Huang X (2019) .. In: China national conference on chinese computational linguistics (Springer), pp 194–206

  33. Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, Su L, Gao J. (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining, pp 849–857

  34. Wang Y, Qian S, Hu J, Fang Q, Xu C (2020) .. In: Proceedings of the 2020 International Conference on Multimedia Retrieval (Association for Computing Machinery, New York, NY, USA), ICMR ’20, p 540–547. https://doi.org/10.1145/3372278.3390713

  35. Zhang Q, Yilmaz E, Liang S (2018) Ranking-based method for news stance detection. In: Companion proceedings of the the web conference 2018, pp 41–42

  36. Zhou X, Wu J, Reza Z (2020) SAFE: Similarity-Aware Multi-Modal Fake News Detection. arXiv:2003.04981

Download references

Author information

Authors and Affiliations

  1. State Key Lab of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Room 519, New Science Research Building, No. 10, Xi Tu Cheng Road, Beijing, China

    Xu Peng & Bao Xintong

Authors
  1. Xu Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bao Xintong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xu Peng.

Ethics declarations

Conflict of Interests

Authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, X., Xintong, B. An effective strategy for multi-modal fake news detection. Multimed Tools Appl 81, 13799–13822 (2022). https://doi.org/10.1007/s11042-022-12290-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12290-8

Keywords

Search

Navigation