Skip to main content
SpringerLink
Log in

FNR: a similarity and transformer-based approach to detect multi-modal fake news in social media

  • Original Article
  • Published:
Social Network Analysis and Mining Aims and scope Submit manuscript

Abstract

Many people today get their news from social media. It is possible to propagate news using textual, visual, or multi-modal information. The popularity of social networks and their wide use by people make them attractive platforms for spreading fake news. Detecting fake news is essential to preventing its spread. Fake news can be a false article or a genuine article with misleading visual information. Adding an actual image to trustworthy unrelated news can also create a fake news story. In this paper, we propose a novel and efficient similarity and transformer-based detection algorithm called Fake News Revealer (FNR), which uses text and images of news to detect fake news. The algorithm uses contrastive loss to consider text and image relations and transformer models to extract contextual and semantic features. According to experiments on two public social media news data sets, the FNR algorithm competes with conventional methods and state-of-the-art fake news detection algorithms by adding a novel mechanism without adding extra parameters or weights.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. https://twitter.com/.

  2. https://weibo.com/.

  3. http://git.dml.ir/fghorbanpoor/FakeNewsRevealer.

References

  • Akiba T, Sano S, Yanase T et al (2019) Optuna: a next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. ACM, pp 2623–2631

  • Antol S, Agrawal A, Lu J et al (2015) Vqa: visual question answering. In: Proceedings of the IEEE international conference on computer vision, pp 2425–2433

  • Bai Y, Mei J, Yuille AL et al (2021) Are transformers more robust than CNNs? In: Advances in neural information processing systems, pp 26831–26843. arXiv:2111.05464

  • Boididou C, Andreadou K, Papadopoulos S et al (2015) Verifying multimedia use at medieval 2015. MediaEval 3:7

    Google Scholar 

  • Chen J, Jia C, Zheng H et al (2022) Is multi-modal necessarily better? Robustness evaluation of multi-modal fake news detection. arXiv:2206.08788

  • Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American Chapter of the Association for Computational Linguistics, vol 1. Association for Computational Linguistics, pp 4171–4186

  • Dosovitskiy A, Beyer L, Kolesnikov A et al (2021) An image is worth 16x16 words: transformers for image recognition at scale. Preprint at arXiv:2010.11929

  • Farajtabar M, Yang J, Ye X et al (2017) Fake news mitigation via point process-based intervention. In: International conference on machine learning. PMLR, pp 1097–1106

  • Fenn E, Ramsay N, Kantner J et al (2019) Nonprobative photos increase truth, like, and share judgments in a simulated social media environment. JARMAC 8:131–138

    Google Scholar 

  • Fukui A, Park DH, Yang D et al (2016) Multimodal compact bilinear pooling for visual question answering and visual grounding. Preprint at arXiv:1606.01847

  • Gross D (2010) Survey: more Americans get news from internet than newspapers or radio. http://www.cnn.com/2010/TECH/03/01/social.network.news/index.html. Accessed 16 Jan 2020

  • Gupta M, Zhao P, Han J (2012) Evaluating event credibility on twitter. In: Proceedings of the 2012 SIAM international conference on data mining. SIAM, pp 153–164

  • He K, Zhang X, Ren S et al (2015) Deep residual learning for image recognition. Preprint at arXiv:1512.03385

  • He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • Hendrycks D, Gimpel K (2016) Gaussian error linear units (GELUs). Preprint at arXiv:1606.08415

  • Hinton GE, Krizhevsky A, Wang SD (2011) Transforming auto-encoders. In: International conference on artificial neural networks. Springer, pp 44–51

  • Jain V, Kaliyar RK, Goswami A et al (2022) AENeT: an attention-enabled neural architecture for fake news detection using contextual features. Neural Comput Appl 34(1):771–782

    Article  Google Scholar 

  • Jin Z, Cao J, Guo H, et al (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia. ACM, MM \(\acute{1}\)7, pp 795–816

  • Jwa H, Oh D, Park K et al (2019) exBAKE: automatic fake news detection model based on bidirectional encoder representations from transformers (BERT). Appl Sci 9(19):4062

    Article  Google Scholar 

  • Kaliyar RK, Goswami A, Narang P et al (2020) FNDNet—a deep convolutional neural network for fake news detection. Cogn Syst Res 61:32–44

    Article  Google Scholar 

  • Kaliyar RK, Goswami A, Narang P (2021a) EchoFakeD: improving fake news detection in social media with an efficient deep neural network. Neural Comput Appl 33(14):8597–8613

    Article  Google Scholar 

  • Kaliyar RK, Goswami A, Narang P (2021b) FakeBERT: fake news detection in social media with a BERT-based deep learning approach. Multimed Tools Appl 80(8):11765–11788

    Article  Google Scholar 

  • Khattar D, Goud JS, Gupta M et al (2019) Mvae: multimodal variational autoencoder for fake news detection. In: The world wide web conference. ACM, pp 2915–2921

  • Khosla P, Teterwak P, Wang C et al (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33:18661–18673

    Google Scholar 

  • Kumari R, Ekbal A (2021) AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst Appl 184(115):412

    Google Scholar 

  • Kwon S, Cha M, Jung K et al (2013) Prominent features of rumor propagation in online social media. In: 2013 IEEE 13th international conference on data mining. IEEE, pp 1103–1108

  • Liu Y, Wu YF (2018) Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. In: Proceedings of the AAAI conference on artificial intelligence

  • Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. Preprint at arXiv:1711.05101

  • Newman EJ, Garry M, Bernstein DM et al (2012) Nonprobative photographs (or words) inflate truthiness. Psychon Bull Rev 19:969–974

    Article  Google Scholar 

  • O’Brien N, Latessa S, Evangelopoulos G et al (2018) The language of fake news: opening the black-box of deep learning based detectors. In: Workshop on “AI for Social Good”. NIPS

  • Palani B, Elango S, Viswanathan KV et al (2022) CB-Fake: a multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT. Multimed Tools Appl 81:5587–5620

    Article  Google Scholar 

  • Peters M, Neumann M, Iyyer M, Zettlemoyer l et al (2018) Deep contextualized word representations. In: Proceedings of the 2018 conference of the North American Chapter of the Association for Computational Linguistics. Human Language Technologies, pp 2227–2237

  • Ping Tian D et al (2013) A review on image feature extraction and representation techniques. Int J Multimed Ubiquitous Eng 8(4):385–396

    Google Scholar 

  • Qi P, Cao J, Yang T et al (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International Conference on Data Mining (ICDM), pp 518–527

  • Radford A, Kim JW, Hallacy C et al (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763

  • Ren S, He K, Girshick R, et al (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, p 28

  • Ruchansky N, Seo S, Liu Y (2017) Csi: a hybrid deep model for fake news detection. In: Proceedings of the 2017 ACM on conference on information and knowledge management, pp 797–806

  • Salama K (2021) Keras documentation: natural language image search with a dual encoder. https://keras.io/examples/nlp/nl_image_search/. Accessed 8 Nov 2021

  • Shu K, Sliva A, Wang S et al (2017) Fake news detection on social media: a data mining perspective. SIGKDD Explor Newsl 19:22–36

    Article  Google Scholar 

  • Shu K, Wang S, Lee D et al (2020) Disinformation, misinformation, and fake news in social media. Springer

    Book  Google Scholar 

  • Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. Preprint at arXiv:1409.1556

  • Singh B, De S, Zhang Y et al (2015) Layer-specific adaptive learning rates for deep networks. In: 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA). IEEE, pp 364–368

  • Singhal S, Shah RR, Chakraborty T et al (2019) Spotfake: a multi-modal framework for fake news detection. In: 2019 IEEE fifth international conference on multimedia big data (BigMM). IEEE, pp 39–47

  • Singhal S, Pandey T, Mrig S et al (2022) Leveraging intra and inter modality relationship for multimodal fake news detection. In: Companion Proceedings of the Web Conference, pp 726–734

  • Song C, Ning N, Zhang Y et al (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manag 58(102):437

    Google Scholar 

  • Wang S, Chen Y, Zhuo J et al (2018a) Joint global and co-attentive representation learning for image-sentence retrieval. In: Proceedings of the 26th ACM international conference on Multimedia, pp 1398–1406

  • Wang Y, Ma F, Jin Z et al (2018b) EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining. ACM, KDD \(\acute{1}\)8, pp 849–857

  • Wang Y, Ma F, Wang H et al (2021) Multimodal emergent fake news detection via meta neural process networks. In: Proceedings of the 27th ACM SIGKDD conference on Knowledge Discovery & Data Mining. ACM, pp 3708–3716

  • Wang B, Feng Y, Xiong X et al (2022a) Multi-modal transformer using two-level visual features for fake news detection. Appl Intell 2022:1–15

    Google Scholar 

  • Wang J, Mao H, Li H (2022b) FMFN: fine-grained multimodal fusion networks for fake news detection. Appl Sci 12(3):1093

    Article  Google Scholar 

  • Wolf T, Debut L, Sanh V et al (2020) Huggingface’s transformers: state-of-the-art natural language processing. Preprint at arXiv:1910.03771

  • Wu B, Xu C, Dai X et al (2020) Visual transformers: token-based image representation and processing for computer vision. Preprint at arXiv:2006.03677

  • Yenter A, Verma A (2017) Deep CNN-LSTM with combined kernels from multiple branches. In: 2017 IEEE 8th annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON), pp 540–546

Download references

Author information

Author notes

    Authors and Affiliations

    1. Department of Computer Engineering, Sharif University of Technology, Tehran, Iran

      Faeze Ghorbanpour, Maryam Ramezani, Mohammad Amin Fazli & Hamid R. Rabiee

    Authors
    1. Faeze Ghorbanpour
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Maryam Ramezani
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Mohammad Amin Fazli
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Hamid R. Rabiee
      View author publications

      You can also search for this author in PubMed Google Scholar

    Contributions

    BG did the conceptualization, formal analysis, Investigation, methodology, software and writing the original draft. MR did the conceptualization, formal analysis, Investigation, methodology, writing, review & editing. MFA did the supervision, conceptualization, formal analysis, review & editing. HRR did the supervision, project administration, conceptualization, methodology, validation, review & editing.

    Corresponding author

    Correspondence to Hamid R. Rabiee.

    Ethics declarations

    Conflict of interest

    The authors declare no competing interests.

    Additional information

    Publisher’s Note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Faeze Ghorbanpour and Maryam Ramezani contributed equally to this study.

    Rights and permissions

    Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Ghorbanpour, F., Ramezani, M., Fazli, M.A. et al. FNR: a similarity and transformer-based approach to detect multi-modal fake news in social media. Soc. Netw. Anal. Min. 13, 56 (2023). https://doi.org/10.1007/s13278-023-01065-0

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • DOI: https://doi.org/10.1007/s13278-023-01065-0

    Keywords

    Search

    Navigation