Skip to main content
SpringerLink
Log in

Emotion aided multi-task framework for video embedded misinformation detection

  • 1230: Sentient Multimedia Systems and Visual Intelligence
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Online news consumption via social media platforms has accelerated the growth of digital journalism. Adverse to traditional media, digital media has lower entry barriers and allows everyone as a content creator, resulting in numerous fake news productions to attract public attention. As multimedia content is more convenient for users than expressing their feelings through text, images and video-embedded fake news is being circulated rapidly on social media nowadays. Emotional appeal in fake news is also a driving factor in its rapid dissemination. Although prior studies have made a remarkable effort toward fake news detection, they give less emphasis on exploring video modality and emotional appeal in fake news. To bridge this gap, this paper presents the following two contributions: i) It first develops a video-based multimodal fake news detection dataset named FakeClips and ii) It introduces a deep multitask framework dedicated to video-embedded multimodal fake news detection in which fake news detection is the main task and emotion recognition is the auxiliary task. The results reveal that investigating emotion and fake news together in a multitasking framework achieves 9.04% and 5.27% gains in terms of accuracy and f-score, respectively over the state-of-the-art model i.e. Fake Video Detection Model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Code Availability

Data and source code will be made publicly available after acceptance of this manuscript. However, before accessing the data and code, the reader must fill out a consent form declaring that the dataset will only be used for research purposes.

Notes

  1. https://www.indiatoday.in/fact-check/story/man-clinging-on-to-wing-afghanistan-taliban-kabul-airport-1842101-2021-08-18

  2. https://www.boomlive.in

  3. https://www.geeksforgeeks.org/implementing-web-scraping-python-beautiful-soup/

  4. https://www.boomlive.in/

  5. https://sites.google.com/site/linkgopher/

  6. https://ostechnix.com/youtube-dl-tutorial-with-examples-for-beginners/

  7. https://pypi.org/project/beautifulsoup4/

  8. https://huggingface.co/mrm8488/t5-base-finetuned-emotion

  9. https://github.com/kkroening/ffmpeg-python/blob/master/examples/README.md

  10. https://pypi.org/project/opencv-python/

  11. https://www.analyticsvidhya.com/blog/2021/06/mfcc-technique-for-speech-recognition/

  12. https://pypi.org/project/ffmpeg-python/

References

  1. Ma J, Gao W, Joty S, Wong K-F (2019)Sentence-level evidence embedding for claim verification with hierarchical attention networks. In:Proceedings of the 57th annual meeting of the association for computational linguistics, pp 2561–2571

  2. Potthast M, Kiesel J, Reinartz K, Bevendorff J, Stein B (2018) A stylometric inquiry into hyperpartisan and fake news. In: ACL (1)

  3. Chakraborty T (2021) Multi-modal fake news detection, pp 41–70. Springer. https://doi.org/10.1007/978-3-030-62696-9_3

  4. IndiaToday (2021) Fact Check: Man clinging on to a wing and a prayer? No, there is a catch - Fact Check News. https://www.indiatoday.in/fact-check/story/man-clinging-on-to-wing-afghanistan-taliban-kabul-airport-1842101-2021-08-18. (Accessed on 02/01/2022)

  5. Boomlive (2022) Video Of Pakistan Customs Destroying Cellphones Peddled As Afghanistan. https://www.boomlive.in/fact-check/world/pakistan-customs-destroying-cellphones-video-peddled-as-afghanistan-16549. (Accessed on 02/01/2022)

  6. Brady WJ, Wills JA, Jost JT, Tucker JA, Van Bavel JJ (2017) Emotion shapes the diffusion of moralized content in social networks. Proc Natl Acad Sci 114(28):7313–7318

    Article  Google Scholar 

  7. Vosoughi S, Roy D, Aral S (2018) The spread of true and false news online. Science 359(6380):1146–1151

    Article  Google Scholar 

  8. Papadopoulou O, Zampoglou M, Papadopoulos S, Kompatsiaris Y, Teyssou D (2018) Invid fake video corpus v2. 0 (version 2.0). Dataset on Zenodo

  9. Palod P, Patwari A, Bahety S, Bagchi S, Goyal P (2018) Misleading metadata detection on youtube. In: European conference on information retrieval, pp 140–147 (2019). Springer

  10. Choi H, Ko Y (2022) Effective fake news video detection using domain knowledge and multimodal data fusion on youtube. Pattern Recogn Lett

  11. Segura-Bedmar I, Alonso-Bartolome S (2022) Multimodal fake news detection. Inf 13(6):284

    Google Scholar 

  12. John A, Meenakowshalya A (2021) Fake news detection using n-gram analysis and machine learning algorithms. J Mob Comput, Commun Mob & Netw 8(1):33–43

    Google Scholar 

  13. Asghar MZ, Habib A, Habib A, Khan A, Ali R, Khattak A (2021) Exploring deep neural networks for rumor detection. J Ambient Intell Humaniz Comput 12(4):4315–4333

    Article  Google Scholar 

  14. Yu F, Liu Q, Wu S, Wang L, Tan T, et al (2017) A convolutional approach for misinformation identification. In: IJCAI, pp 3901–3907

  15. Ma J, Gao W, Wong K-F (2018) Rumor detection on twitter with tree-structured recursive neural networks. In: ACL (1)

  16. Zhang J, Dong B, Philip SY (2020) Fakedetector: Effective fake news detection with deep diffusive neural network. In: 2020 IEEE 36th international conference on data engineering (ICDE) pp 1826–1829. IEEE

  17. Shu K, Cui L, Wang S, Lee D, Liu H (2019) Defend: Explainable fake news detection. In: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp 395–405

  18. Fang Y, Gao J, Huang C, Peng H, Wu R (2019) Self multi-head attention-based convolutional neural networks for fake news detection. PloS one 14(9):0222713

    Article  Google Scholar 

  19. Saikh T, De A, Ekbal A, Bhattacharyya P (2020) A deep learning approach for automatic detection of fake news. In: Proceedings of the 16th international conference on natural language processing, pp 230–238

  20. Rao D, Miao X, Jiang Z, Li R (2021) Stanker: Stacking network based on level-grained attention-masked bert for rumor detection on social media. In: Proceedings of the 2021 conference on empirical methods in natural language processing, pp 3347–3363

  21. Ghadiri Z, Ranjbar M, Ghanbarnejad F, Raeisi S (2022) Automated fake news detection using cross-checking with reliable sources. arXiv:2201.00083

  22. Jin Z, Cao J, Zhang Y, Zhou J, Tian Q (2016) Novel visual and statistical image features for microblogs news verification. IEEE Trans Multimedia 19(3):598–608

    Article  Google Scholar 

  23. Qi P, Cao J, Yang T, Guo J, Li J (2019) Exploiting multi-domain visual information for fake news detection. In: 2019 IEEE International conference on data mining (ICDM) pp 518–527. IEEE

  24. Jin Z, Cao J, Guo H, Zhang Y, Luo J (2017) Multimodal fusion with recurrent neural networks for rumor detection on microblogs. In: Proceedings of the 25th ACM international conference on multimedia, pp 795–816

  25. Wang Y, Ma F, Jin Z, Yuan Y, Xun G, Jha K, Su L, Gao J (2018) Eann: Event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th Acm Sigkdd international conference on knowledge discovery & data mining, pp 849–857

  26. Khattar D, Goud JS, Gupta M, Varma V (2019) Mvae: Multimodal variational autoencoder for fake news detection. In: The world wide web conference, pp 2915–2921

  27. Singhal S, Shah RR, Chakraborty T, Kumaraguru P, Satoh S (2019) Spotfake: A multi-modal framework for fake news detection. In: 2019 IEEE Fifth international conference on multimedia big data (BigMM) pp 39–47. IEEE

  28. Singh VK, Ghosh I, Sonagara D (2021) Detecting fake news stories viaDetecting fake news stories via multimodal analysis. J Assoc Inf Sci Technol 72(1):3–17

    Article  Google Scholar 

  29. Kumari R, Ekbal A (2021) Amfb: Attention based multimodal factorized bilinear pooling for multimodal fake news detection. Expert Syst Appl 184:115412

    Article  Google Scholar 

  30. Song C, Ning N, Zhang Y, Wu B (2021) A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks. Inf Process Manage 58(1):102437

    Article  Google Scholar 

  31. Wang J, Mao H, Li H (2022) Fmfn: Fine-grained multimodal fusion networks for fake news detection. Appl Sci 12(3):1093

    Article  Google Scholar 

  32. Song C, Ning N, Zhang Y, Wu B (2021) Knowledge augmented transformer for adversarial multidomain multiclassification multimodal fake news detection. Neurocomputing 462:88–100

    Article  Google Scholar 

  33. Choi H, Ko Y (2021) Using topic modeling and adversarial neural networks for fake news video detection. In: Proceedings of the 30th ACM international conference on information & knowledge management, pp 2950–2954

  34. Zhang X, Cao J, Li X, Sheng Q, Zhong L, Shu K (2021) Mining dual emotion for fake news detection. Proceedings of the Web Conference 2021:3465–3476

    Google Scholar 

  35. Guo C, Cao J, Zhang X, Shu K, Yu M (2019) Exploiting emotions for fake news detection on social media

  36. Kumari R, Ashok N, Ghosal T, Ekbal A (2021) Misinformation detection using multitask learning with mutual learning for novelty detection and emotion recognition. Inf Process Manage 58(5):102631

    Article  Google Scholar 

  37. Kumari R, Ashok N, Ghosal T Ekbal A (2021) A multitask learning approach for fake news detection: Novelty, emotion, and sentiment lend a helping hand. In: 2021 International joint conference on neural networks (IJCNN) pp 1–8 . IEEE

  38. Kumari R, Ashok N, Ghosal T, Ekbal A (2022) What the fake? probing misinformation detection standing on the shoulder of novelty and emotion. Inf Process Manage 59(1):102740

    Article  Google Scholar 

  39. Gupta V, Kumari R, Ashok N, Ghosal T, Ekbal A (2022) Mmm: An emotion and novelty-aware approach for multilingual multimodal misinformation detection. Findings of the association for computational linguistics: AACL-IJCNLP 2022:464–477

    Google Scholar 

  40. Kumari R, Ashok, N, Agrawal PK, Ghosal T, Ekbal A (2023) Identifying multimodal misinformation leveraging novelty detection and emotion recognition. Journal of Intelligent Information Systems, pp 1–22

  41. Singhal S, Dhawan M, Shah RR, Kumaraguru P (2021) Inter-modality discordance for multimodal fake news detection. In: ACM Multimedia Asia, pp 1–7

  42. Zhang W, Gui L, He Y (2021) Supervised contrastive learning for multimodal unreliable news detection in covid-19 pandemic. In: Proceedings of the 30th ACM international conference on information & knowledge management, pp 3637–3641

  43. Zimdars M (2016) Resource-False-Misleading-Clickbait-y-and-Satirical-“News” -Sources-1.pdf. https://docs.google.com/document/d/10eA5-mCZLSS4MQY5QGb5ewC3VAL6pLkT53V_81ZyitM/preview. (Accessed on 05/17/2021)

  44. Jindal S, Sood R, Singh R, Vatsa M, Chakraborty T (2020) Newsbag: A multimodal benchmark dataset for fake news detection

  45. Mezaris V (2016) InVID Verification Plugin - InVID project. https://www.invid-project.eu/tools-and-services/invid-verification-plugin/. (Accessed on 12/26/2021)

  46. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  47. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21:1–67

    MathSciNet  Google Scholar 

  48. Saravia E, Liu H-CT, Huang Y-H, Wu J, Chen Y-S (2018) Carer: Contextualized affect representations for emotion recognition. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 3687–3697

  49. Livingstone SR, Russo FA (2018) The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one 13(5):0196391

    Article  Google Scholar 

  50. Khosla P, Teterwak P, Wang C, Sarna A, Tian Y, Isola P, Maschinot A, Liu C, Krishnan D (2020) Supervised contrastive learning. Adv Neural Inf Process Syst 33

  51. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv:1412.6980

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, 751024, India

    Rina Kumari

  2. Department of Computer Science and Engineering, Indian Institute of Technology, Patna, 801106, Bihar, India

    Vipin Gupta & Asif Ekbal

  3. Department of Computer Science, University of Massachusetts, Amherst, USA

    Nischal Ashok

  4. National Center of Computational Sciences, Oak Ridge, National Laboratory, Tennessee, USA

    Tirthankar Ghosal

Authors
  1. Rina Kumari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vipin Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nischal Ashok
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tirthankar Ghosal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Asif Ekbal
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Rina Kumari - Data curation, Resources, Formal Analysis, Investigation, Methodology, Writing Original Draft; Vipin Gupta - Investigation, Methodology, Writing Original Draft; Nischal Ashok - Data Curation, Resources, Formal Analysis, Investigation, Methodology; Pawan Kumar Agrawal - Data Curation, Resources Tirthankar Ghosal - Conceptualization, Writing Review & Editing; Asif Ekbal - Conceptualization, Supervision, Writing Review & Editing

Corresponding author

Correspondence to Rina Kumari.

Ethics declarations

Conflicts of interest

The authors have no conflicts of interest to declare relevant to the content of this article.

Ethical

The developed resource for this work includes publicly available videos on YouTube and their metadata. We have accompanied the policies of using YouTube video data and have not harmed any copyright issues. Our dataset also does not contain any sensitive information and does not disclose and affect anyone’s personal information and sentiment. We will make available the complete dataset only after filling and signing an agreement declaring that the data will be used only for research purposes.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumari, R., Gupta, V., Ashok, N. et al. Emotion aided multi-task framework for video embedded misinformation detection. Multimed Tools Appl 83, 37161–37185 (2024). https://doi.org/10.1007/s11042-023-17208-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-17208-6

Keywords

Search

Navigation