Abstract
The world is witnessing a growing epidemic of misinformation. Misinformation can have severe impacts on society across multiple domains: including health, politics, security, the environment, the economy and education. With the constant spread of misinformation on social media networks, a need has arisen to continuously assess the veracity of digital content. This need has inspired numerous research efforts on the development of misinformation detection (MD) models. However, many models do not use all information available to them and existing research contains a lack of relevant datasets to train the models, specifically within the South African social media environment. The aim of this paper is to investigate the transferability of knowledge of a MD model between different contextual environments. This research contributes a multimodal MD model capable of functioning in the South African social media environment, as well as introduces a South African misinformation dataset. The model makes use of multiple sources of information for misinformation detection, namely: textual and visual elements. It uses bidirectional encoder representations from transformers (BERT) as the textual encoder and a residual network (ResNet) as the visual encoder. The model is trained and evaluated on the Fakeddit dataset and a South African misinformation dataset. Results show that using South African samples in the training of the model increases model performance, in a South African contextual environment, and that a multimodal model retains significantly more knowledge than both the textual and visual unimodal models. Our study suggests that the performance of a misinformation detection model is influenced by the cultural nuances of its operating environment and multimodal models assist in the transferability of knowledge between different contextual environments. Therefore, local data should be incorporated into the training process of a misinformation detection model in order to optimize model performance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Alonso-Bartolome, S., Segura-Bedmar, I.: Multimodal fake news detection. arXiv (2021)
Bentzen, N.: Trump’s disinformation ‘magaphone’: consequences, first lessons and outlook (2021). https://www.europarl.europa.eu/. Accessed from European Parliament: 20 Aug 2023
Brownlee, J.: A gentle introduction to concept drift in machine learning (2017). https://machinelearningmastery.com/gentle-introduction-concept-drift-machine-learning/. Accessed from Machine Learning Mastery: 20 Aug 2023
Capuano, N., Fenza, G., Loia, V., Nota, F.D.: Content-based fake news detection with machine and deep learning: a systematic review. Neurocomputing 530, 91–103 (2023)
Figueira, A., Oliveira, L.: The current state of fake news: challenges and opportunities. Procedia Comput. Sci. 121, 817–825 (2017)
Khattar, D., Goud, J.S., Gupta, M., Varma, V.: MVAE: multimodal variational autoencoder for fake news detection. In: The World Wide Web Conference, pp. 2915–2921. Association for Computing Machinery, New York (2019)
Lapping, G.: The July riots: an inflection point for digital media (2021). https://www.businesslive.co.za/redzone/news-insights/2021-09-01-the-july-riots-an-inflection-point-for-digital-media/. Accessed from Business Live: 20 Aug 2023
Nakamura, K., Levy, S., Wang, W.Y.: Fakeddit: a new multimodal benchmark dataset for fine-grained fake news detection. In: Proceedings of the Twelfth Language Resources and Evaluation Conference, Marseille, France, pp. 6149–6157. European Language Resources Association (2020)
Palani, B., Elango, S., Viswanathan, K.V.: CB-fake: a multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT. Multimed. Tools Appl. 81(4), 5587–5620 (2022)
Raza, S., Ding, C.: Fake news detection based on news content and social contexts: a transformer-based approach. Int. J. Data Sci. Anal. 13(4), 335–362 (2022)
Real411: Home (2023). https://www.real411.org/. Accessed from Real 411: 20 Aug 2023
Shu, K., Mahudeswaran, D., Wang, S., Lee, D., Liu, H.: FakeNewsNet: a data repository with news content, social context and spatialtemporal information for studying fake news on social media. arXiv (2019)
Singhal, S., Kabra, A., Sharma, M., Shah, R.R., Chakraborty, T., Kumaraguru, P.: SpotFake+: a multimodal framework for fake news detection via transfer learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 10, pp. 13915–13916 (2020)
Wardle, C.: Fake news. it’s complicated (2017). https://firstdraftnews.org/articles/fake-news-complicated/. Accessed from First Draft News: 20 Aug 2023
WHO: Infodemic (2023). https://www.who.int/health-topics/infodemic/understanding-the-infodemic-and-misinformation-in-the-fight-against-covid-19. Accessed from World Health Organization: 20 Aug 2023
Wu, L., Morstatter, F., Carley, K.M., Liu, H.: Misinformation in social media: definition, manipulation, and detection. SIGKDD Explor. Newsl. 21(2), 80–90 (2019)
Yuan, Y., et al.: EANN: event adversarial neural networks for multi-modal fake news detection. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2018, pp. 849–857. Association for Computing Machinery, New York (2018)
Zhang, X., Ghorbani, A.A.: An overview of online fake news: characterization, detection, and discussion. Inf. Process. Manag. 57(2), 102025 (2020)
Zhou, Y., Yang, Y., Ying, Q., Qian, Z., Zhang, X.: Multimodal fake news detection via clip-guided learning. In: 2023 IEEE International Conference on Multimedia and Expo (ICME), Los Alamitos, CA, USA, pp. 2825–2830. IEEE Computer Society (2023)
Acknowledgements
We want to acknowledge Real411, a program of Media Monitoring Africa, for the dataset. We would like to acknowledge funding from the ABSA Chair of Data Science, Google and the NVIDIA Corporation hardware Grant.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
De Jager, A., Marivate, V., Modupe, A. (2023). Multimodal Misinformation Detection in a South African Social Media Environment. In: Pillay, A., Jembere, E., J. Gerber, A. (eds) Artificial Intelligence Research. SACAIR 2023. Communications in Computer and Information Science, vol 1976. Springer, Cham. https://doi.org/10.1007/978-3-031-49002-6_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-49002-6_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-49001-9
Online ISBN: 978-3-031-49002-6
eBook Packages: Computer ScienceComputer Science (R0)