Transformer-based models for combating rumours on microblogging platforms: a review

Abstract

The remarkable success of Transformer-based embeddings in natural language tasks has sparked interest among researchers in applying them to classify rumours on social media, particularly microblogging platforms. Unlike traditional word embedding methods, Transformers excel at capturing a word’s contextual meaning by considering words from both the left and right of a word, resulting in superior text representations ideal for tasks like rumour detection on microblogging platforms. This survey aims to provide a thorough and well-organized overview and analysis of existing research on implementing Transformer-based models for rumour detection on microblogging platforms. The scope of this study is to offer a comprehensive understanding of this topic by systematically examining and organizing the existing literature. We start by discussing the fundamental reasons and significance of automating rumour detection on microblogging platforms. Emphasizing the critical role of text embedding in converting textual data into numerical representations, we review current approaches to implement Transformer models for rumour detection on microblogging platforms. Furthermore, we present a novel taxonomy that covers a wide array of techniques and approaches employed in the deployment of Transformer-based models for identifying misinformation on microblogging platforms. Additionally, we highlight the challenges associated with this field and propose potential avenues for future research. Drawing insights from the surveyed articles, we anticipate that promising results will continue to emerge as the challenges outlined in this study are addressed. We hope that our efforts will stimulate further interest in harnessing the capabilities of Transformer models to combat the spread of rumours on microblogging platforms.

1 Introduction

The emergence of internet technology and the widespread use of online social media have revolutionised communication. Microblogging social media platforms like Twitter (currently know as X), Tumblr, Instragram, etc. have become powerful tools for information sharing, networking, and real-time communication. Users can share concise messages with their followers, engage in conversations, and stay updated on breaking news and trending topics.

However, the accessibility and fast-paced nature of microblogging on social media can also contribute to the spread of false information and rumours. The platforms’ quick sharing capabilities allow rumours to spread rapidly before they can be manually verified or corrected. The lack of strict verification processes and the algorithmic amplification of engaging conversation further exacerbate the issue. As a result, rumours on microblogging platforms can range from accurate to completely fabricated, causing unease and leading to irrational actions among those who follow them.

Several notable examples illustrate the impact of rumours on microblogging platforms. In 2013, a fake tweet on platform X about explosions at White House and President Obama’s injuries caused a temporary dip in the stock market, resulting in significant financial losses totalling to USD 136.5 million (Guardian 2013; Godsey et al 2021). Similarly, a false rumour in 2017 about the forceful removal of a passenger from a United Airlines flight resulted in a significant drop in the company’s stock price, causing nearly USD 1 billion of the company’s value to be wiped out in trading on the following day (Guardian 2017).

The "Pizzagate" incident exemplifies how false rumours disseminated as a tweet on X can incite irrational actions, such as an armed individual entering a restaurant due to unfounded allegations of high-level Democrat party officials’ involvement in a child sex trafficking ring (Times 2016; Cosentino 2020). Other more recent illustrations highlight the detrimental impact of spurious COVID-19-related rumours, which are believed to have contributed to the deaths of a minimum of 800 individuals globally during the initial three months of 2020. The same study also approximates that approximately 5,800 individuals were hospitalized due to misinformation disseminated on social media platforms (Islam et al 2020b). In the context of the Ukrainian war, scholars draw attention to the phenomenon where few social media users have generated counterfeit live streams depicting Ukraine-Russian conflicts by repurposing archived footage from unrelated incidents. These deceptive streams gave the illusion of authenticity and misled the audience. Subsequently, this fabricated information is shared on microblogging platforms and leveraged to solicit donations and contributions, with the final beneficiaries being the creators themselves (Stănescu 2022).

These examples illustrate that in today’s digital landscape, microblogging platforms have become a breeding ground for countless rumours across various topics, with the potential to influence people’s views and actions significantly. These rumours can disrupt financial markets, tarnish reputations, and fuel political unrest. Therefore, it is imperative to develop automated and intelligent tools to detect the veracity of information shared on microblogging platforms and identify rumours, as manual fact-checking is a laborious and daunting task. For this aim, many methods have been proposed in the literature to detect and prevent the spread of rumours over the past decade by leveraging the content and social context of the messages.

Rumours detection on microblogging platforms presents several inherent challenges that complicate the identification and mitigation of false information. These challenges stem from the unique characteristics of microblogging platforms, which include quick and unverified information sharing among users in real time. The sheer volume of content generated on these platforms makes it impossible for human moderation. Additionally, the diverse forms of misinformation generated in rumours by including conspiracy theories, and fake and manipulated media, further exacerbate the challenge of distinguishing between genuine information and false rumours.

One of the primary difficulties in rumour detection on microblogging platforms is the contextual ambiguity inherent in short, often cryptic posts. Microblogging users frequently express themselves using abbreviated language and emojis, making it challenging to discern the true intent and veracity of the information shared. Rumours can be disguised as genuine news or opinions, and their spread can be amplified by bot accounts and viral networks, complicating the detection process further.

The goal of rumour detection on microblogging platforms is to determine if a post contains a rumour and to provide evidence-based information to counteract the dissemination of misinformation. To accomplish this objective, it is vital to extract features from the posts that capture the semantic and syntactic attributes of rumour content. Since the primary component of these posts is text, employing effective text encoding methods is crucial for transforming raw text data into numerical representations that computer machines can effectively process.

Transformer-based models offer a promising solution to these challenges. Their success in the recent revolution of Large Language Model (LLM) in our daily lives such as ChatGPT, Google Gemini and MS Co-Pilot has made them the post popular model in the existing solutions. The Transformer-based models excel at capturing semantic relationships and contextual nuances in text (Vaswani et al 2017; Devlin et al 2018), enabling more accurate analysis of microblogging posts. By considering the broader context of a conversation or thread, Transformer-based models can better discern the intent and veracity of the information shared. Additionally, these models can integrate information from multiple modalities, including text, images, and user metadata, to provide a more comprehensive understanding of the content and context of a post, enhancing detection accuracy.

Recent studies have highlighted the effectiveness of Transformer-based models in text classification and misinformation verification tasks, positioning them as state-of-the-art (SOTA) models (Kula et al 2021; Schütz et al 2021; Sharma et al 2022b). These models capture the contextual meaning of words by considering both their preceding and succeeding contexts, thereby generating more comprehensive textual representations. Table 1 presents the SOTA models for rumour detection using Transformer models.

Table 1 The performance of the state-of-the-art Transformer-based models in rumour classification and their improvements compared to the baseline models

While Transformer-based models have demonstrated impressive performance in various tasks, implementing Transformers to address the issue of rumour propagation on microblogging social media remains a significant challenge that requires further attention and solutions. We acknowledge the existence of the survey articles on misinformation detection on social media platforms. However, we have observed a deficiency in a survey article that delves in-depth by focusing on the implementation of Transformer-based models for detecting rumours on microblogging platforms.

We believe that our research fills a significant gap in the literature by focusing specifically on the implementation of Transformer-based models including BERT, which is most popular Transformer model, used for rumour detection on microblogging platforms. It involves gathering and synthesizing information from various sources to summarize the current state of knowledge, identify trends, highlight gaps, and propose future research directions. The primary objective is to provide a holistic understanding of the subject matter by systematically and coherently examining and organizing the existing literature.

As mentioned earlier, several survey articles on misinformation detection on social media have been published (Bondielli and Marcelloni 2019; Sharma et al 2019; Islam et al 2020a; Collins et al 2021; Varshney and Vishwakarma 2021; Hu et al 2022; Varlamis et al 2022; Hangloo and Arora 2022; Tan et al 2023; Ghani Khan et al 2023; Phan et al 2023), providing a comprehensive overview and analysis of existing research studies. An overview of the relevant literature is as follows: Bondielli and Marcelloni (2019) and Varshney and Vishwakarma (2021) conducted a survey that explored the various methodologies proposed in the literature for automatically detecting fake news and rumours. The study examined the diverse definitions of rumours, data collection techniques, the range of features considered, and the different approaches employed to detect rumours. Sharma et al (2019) and Hangloo and Arora (2022) addressed the technical challenges associated with detecting fake news and discussed the existing methods and techniques, emphasising the advantages and limitations of each approach in their survey. Additionally, they provided a summary of the characteristic features of available datasets and outlined new research directions to foster the future development of practical and interdisciplinary solutions. Islam et al (2020a), Hu et al (2022), and Tan et al (2023) conducted a review of misinformation detection, focusing on applying deep learning techniques for automatic data processing and decision-making. They highlighted the effectiveness and scalability of deep learning in achieving state-of-the-art results. Collins et al (2021) examined different forms of fake news and reviewed recent advancements in strategies aimed at curtailing the dissemination of false information on social media platforms. Varlamis et al (2022) and Phan et al (2023) conducted surveys that specifically examine the popular and promising graph representation techniques employed in the detection of fake news disseminated within social networks. Although Hu et al (2022), Hangloo and Arora (2022), and Tan et al (2023) have mentioned Transformer-based models in their survey articles, they only mentioned that them as one of the state-of-the-art models for natural language processing tasks which are used for rumour detection, along with other deep learning models without providing without delving deep that how Transformers are applied to this task.

In contrast to previous studies, our survey focuses on the depiction, analysis, and discourse surrounding the latest Transformer-based classification systems used in rumour detection, particularly on microblogging platforms. We believe that this article serves as an essential and foundational point of reference for new researchers and practitioners in navigating current challenges and shaping future directions for enhancing the performance of rumour detection systems utilising Transformer models.

This article provides significant contributions in the following areas:

• We offer a comprehensive survey of Transformer-based classification models’ implementation for rumour detection on microblogging platforms, covering characteristics, related features, approaches, and benchmark datasets.

• We outline the key concepts and methodologies of various Transformer-based classification systems for rumour detection, providing researchers with a foundation and guidance for creating new Transformer-based classification models.

• We present a novel taxonomy that encompasses various methodologies and strategies used in implementing Transformer-based models for detecting misinformation on microblogging platforms. This taxonomy offers a comprehensive overview of the field from both theoretical and practical standpoints.

• We discuss challenges and open issues in rumour detection on microblogging platforms using Transformer-based models, conducting a detailed analysis of each problem and suggesting future research directions, particularly concerning model depth and scalability trade-offs.

The rest of the paper is structured as follows: In Sect. 2, we delve into the background of microblogging platforms, rumour detection, and Transformer architecture. Section 3 elaborates on our survey methodology for implementing a Transformer-based model to detect rumours on microblogging platforms. In Sect. 4, we conduct both quantitative and qualitative analyses of eligible articles, offering detailed insights into their methodologies, strengths, and weaknesses. We also provide significant insights into our survey findings, accompanied by the introduction of a novel taxonomy for implementing such models to combat misinformation on microblogging platforms. Section 5 explores the challenges of implementing Transformer models for rumour detection on microblogging platforms and suggests future research directions. Lastly, Sect. 6 concludes the study’s findings with concluding remarks.

2 Background

This section presents background information on microblogging on social media. Additionally, this section illustrates how researchers attempted to address the spread of rumours by detecting them at the early stages of propagation. Lastly, this section explains the technical details of Transformer-based models and their variants.

2.1 Microblogging on social media

The advent of internet technology and online social media has ushered in a revolutionary change in communication due to their widespread usage. With the accessibility and user-friendly nature of social media, individuals can quickly access the latest news, share content (with or without verification), and express their views. The term “micro” in microblogging denotes the limited size of individual posts, which are often akin to snippets or micro-updates. This format contrasts with traditional blogging, where authors have more space to elaborate on topics in greater detail. Microblogging platforms, such as Twitter, Weibo, and Tumblr, prioritize quick and immediate communication, making them ideal tools for sharing concise messages, engaging in real-time conversations, and keeping up with the latest trends and developments. The succinct nature of microblogging facilitates the rapid dissemination of information and encourages users to engage with bite-sized content.

The most popular microblogging platforms is Twitter, a robust tool for communication, information sharing, and networking. Recently, Twitter officially rebranded itself as ’X,’ introducing a new identity. Since August 2023, Twitter has replaced its iconic blue bird logo with a stylized “X” logo and removed the Twitter name from its websites, apps, and headquarters. Despite the name change to “X”, its primary website address remains twitter.com as of November 2023, with the x.com domain name redirecting to that address, and there is no significant difference in policy and features for users.

Users on this platform can share short messages called “tweets” with their followers. Given that tweets are limited to 280 characters, users are encouraged to be concise and to the point, aligning with Twitter’s fast-paced, real-time communication nature. The platform is frequently utilized for breaking news, live events, and discussions around trending topics. Twitter users can follow others to see their tweets in their timelines and can also like, retweet, or reply to tweets. Its unique format and features make it the most popular platform for individuals, businesses, and organizations alike. According to a recent statistical report released in February 2024, Twitter currently boasts 528.3 million monetizable monthly active users in 2023, projected to reach 652.23 million by 2028. Additionally, there are 237.8 million daily active users (DAU) on Twitter (Sage 2022).

Another popular microblogging platform is Weibo. In terms of policy, Weibo shares similarities with Twitter, particularly regarding content moderation and censorship. In social media, Weibo stands out as a pivotal platform shaping digital discourse in China. At its core, Weibo is a microblogging platform similar to Twitter, allowing users to post short messages and multimedia content, and engage in real-time conversations. Based on the last report released in February 2024, as of the third quarter of 2023, Weibo boasts 605 million monthly active users (MAUs) and 260 million average daily active users (DAUs), making it one of the most-used platforms in the world.

While Microblogging platforms offer numerous benefits, they can also lead to the spread of false information or rumours for various reasons. Firstly, a microblogging platform is a fast-paced platform where users can quickly share and consume information, leading to rumours spreading rapidly before they can be fact-checked or corrected. Secondly, the ease of sharing through retweeting, forwarding or reblogging, allow users to quickly disseminate posts to their followers, resulting in a rumour being shared multiple times in a short period of time. Additionally, users can create new posts that are similar to the original, further amplifying the rumour. Thirdly, the microblogging platforms do not have a strict verification process for information shared on the platform, leading to disseminating false information without any consequences. Furthermore, the platform’s algorithmic amplification can promote messages that receive more engagement, such as likes, retweets, and comments, leading to rumours being amplified even further and reaching a larger audience.

2.2 Rumour detection on microblogging platforms

There are various definitions of rumour. This study adopts the definition from the Cambridge Dictionary, where a rumour is described as unofficial news or a story that might be either true or fabricated and rapidly spreads from one individual to another (Combley 2011). Consequently, rumours can be accurate, partially accurate, untrue, or unverifiable (Bondielli and Marcelloni 2019). They circulate with uncertain reliability, fostering unease and prompting illogical actions among those who hear or read them (Zubiaga et al 2014; Pamungkas et al 2019).

Most existing studies in the rumour detection area utilised features from both the text content of tweets and their surrounding information, including users’ and network’s information. The features extracted from the text are called content-based features, which include linguistic and lexical characteristics such as doubt words, negation, clear expression, and abbreviations that can indicate a message’s trustworthiness. On the other hand, the features extracted from the users and network information surrounding a message are called context-based features, examples of which include the number of followers, posts, and retweets/resharing. Therefore, we can generally classify rumour detection techniques on microblogging platforms into content-based and context-based approaches.

The content of the text has been extensively analysed in literature using various Natural Language Processing (NLP) methods at different levels. These methods include analysing the words used in the text, such as word counts, TF-IDF, bi-grams, and word embedding. At the sentence level, features such as syntax and complexity are used (Castillo et al 2011; Ito et al 2015; Hassan and Haggag 2018; Herzallah et al 2018; Benamira et al 2019). Additionally, the text as a whole is analyzed using features like topic or sentiment analysis (Ito et al 2015; Cui et al 2019; Islam et al 2019). However, these text-processing methods cannot recognize the order and context of words and are limited in capturing complex syntactic or semantic patterns.

On the other hand, context-based features are extracted from the information of users and their network surroundings. This includes details such as the user’s account description, the number of friends and followers, and the counts of posts and retweets/resharing (Castillo et al 2011; Ito et al 2015; Hassan and Haggag 2018; Herzallah et al 2018; Wu et al 2020b, a; Huang et al 2020; Bian et al 2020; Liu et al 2022; Wei et al 2022; Ran et al 2022). Earlier literature on rumour detection on microblogging platforms employed user profiling to extract context-based features from the platform by analyzing user attributes such as follower/following relationships, account age, frequency of post, and history of the post (Castillo et al 2011; Ito et al 2015; Hassan and Haggag 2018; Herzallah et al 2018). This information was used to assess the credibility and reliability of users spreading the information. Few studies leveraged propagation and social network analysis techniques to extract context-based features. This involved examining the retweet and mention networks to identify influential users, hubs, and communities (Wu et al 2020b, a; Huang et al 2020; Bian et al 2020; Liu et al 2022). These approaches aim to provide a comprehensive understanding of the context surrounding the posted message. Context-based features can improve accuracy; however, obtaining context-based information is challenging as it requires additional mining using Application Programming Interfaces (APIs) to retrieve more comprehensive details about the user and their networks. All the existing published datasets only contain message IDs due to microblogging platform’s confidentiality and privacy restrictions.

In the early literature on rumour detection on microblogging platforms, researchers used traditional supervised machine learning techniques to identify rumours, including Support Vector Machine (SVM), Logistic Regression (LR), K-Nearest Neighbour (KNN), and Naive Bayes Classifier (NBC) (Castillo et al 2011; Ito et al 2015; Hassan and Haggag 2018; Herzallah et al 2018; Sato et al 2018). However, a major difficulty encountered in many of these works was extracting features (content-based as well as context-based) from the datasets. The manual extraction of features is a labour-intensive and time-consuming process that often yields suboptimal results in identifying rumours.

Later, deep neural network (DNN) methods were introduced due to their promising results in numerous feature extraction and classification tasks including natural language processing (NLP). Moreover, DNN approaches offer the advantage of simplifying the feature extraction process while providing robust abstract representation. As a result, many researchers adopted DNN-based approaches, including Recurrent Neural Networks (RNN) (Ma et al 2015; Ruchansky 2017; Kochkina et al 2018; Alkhodair et al 2020), Convolutional Neural Networks (CNN) (Bharti and Jindal 2021; Santhoshkumar and Dhinesh Babu 2020), hybrid DNN-based methods (Ajao 2018; Wu and Rao 2020; Wu et al 2020a), and graph-based neural networks (Wu et al 2020b; Huang et al 2020; Bian et al 2020; Liu et al 2022; Wei et al 2022; Ran et al 2022) to attain high performance in detecting rumours.

Recently, a Transformer model called BERT (Bidirectional Encoder Representation from Transformer) has shown to advance the performance of NLP tasks significantly. Researchers are increasingly leveraging BERT to detect rumours on microblogging platforms, including Twitter and Weibo. BERT’s contextual understanding of text enables precise differentiation between real news and rumour news. This can have profound implications for reducing the spread of false information and improving digital literacy. In the next subsection, we explain the Transformer’s technical details and how it is employed for rumour detection.

2.3 Transformer architecture

The Transformer, a revolutionary deep learning model architecture proposed by Vaswani et al (2017), has become a cornerstone in Natural Language Processing (NLP). Unlike traditional models that rely on Recurrent Neural Networks (RNNs), the Transformer architecture utilizes a self-attention mechanism to capture dependencies within input sequences. It features an encoder-decoder architecture originally designed for language translation tasks. In this architecture, the encoder processes the input sequence, while the decoder generates the output sequence. Both components consist of stacked identical layers, incorporating attention and feed-forward layers as shown in Fig. 1

Fig. 1

The general architecture of Transformer (Vaswani et al 2017)

The core innovation of the Transformer lies in its self-attention mechanism, which enables the model to focus on different parts of the input sequence simultaneously. Unlike sequential processing in traditional models, the Transformer processes all positions in the sequence concurrently, facilitating the capture of dependencies and relationships between words without sequential constraints.

The self-attention mechanism operates as follows: first, the input sequence is transformed into word embeddings, capturing the semantic meaning of each word. These embeddings are then transformed into query, key, and value vectors through learned weight matrices. Next, attention weights are computed for each word, indicating its relevance to other words in the sequence. These weights are normalized using a softmax function and used to weight the value vectors, producing the final representation for each word. The attention mechanism is formulated as:

$$\begin{aligned} \text {Attention}(Q, K, V) = \text {softmax}\left( \frac{QK^T}{\sqrt{d_k}}\right) V \end{aligned}$$

(1)

By allowing each position in the sequence to attend to all others, the self-attention mechanism enables the model to capture long-range dependencies and generate contextually informed representations, enhancing its ability to understand word relationships and context.

The Transformer’s architecture allows to achieve superior performance in various NLP tasks. Furthermore, the versatility of Transformers has led to the development of different variants, such as GPT (Generative Pre-trained Transformer) (Yang et al 2019), BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al 2018), RoBERTa (Robustly Optimized BERT Approach) (Liu et al 2019), and T5 (Text-To-Text Transfer Transformer) (Raffel et al 2020). Each variant incorporates unique modifications to the original Transformer architecture, optimizing it for specific tasks or improving its performance in certain areas.

Among these variants, BERT has emerged as one of the most popular and widely used models in the NLP community. Its popularity can be attributed to several factors, including its bidirectional architecture, which allows it to capture contextual information from both left and right contexts simultaneously. Additionally, BERT employs a pre-training strategy using large-scale corpora, followed by fine-tuning on task-specific datasets, enabling it to achieve impressive results across a wide range of NLP tasks with minimal task-specific modifications. Moreover, BERT’s open-source implementation and availability of pre-trained models have contributed to its widespread adoption and popularity among researchers and practitioners in the field. For these reasons, we discuss BERT’s architecture in detail in the following subsection.

2.3.1 BERT architecture

BERT, a Transformer-based and highly influential NLP model, was introduced by Jacob Devlin et al. in 2018 (Devlin et al 2018). This Transformer-based model specifically utilised the encoder component. In essence, BERT is a composition of multiple encoder blocks. It offers two architectural variants: BERT_BASE, featuring 12 encoder layers, 768 hidden layers, and 110 million parameters, and BERT_LARGE, with 24 encoder layers, 1024 hidden layers, and 340 million parameters. The distinction lies in their scale, with BERT_LARGE providing a more extensive architecture for more complex language understanding tasks.

BERT revolutionised NLP by leveraging the bidirectional nature of language and achieving remarkable advancements in various NLP tasks. Furthermore, BERT’s exceptional achievements across numerous benchmark datasets and tasks validate its efficacy in comprehending and generating natural language and serving as a cornerstone model that has paved the way for future advancements in the field.

The key idea behind BERT is to train a language model that learns contextual representations of words based on their surrounding words. Unlike earlier models that relied on unidirectional language modelling, BERT incorporates both the left and right context during training. This bidirectional approach enables the model to have a deeper understanding of the dependencies and relationships between words in a sentence.

Figure 2 provides an illustration exemplifying the bidirectional model through two sentences: “It’s essential to save money for unexpected expenses.” and “The lifeguard moved swiftly to save the drowning swimmer.” Merely considering either the left or right context independently would yield an inaccurate representation of the meaning of the word ‘save’. BERT, on the other hand, comprehensively incorporates both the left and right contexts surrounding the word ‘save’ to generate a more accurate representation. In the first sentence, BERT considers ‘It’s essential’ and ‘money,’ while in the second sentence, it takes into account ‘the lifeguard moved swiftly’ and ‘the drowning swimmer’ to represent the word ‘save’. Consequently, the word ‘save’ in the sentences “It’s essential to save money for unexpected expenses” and “The lifeguard moved swiftly to save the drowning swimmer” are assigned distinct vector representations.

Fig. 2

How BERT encompasses the context from both the left and right of a given word

BERT leverages the attention model from the Transformer to get a deeper understanding of the language context. A BERT model is trained using the masked language model (MLM) and next sentence prediction (NSP). In MLM, a specific percentage of input tokens are randomly masked, prompting the model to predict the original masked tokens. This empowers BERT to learn contextual word representations effectively. On the other hand, NSP trains the model to predict whether two sentences appear consecutively in the original text or not, enabling BERT to capture relationships at the sentence level.

The input text is separated into tokens as in the Transformer model, and each token will be transformed into a vector at the output of BERT to produce contextualised text embeddings. Every layer of the encoder generates a feature for each token along its path, serving as a representation of that particular token. Figure 3 illustrates how text embedding is processed through encoder layers of BERT.

Fig. 3

The text processing through encoder layers of BERT to generate contextualised text embeddings

3 Survey methodology

We follow the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) protocol, which is an established methodological protocol for systematic reviews (Moher et al 2009). The subsequent section elaborates on the criteria used to determine eligibility, the sources of information and the search methods employed, the process of selecting relevant articles, and the outcomes obtained.

The literature search involved the selection of scientific articles written in English and published between 2018 and 2023, as Transformer was introduced in 2017. The articles chosen were based on the inclusion of the following specific keywords: “misinformation” OR “rumour” OR “rumor” OR “credibility” OR “fake news”, combined with the terms “detection” OR “classification” and “Transformer” OR “BERT” OR “pre-trained language model” OR “pre-trained model.” Additionally, the search criteria included the term “social media”, OR “microblogging”, OR “Twitter.” The selection process excluded review articles, surveys, comparisons, and articles that were not accessible in full text. The sources used for extracting these articles were Scopus, Web of Science (WoS), and Google Scholar. We present the survey methodology of this study in Fig. 4.

Fig. 4

The survey methodology of the implementation of Transformer-based classification system for detecting rumours on Twitter

We searched articles from Scopus and Web of Science (WoS) because they primarily index peer-reviewed articles, ensuring a high standard of quality for the included content. These databases cover articles from reputable sources such as IEEE Xplore, Springer, ACM Digital Library, ScienceDirect, and others. Peer review helps ensure that the research is rigorously evaluated by experts in the field, enhancing the reliability and credibility of the research. Furthermore, Scopus and WoS are known as the largest abstract and citation databases, covering a wide range of academic disciplines, including science, technology, medicine, social sciences, and more. It indexes a vast number of journals, conference articles, and other scholarly literature, providing a comprehensive collection of research articles from diverse fields. Additionally, Scopus and WoS offer advanced search functionalities to perform precise searches based on keywords, authors, affiliations, and more. It also provides filtering options to refine search results based on various criteria such as publication date, document type, and citation count, making it easier to identify relevant articles. In addition to Scopus and WoS, we also searched Google Scholar to complement our literature search and access full-text articles, as few articles were not be accessible through Scopus and WoS.

By using the mentioned strategy, we obtained a total of 245 articles. Following that, we conducted a preliminary assessment by skimming through the abstracts of these identified articles to determine their eligibility for our comprehensive evaluation. Out of the initial pool, 185 articles were deemed ineligible for our research objectives and were consequently excluded. The next step involved a detailed examination of the remaining 107 articles, considering factors such as their model, datasets utilised, performance results, contribution, and a brief evaluation of the strengths and weaknesses of the model.

4 Survey analysis

Numerous surveys and review articles have addressed the topic of rumour detection and explored a range of methodologies. In this study, we undertake an in-depth analysis of the latest rumour detection techniques that rely on Transformer-based models. Based on the research methodology mentioned in Sec. 3, we examined and evaluated a total of 117 articles published between 2019 and 2023 which exclusively focused on the implementation of Transformer-based models for detecting rumours on microblogging platforms. Firstly, we analyse the quantitative results of the eligible articles using statistical methods to identify trends, patterns and relationships within the body of literature. Later, we perform a qualitative analysis to provide a comprehensive interpretation of the results within the broader context of existing literature, theoretical frameworks, and real-world implications. Finally, we discuss various datasets available in this area of research.

4.1 Quantitative analysis

Figure 5 presents the trend and distribution of articles published over the years on the topic of Transformer-based rumour classification on microblogging platforms. The findings from the literature survey revealed a significant and accelerating interest among researchers in the implementation of Transformer-based models for detecting rumours on Twitter and other microblogging platforms. In 2019, only one article was found, reflecting a nascent interest in the field. However, in the following years, there was a notable increase in research activity, with 6 articles in 2020, followed by a substantial leap to 33 articles in 2021. This trend continued into 2022 and 2023, with 34 and 33 articles, respectively, indicating a sustained and growing interest in leveraging Transformer-based models for rumour detection.

The rapid proliferation of research in this area underscores the importance and urgency of addressing the challenges posed by misinformation and fake news in online social networks. Researchers are increasingly recognising the potential of Transformers, to enhance the accuracy and effectiveness of rumour detection systems, thereby contributing to the ongoing efforts to promote information credibility and trustworthiness in online discourse. The consistent number of published articles from 2021 to 2023 suggests a period of consolidation and refinement in the field of Transformer-based rumour detection research, characterized by a focus on optimizing existing methodologies and addressing practical challenges rather than introducing entirely new paradigms or approaches.

When examining the type of dataset modalities used in the published research articles, we observed that few articles evaluated their model using various datasets both containing text only, and text and images. However, we found that the majority of studies (85 articles or 79.5% of published articles) focused solely on text data as compared to text and image data (22 articles or 20.5% of published articles). This emphasis is understandable given that microblogging platforms are primarily for sharing short text-based information. Although users have the option to attach images, audio, or video to their posts, the predominant content in most posts remains text only. Furthermore, acquiring images, audio, or video from tweets necessitates additional mining which requires time and computational resources. Interestingly, despite the existence of the option to add videos in tweet posts, as of 2023, no article has reported any rumour classification model for video-based datasets or proposed video-based dataset for rumour classification.

Fig. 5

The distribution trend of articles published over the years from 2019 to 2023 for rumour detection or classification on microblogging platforms

Regarding the implementation strategies, researchers have the option to utilise Transformer models in two ways: as a pre-trained model or by fine-tuning it for a specific task. Based on the literature survey conducted on the implementation of Transformer models for detecting rumours on microblogging platforms, it was found that a significant proportion of the published articles utilize pre-trained models, which is approximately 25.3% of the articles. This suggests that researchers are leveraging the pre-existing knowledge and capabilities encoded in these pre-trained models, such as BERT or GPT, to enhance the efficiency and effectiveness of rumour detection tasks. On the other hand, the majority of articles, approximately 74.7%, opted for fine-tuned models. This indicates a prevalent trend among researchers to fine-tune pre-trained models on specific rumour detection tasks or datasets, allowing for adaptation to the nuances and characteristics of microblogging platforms. Overall, the widespread adoption of both pre-trained and fine-tuned Transformer models underscores their versatility and effectiveness in addressing the challenges of rumour detection on microblogging platforms.

Regarding the languages, it can be observed that the majority of surveyed articles focus on detecting rumours in English, with a significant total of 52 articles dedicated to this language. This reflects the dominance of English as a global language and the prevalence of English-language content on microblogging platforms. Additionally, a notable emphasis on other languages was observed, highlighting the global scope of misinformation detection efforts. Mandarin (Chinese) emerges as the second most studied language, with 20 articles dedicated to detecting rumours in this language, reflecting the immense popularity of microblogging platforms like Weibo in China. Furthermore, there is a diverse array of languages represented in the surveyed articles, including Arabic, Hindi, Persian, Bahasa Indonesia, Russian, Sindhi, Turkish, and Bengali, each with a smaller number of articles dedicated to rumour detection. This diversity underscores the importance of addressing misinformation across different linguistic and cultural contexts, as rumours propagate in all languages and regions.

Regarding the number of classes of rumour addressed in the surveyed articles, we found a predominant emphasis on binary classification, with a substantial 65 articles dedicated to detecting rumours on Twitter and microblogging platforms. This focus suggests a primary interest in discerning between true and false information, aligning with the fundamental objective of rumour detection systems. However, our survey also reveals a noteworthy number of articles exploring multi-class rumour classification approaches. We identified 15, 6, 1, and 1 articles delving into rumour detection across three, four, five, and six classes, respectively. This diversity in classification schemes highlights the complicated nature of rumour detection tasks, as researchers endeavour to categorize rumours into distinct categories based on factors such as severity, stance, and context. Multi-class classification strategies facilitate granular analysis and decision-making, enabling more refined responses to various types of rumours and misinformation.

4.2 Qualitative analysis

In this subsection, we delve into the qualitative findings of the surveyed articles that employ Transformer-based models for rumour detection. We expanded our analysis by providing detailed insights into the methods, and their strengths and weaknesses. Table 2 presents the summary of our in-depth analysis of the literature.

Moreover, based on the survey findings and observations, we presented a new taxonomy of the implementation of such models for combating misinformation spread on microblogging platforms. The taxonomy is developed to cover a wide range of methodologies and strategies employed in the implementation of Transformer-based models for misinformation detection, providing a comprehensive overview of the field from both theoretical and practical perspectives, as shown in Fig. 6.

Fig. 6

Taxonomy on the implementation of Transformer-based model for combating misinformation on microblogging platforms

The taxonomy aims to facilitate further research and development in this important area of study by offering a structured framework for organizing and comparing different approaches. It encompasses five main categories:

• 1. Training Approaches. This categorisation is essential for facilitating a deeper understanding of the underlying methodologies and their implications for performance. Based on our findings, two main categories emerge from the surveyed literature, including utilising pre-trained models without modification and fine-tuning the Transformer model on a specific dataset.

• 2. Supervised Learning Approaches. This category encompasses the foundational approach of fine-tuning Transformer models on labelled datasets for misinformation detection. It includes specific subcategories focusing on event-based detection, temporal analysis, and multi-modal detection, highlighting the different dimensions of supervised learning applied to misinformation detection tasks.

• 3. Advanced Learning Approaches. This category delves into more sophisticated techniques beyond simple fine-tuning, such as multi-task learning, adversarial learning, reinforcement learning, and exploration of different model variants. These approaches represent advancements in the field aimed at improving model robustness, adaptability, and performance.

• 4. Data Augmentation and Feature Engineering. This category recognizes the importance of data augmentation and feature engineering in enhancing the quality and effectiveness of misinformation detection models. It includes techniques for increasing data diversity and incorporating additional features beyond those provided by the Transformer model itself.

• 5. Ensemble Approaches. This category covers ensemble methods, which combine predictions from multiple models or integrate features from various sources to achieve superior performance in misinformation detection.

4.2.1 Training approaches on transformer-based models

Based on the training strategy, we observed that the implementation of Transformer models proposed for combating misinformation in the surveyed articles can be clustered into one of the two different approaches: pre-trained or fine-tuned Transformer models. This categorization aims to help identify the strengths and weaknesses of each training strategy, enabling researchers to make informed decisions about which approach is most suitable for their particular application.

4.2.2 Pre-trained transformer model approach

In this approach, the proposed solutions utilise a pre-trained Transformer model to extract features from the text without further fine-tuning using the experimental dataset. The input text data is tokenized and passed through the pre-trained model, and the output representations from one or more layers of the model are extracted as features. This approach is beneficial when fine-tuning the entire Transformer model may not be necessary or feasible for a specific task, such as when working with limited labelled data or when the task does not require extensive adaptation of the Transformer’s parameters.

Generally, researchers implemented a pre-trained model approach by adding a fully connected layer directly to the last encoder layer as a classifier (Das et al 2021; Tondulkar et al 2022; Rifai et al 2023; Kalra et al 2023a; Zhu et al 2023), or by saving the vectors from Transformer models and applying a classifier model to identify rumours on the Twitter platform Ali and Malik (2023).

The incorporation of the pre-trained Transformer model without fine-tuning for a specific task dataset has both advantages and disadvantages. Despite not undergoing fine-tuning, a pre-trained Transformer model can still capture the nuances and context of messages to some extent, facilitating the identification of potentially misleading information. Furthermore, without fine-tuning, implementing a pre-trained model can be more efficient and straightforward, eliminating the need for task-specific labelled datasets or additional training. This streamlined process can result in a quicker deployment timeline, making it particularly valuable for rapid-response scenarios or situations with limited resources.

However, there are limitations concerning task specificity, adaptation to the microblogging context, and accuracy. While pre-trained models exhibit general language understanding, they may not be attuned to the specific nuances, vocabulary, and context of microblogging platforms. This lack of domain adaptation may impede their effectiveness in rumour detection. Additionally, pre-trained model approaches may not be optimised for combating misinformation, lacking the task-specific knowledge and training data necessary to distinguish between true and false information on the microblogging platforms.

4.2.3 Fine-tuned transformer model approach

In this approach, researchers fine-tune the Tranformer model on the task-specific dataset. During training, the weights of the pre-trained Transformer model will be updated to minimise the loss between predicted and actual labels by monitoring training progress using tracking metrics such as loss and accuracy on a validation dataset. Iteration through multiple epochs during training continues until convergence, and adjusting hyperparameters as needed.

Similarly, with the pre-trained-based approach, few studies build a fine-tuned model architecture by adding a classification layer on top of the fine-tuned Transformer model, such as Slimi et al (2021); Leonardi et al (2021); Heidari et al (2021); Malla and Alphonse (2021); Albalawi et al (2023), and more, as summarised in Table 2. This additional layer will adapt a Transformer model to the specific task of rumour detection. Alternatively, few researchers used a fine-tuned Transformer model as a text embedder or vectorizer to generate the improved text representations as the input to train a classifier model to identify rumours on microblogging platforms, such as Zhou et al (2023); Jing et al (2023); Salini and Harikiran (2023); Al Obaid et al (2023); Wani et al (2023); Anggrainingsih et al (2023). Table 2 provides more examples of these approaches.

Fine-tuning a pre-trained Transformer model with a task-specific dataset typically yields higher accuracy and better adaptation to the microblogging context. This is because fine-tuning enables the model to learn from labelled data specific to rumour detection on microblogging platform, enhancing its capacity to discern false information and adapt to the unique characteristics and context of the platform, including slang, abbreviations, and trending topics. Consequently, fine-tuned model approaches are generally found to outperform the pre-trained counterparts in terms of rumour detection and classification accuracy.

Nevertheless, the process of fine-tuning comes with its own challenges. It necessitates high computational resources, which can be expensive. Additionally, fine-tuning involves adjusting hyperparameters, model architecture, and training procedures, introducing complexity that require specialized expertise. Furthermore, the effectiveness of fine-tuning is contingent on the quality and size of the task-specific dataset. Small or biased datasets potentially limit model’s performance (Bao et al 2023; Garcia-Silva et al 2021).

4.2.4 Learning approaches on transformer-based models

Almost all the surveyed articles focused on supervised learning approaches for various objectives. Below are the main objectives:

• 1. Binary or multiclass classification In the context of combating rumours on microblogging platforms, binary and multiclass classification approaches have distinct characteristics, objectives, and applications. In binary classification, models are trained on labelled datasets where each post is assigned a binary label indicating its rumour status. The model learns to distinguish between the features associated with its label. We observed in our survey of the literature that the majority of surveyed articles are analysing binary classification that aims to classify each microblogging platform post into one of two categories: rumour/non-rumour, fake/real or false/true. The primary goal is to differentiate between posts that contain misinformation or not Singhal et al (2019); Zhang et al (2020); Malhotra and Vishwakarma (2020); Heidari et al (2021); Qian et al (2021); Jing et al (2021); Bing et al (2022); Sharma et al (2022a); Zhu et al (2023); Alawadh et al (2023). Similarly, in multiclass classification, models are trained on labelled datasets with multiple rumour categories, with each post assigned to one of these categories. The model learns to differentiate between different types of rumours and assign the most appropriate label to each post. The objective of multiclass classification is to categorize rumours into distinct classes based on various attributes such as severity or credibility. In our literature survey, we observed that different studies have handled the multiclass classification problem with different number of classes, such as three classes by (Fatima et al 2022; Nassif et al 2022; Bahurmuz et al 2022; Panagiotou et al 2021), four classes by (Luo et al 2021; Wani et al 2020; Xu et al 2022; Tondulkar et al 2022; Anggrainingsih et al 2023), five classes by (Roshan et al 2023), and six classes by (Kalra et al 2023a).

• 2. Event-based rumour detection In these studies, researchers used supervised learning on Transformer-based models to detect rumours for specific events. To achieve the objectives, event-specific datasets are used which are associated with particular events or topics, such as:

  • Breaking news (Slimi et al 2021; Ying et al 2021; Yuan et al 2021; Xu et al 2022; Wei et al 2023; Wu et al 2023; Al Obaid et al 2023; Ali and Malik 2023)

  • COVID-19 issues (Heidari et al 2021; Ayoub et al 2021; Leonardi et al 2021; Malla and Alphonse 2021; Tafannum et al 2021; Khandelwal 2021; Wani et al 2020; Yang et al 2021; Kar et al 2021; Das et al 2021; Fatima et al 2022; Shrivastava and Sharma 2022; Bozuyla and ÖZÇİFT 2022; Obeidat et al 2022; Wahle et al 2022; Abd Elaziz et al 2023; Taha et al 2023; Roshan et al 2023; Rifai et al 2023; Wu et al 2023; Kalra et al 2023a; Wani et al 2023).

  • Political events (Zervopoulos et al 2022; Sharma et al 2022a)

  • Vaccine and health related issues (Hayawi et al 2022; Upadhyay et al 2023; Salini and Harikiran 2023)

• 3. Temporal analysis of rumour spread Temporal analysis typically involves collecting and analyzing timestamped data from microblogging platforms to track the chronological sequence of misinformation posts and their interactions (e.g., retweets, replies, and likes) over time. By analysing temporal patterns, researchers can better understand the mechanisms driving misinformation propagation and develop more effective strategies for detection and mitigation. In our survey, we observed that a few studies used Transformer-based models to analyse sequences of posts and identify key moments of rumour spread and temporal patterns of rumour propagation by combining Transformer model with graph-based models Malhotra and Vishwakarma (2020); Song et al (2021); Luo et al (2021); Bing et al (2022); Saikia et al (2022); Wei et al (2023).

• 4. Multi-modal rumor detection Few studies employed multi-modal rumour detection to enhance the text representation provided by the Transformer model. These studies proposed to integrate information from other modalities such as images or videos. In this approach, a Transformer model is utilized to encode microblogging posts into text embeddings, while an image encoder is employed to encode images into image embeddings. Subsequently, both text and image embeddings were fused as new features to represent the posts (Singhal et al 2019; Zhang et al 2020; Qian et al 2021; Ying et al 2021; Yang et al 2021; Tuan and Minh 2021; Jing et al 2021; Cheema et al 2022; Singh and Sharma 2022; Li et al 2022b; Yadav and Kumar 2023; Sharma et al 2023; Wu et al 2023; Ghorbanpour et al 2023; Jing et al 2023; Albalawi et al 2023; Zhou et al 2023; Singh et al 2023).

4.2.5 Miscellaneous learning approaches of transformer-based models

In our survey, we discovered that certain studies employed different and more sophisticated techniques or innovative training paradigms to improve the performance or capabilities of the transformer-based model. We categorize these advanced learning methods into multi-task learning, adversarial learning, reinforcement learning, and few-shot learning, which we are explained as follows.

• 1. Multi-task learning on transformer model Few studies employed multitask learning by training a single Transformer model on multiple related tasks simultaneously, such as rumour detection, sentiment analysis, or event detection (Do et al 2021; Khandelwal 2021; Jing et al 2021; Bahurmuz et al 2022; Abd Elaziz et al 2023; Zhu et al 2023; Alawadh et al 2023).

• 2. Adversarial learning on transformer model In this strategy, researchers train Transformer to generate robust representations that are invariant to adversarial perturbations, improving the model’s robustness to noise or adversarial attacks (Song et al 2021).

• 3. Reinforcement learning on transformer model In this approach, Transformer is trained to maximize a reward signal based on its actions and feedback received from the environment, enabling adaptive and dynamic behaviour in rumour detection (Yuan et al 2021).

• 4. Few-shot learning on transformer model This type of learning offers advantages by enabling models to generalize effectively from limited labelled examples to quickly adapt to new topics and rumours with a small number of examples, and reduce the annotation effort needed for large datasets. This is particularly beneficial in the vast microblogging data landscape where manual annotation of recently released data is impractical. Tian et al (2021) presented an innovative zero-shot cross-lingual transfer learning framework for detecting rumours by utilising multilingual BERT and self-training to adjust the model from the source language to the target language.

4.2.6 Supplementing transformer-based models

In our survey, we also observed that several studies utilised techniques for enhancing data diversity and incorporating additional features beyond those provided by the Transformer model. The major techniques are discussed below which are data augmentation, feature engineering, and training models in different languages to improve the effectiveness of misinformation detection models.

4.2.7 Data augmentation

Few studies employed different augmentation techniques to increase data diversity. On text data, they generated synthetic text samples using various techniques, including:

• Synonym replacement: replacing words with their synonyms (Singhal et al 2019).

• Back translation: translating the original texts into another language and then translating them back into the original language (Ayoub et al 2021).

• Paraphrasing: original text to generate new texts with different expressions but similar meanings (Wu et al 2023).

• Adversarial generator: employing an adversarial response generator to generate realistic and misleading responses to a source post (Song et al 2021).

• Integration: two or more techniques are integrated to enhance the performance of the model (Tian et al 2020; Do et al 2021; Khandelwal 2021; Zervopoulos et al 2022; Bozuyla and ÖZÇİFT 2022; Saikia et al 2022; Panagiotou et al 2021; Alghamdi et al 2023a; Al Obaid et al 2023)

In the image data, several studies have employed synthetic image generation techniques by applying random transformations such as rotation, scaling, cropping, and flipping (Singhal et al 2019; Singh and Sharma 2022; Wu et al 2023). Notably, a subset of these studies focused on multi-modal datasets, implementing augmentation techniques on both text and image data concurrently (Singhal et al 2019; Wu et al 2023).

Furthermore, in the domain of graph-based data, augmentation strategies have been applied by randomly adding or deleting edges from dynamic graphs to enhance the diversity of graph structures (Wei et al 2023). Interestingly, certain researchers have adopted a comprehensive approach by augmenting both text and graph data in their investigations Do et al (2021); Kalra et al (2023b).

4.2.8 Feature engineering

Few researchers incorporated additional features beyond those provided by the Transformer model by incorporating features from both content and well as context of the post. The content-based features include discrete emotions, linguistic features, and metadata features, which are concatenated with the context features and fed into a classifier Ali and Malik (2023).

4.2.9 Multiple languages training

To enhance the ability of the model to detect misinformation across diverse linguistic contexts on microblogging platforms, few studies proposed to train Transformer models on data in multiple languages other than English, such as Arabic (Bahurmuz et al 2022; Obeidat et al 2022; Taha et al 2023; Albalawi et al 2023), Persian (Jahanbakhsh-Nagadeh et al 2021), Turkish (Bozuyla and ÖZÇİFT 2022), Hindi and Bengal (Kar et al 2021), Bahasa Indonesia (Isa et al 2022; Rifai et al 2023), Sindhi (Roshan et al 2023), and Chinese (Qian et al 2021; Jing et al 2021; Bing et al 2022; Tondulkar et al 2022; Wei et al 2023; Fu et al 2023)

4.2.10 Variants of transformer model

While BERT-based models are widely favored for combating rumors on microblogging platforms, it’s important to note that other variant Transformer models offer alternative approaches for rumor detection. Out of the 83 surveyed articles, 58 reported the implementation of BERT models, indicating their popularity in this domain. However, this should not overshadow the potential of other Transformer variants.

These variants incorporate modifications or enhancements to the original Transformer architecture to address specific challenges or to improve performance on various NLP tasks. Each architectural Transformer variant in the literature offers unique improvements for specific tasks, providing researchers and practitioners with a range of options tailored to address their specific needs. By exploring different Transformer variants, researchers can identify the most suitable model architectures and techniques for addressing the unique challenges of rumor detection on microblogging platforms.

The summarized list of variants of Transformers used in combating rumours on microblogging platforms other than BERT, which has been mentioned in Sect. 2.3, are as follows:

• Robustly Optimized BERT Approach (RoBERTa) was introduced by Liu et al. (2019) to enhance BERT’s training model through various modifications, including expanding the training dataset, eliminating the next sentence prediction task, extending the training sequences, and adapting the masking pattern dynamically based on the training data. RoBERTa was trained by following the architecture of BERT_LARGE. However, in contrast to BERT, which was initially trained on a 16GB dataset, RoBERTa utilized a much larger training dataset of 160GB of text. Furthermore, RoBERTa underwent an increased number of training iterations, from 300,000 to 500,000. The advancement of RoBERTa attracts researchers to obtain a better representation of tweets for classifying rumours on Twitter (Hande et al 2021; Samadi et al 2021; Joy et al 2022; Ghayoomi and Mousavian 2022; Singh and Sharma 2022; Guo et al 2020; Malhotra and Vishwakarma 2020; Pelrine et al 2021; Kasnesis et al 2021; Shrivastava and Sharma 2022; Bozuyla and ÖZÇİFT 2022; Zervopoulos et al 2022)

• Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA) differs from BERT primarily in its training objective, which involves a discriminator tasked with identifying replaced tokens (Clark et al 2020). This approach makes ELECTRA computationally efficient and capable of achieving competitive results with smaller training datasets compared to BERT and motivated some researchers to evaluate and compare text embedding from ELECTRA to other BERT models (Singh et al 2023; Kasnesis et al 2021).

• COVID-Twitter-BERT (CT-BERT) is designed to analyze COVID-19-related content on Twitter, aiding in tasks like classification, question-answering, and chatbots. It was pre-trained on a large corpus of COVID-19-related Twitter messages (Müller et al 2020). Studies that utilised CT-BERT are Malla and Alphonse (2021); Wahle et al (2022).

• A lite BERT (AlBERT) is a variant of the BERT designed to address some of the limitations of BERT while achieving better efficiency in terms of model size, training time, and computational resources through parameter reduction, the introduction of the SOP task, and cross-layer parameter sharing, among other innovations (Lan et al 2019). In literature, researchers applied AlBERT to compare the quality embedding with other BERT models (Joy et al 2022) and to develop FAKEfinder, a mobile application to detect rumours (Tian et al 2020).

• DistilBERT, a variant of BERT developed to provide a more efficient and lightweight alternative to the original BERT model while retaining much of its performance through a reduced model size and knowledge distillation from a teacher BERT model (Sanh et al 2019). Few researchers utilised DistillBERT to extract contextualised text features from tweets (Karande et al 2021; Ayoub et al 2021) and to make a comparison of the embedding quality with other BERT models (Bozuyla and ÖZÇİFT 2022).

• Multilingual BERT (mBERT), is a BERT-based model that is trained to handle multiple languages without requiring language-specific adaptations (Devlin et al 2018). It serves as an efficient and versatile solution for multilingual natural language processing tasks and offers a balance between performance and coverage across various languages. In the context of rumour detection on Twitter, few studies proposed variant of mBERT to deal with text in different languages, including IndoBERT (Isa et al 2022) for Bahasa Indonesia, JointBERT (Shishah 2022) and AraBERT (Obeidat et al 2022) for Arabic, ParsBERT (Jahanbakhsh-Nagadeh et al 2021) for Persian and BERTurk (Bozuyla and ÖZÇİFT 2022) for Turkish.

• BERTtweet is a BERT-based model that is fine-tuned and pre-trained specifically for handling Twitter data. It is optimized to understand and process Twitter-specific language and features, making it a valuable tool for NLP tasks involving tweets and Twitter content (Nguyen et al 2020). De et al. utilised and incorporated embeddings from BERT, BERTweet and CT-BERT to obtain rich semantic text representation from tweets (De and Desarkar 2022).

• Decoding-enhanced BERT (DeBERTa) is an enhanced version of BERT model, featuring disentangled attention mechanisms, improved pre-training and fine-tuning techniques, and focuses on capturing global dependencies in text (He et al 2020). Li et al. utilised DeBERTa to extract features from tweets for detecting rumours on Twitter (Li et al 2022a; Zhou et al 2023; Jing et al 2023).

• Extra-Long Transformer Network (XLNet) is a variant of Transformer-based models designed to address some of the limitations of previous architectures like BERT by introducing several key innovations that contribute to its improved performance on various natural language processing tasks (Yang et al 2019). XLNet has a larger model capacity compared to BERT, with more parameters and layers. The increased model capacity allows XLNet to capture more complex patterns and relationships in the input data, leading to improved performance on a wide range of natural language processing tasks such as Khandelwal (2021); Xu et al (2022); Taha et al (2023); Rifai et al (2023); Sharma et al (2023); Al Obaid et al (2023)

• Text-To-Text Transfer Transformer (T5) is a versatile and powerful Transformer-based model (Raffel et al 2020). It casts all NLP tasks into a text-to-text format, where both inputs and outputs are represented as text strings. T5 follows the encoder-decoder architecture commonly used in Transformer models, where the encoder processes the input text and the decoder generates the output text. While there may not be specific articles focusing solely on T5 for combating misinformation on Twitter or microblogging platforms, T5’s versatility and effectiveness in various NLP tasks make it a potential candidate for rumour detection tasks. For instance a study by Salini and Harikiran (2023) proposed a novel and effective model that combines two Transformer-based models, T5 and RoBERTa, with a multiplicative vector fusion technique to provided an efficient solution to NLP tasks.

4.2.11 Ensemble approaches

The approach of combining predictions from multiple models or integrating features from various sources to enhance overall performance in misinformation detection are normally known as Ensemble approaches. Their objective is to leverage the diversity of individual models or features and combine their predictions or representations, aiming to mitigate the weaknesses of individual models and enhance the overall effectiveness of misinformation detection systems.

For instance, Malla and Alphonse (2021) proposed an ensemble model that combines the outputs of three pre-trained models: BERT, RoBERTa, and CT-BERT. This ensemble model utilizes a weighted average of the probabilities from each model to make the final prediction. Das et al (2021) introduced a heuristic-driven uncertainty-based ensemble framework for misinformation detection. The framework incorporates three components: a multilingual model (teacher), a monolingual or multilingual model (student), and a heuristic algorithm to detect fake news in tweets and news articles. Albalawi et al (2023) presented a multimodal model that combines textual and visual features of tweets using two fusion techniques: early fusion and late fusion. Early fusion concatenates the features before feeding them to a classifier, while late fusion employs separate classifiers for each modality and aggregates their outputs.

Similarly, Al Obaid et al (2023) proposed a novel semi-supervised method for fake news recognition on Twitter, leveraging effective data augmentations and an ensemble of diverse deep learners for detecting fake news. Singh et al (2023) introduced the Stacked Ensemble-Based Multimodal Inference for Faster Fake News Detection (SEMI-FND) framework on microblogging platforms, which comprises four components: data preprocessing, T5 model for sentence generation, and RoBERTa model for rumour detection, and multiplicative vector fusion for classification.

Table 2 Summary of the surveyed articles with technical details, time of publishing and the strengths and weaknesses of the proposed model for rumour detection on micorblogging platforms

4.3 Datasets

Many datasets are proposed over time for evaluating the performance of models in rumour detection on microblogging platforms. However, we selected few major datasets that present significant challenges and have recently gained popularity. Few of these datasets are sourced from earlier works (Bondielli and Marcelloni 2019; Sharma et al 2019; Islam et al 2020a; Varshney and Vishwakarma 2021; Varlamis et al 2022). Table 3 presents the summary of the shot-text datasets used for rumour detection and classification. Their details are presented below:

• Twitter mediaEval (Boididou et al 2015): It comprises English tweets and associated images covering diverse events. This dataset includes around 7000 fake and 5000 genuine news tweets in the training set, and around 2500 fake and 1200 real tweets in the test set. The dataset is publicly available at https://github.com/MKLab-ITI/image-verification-corpus/tree/master/mediaeval2015.

• Pheme (Zubiaga et al 2016): This dataset encompasses 6425 tweets related to nine different news events. Journalists verified whether these tweets were rumours (2402 tweets) or non-rumours (4,023 tweets). The rumour tweets are further classified as true rumours (1067 tweets), false rumours (638 tweets) or unverified rumours (697 tweets). The dataset is publicly available at https://figshare.com/articles/PHEME_dataset_of_rumours_and_non-rumours/4010619.

• Weibo dataset (Jin et al 2017): This dataset encompasses real news sourced from reputable Chinese news outlets like Xinhua News Agency, alongside fake news in Chinese collected from Weibo microblogging platform between May 2012 and June 2016. The gathered news items were vetted by Weibo’s official rumour debunking system. The dataset is divided into training and testing sets, which consist of 3749 rumour and 3783 non-rumour tweets for the training set and 1000 rumour and 996 non-rumour tweets for the testing set. The dataset is publicly available at https://github.com/wangzhuang1911/Weibo-dataset.

• Twitter 15–16 (Ma et al 2017): The publicly available datasets, Twitter 15 and 16, contain 1490 and 818 tweets respectively. Journalists categorized each tweet as a rumour or non-rumour. Subsequently, each rumour tweet was further classified as true rumour, false rumour, or unverified rumour. The dataset is publicly available at https://www.dropbox.com/s/7ewzdrbelpmrnxu/rumdetect2017.zip?dl=0.

• KNTUPT (Mahmoodabad et al 2018): This is a highly unbalanced dataset that includes 3,598,049 tweets, with 4345 tweets labelled as rumours. It involves 60 different rumours about the Kermanshah earthquake, with 58 being incorrect rumours and two being true. Rumor tweets make up around 0.1% of all the tweets. The dataset is publicly available at https://trlab.ir/res.php?resourceid=3.

• Fake news detection task (Pogorelov et al 2020): This dataset consists of 6458 tweets and 2327 retweet graphs in the training set, as well as 3,230 tweets and 1165 retweet graphs in the test set. It features three categories: The 5GCorona Conspiracy class, the other Conspiracy class, and the non-conspiracy class, all presented in English. Unfortunately, there is no information about its publicly availability.

• Indic-covidemic: This dataset includes Bengali language tweets regarding COVID-19, totalling 36,117 unique tweet IDs spanning from December 2019 to May 2020. Keywords like ‘corona ’, ‘covid ’, ‘sarscov2 rq, ‘covid19 ’, and ‘coronavirus ’were used for tweet retrieval. The dataset is publicly available at https://ieeedataport.org/openaccess/covid-19-tweetsdataset-bengali-language.

• COVID-19 fakes (Elhadad et al 2021): This is a bilingual (Arabic/English) COVID-19 Twitter dataset collected between February 4, 2020, and March 10, 2020. It is annotated as Real and Misleading using 13 machine learning algorithms and 7 feature extraction techniques, drawing from official sources like WHO, UNICEF, and fact-checking websites. The dataset is publicly available at https://github.com/mohaddad/COVID-FAKES.

• Weibo21 (Nan et al 2021): It consists of 4,488 fake news samples and 4,640 real news samples from 9 different domains. Each sample is labeled with a domain category. The dataset facilitates multi-domain analysis. The dataset is publicly available at https://paperswithcode.com/dataset/weibo21.

• Arabic COVID-19 (Obeidat et al 2022): This is the first Arabic language misinformation dataset about COVID, encompassing around 6,700 tweets with multi-class and multi-label misinformation annotations. The dataset is publicly available at DOI:10.7717/peerj-cs.1151/supp-1.

• MM-Claims (Cheema et al 2022): This dataset involves tweets and corresponding images across three topics: COVID-19, climate change, and technology. It incorporates approximately 86,000 tweets, with around 3,000 manually labelled for training and evaluating multimodal models by multiple annotators, and roughly 82,000 image-text tweets introduced remain unlabelled. For labelled tweets the dataset consists of Factual claim (1,840 tweets) and Not a claim (1,210 tweets). The dataset is publicly available at https://data.uni-hannover.de/dataset/mm_claims.

• Twitter-Hongkong dataset (Zervopoulos et al 2022): This dataset comprises discussions on the 2019 Hong Kong protests on Twitter, meticulously curated by journalists to categorize tweets as either fake or real news. The compiled dataset includes 3,908 tweets disseminating fake news and 5,388 tweets disseminating authentic news. The dataset is publicly available at https://tinyurl.com/y3ffrblt.

• FNID (Fake News Inference Dataset) (Sadeghi et al 2022): This dataset is designed to be compatible to use both FakeNewsNet-PolitiFact (two classes) and LIAR dataset (six classes) datasets. The two-classes dataset consist of 8,557 fake news and 8,767 real news on microblogging platforms. While the six-classes dataset consist of six different news labels, including ‘pants-fire ’(2,012), ‘false’(3,809), ‘barely-true’(2,897), ‘half-true’(3,339), ‘mostly-true ’(3,096), ‘true’(2,430). The dataset is publicly available at https://ieee-dataport.org/open-access/fnid-fake-news-inference-dataset.

• Arabic Twitter dataset (AWAJAN 2023): With 206,080 tweets, this Arabic dataset contains 159,284 legitimate tweets and 46,796 fake tweets collected from the Anti-Rumor Authority established in 2012 by the Jordanian government. There is no information on whether the dataset is publicly available or not.

Table 3 Rumour datasets on microblogging platforms

5 Challenges and future directions

Implementing Transformers on Twitter poses numerous challenges due to the platform’s unique characteristics, particularly concerning microblogging platforms’ policy constraints and the diverse language used in its messages. Based on our thorough analysis of the surveyed articles, as summarised in Table 2, these challenges can be categorized into four clusters. First, there are challenges associated with the microblogging platform’s content and text characteristics, given the informal and noisy nature of communication on the platform, including abbreviations, slang, and non-standard grammar. Second, difficulties arise in collecting data from microblogging platforms, influenced by restrictions imposed by the platform’s terms of service and privacy policies. Third, challenges emerge in selecting appropriate techniques and evaluation metrics for effective rumour detection. Lastly, there are computational requirements, as Transformer models are known for their large size and can be demanding on computational resources (Fig. 7).

Fig. 7

The challenges of implementing Transformer-based models for rumour detection on microblogging platforms

5.1 Challenges of content and text characteristics of posts

Microblogging platform distinctive style of communication is characterised by its informal and noisy language, featuring elements like abbreviations, slang, misspellings, and non-standard grammar. General Transformer models developed for NLP tasks encounter difficulties in handling this level of linguistic diversity, necessitating additional preprocessing, post-processing, or tailoring of the model. Furthermore, microblogging platforms impose constraints on the length of messages in a single post, making it challenging for Transformer models to grasp the complete conversation and context surrounding a tweet, especially in cases where rumours span multiple tweets. Moreover, Twitter extensively employs features like hashtags (#) and mentions (@), that require Transformer models to be adapted to recognize and interpret these elements, as they hold significance in rumour detection.

Microblogging platform’s global reach results in a multitude of languages, which presents another challenge to detect rumours across linguistic boundaries. Additionally, vocabulary used in posts has a constant influx of new words, hashtags, and emerging trends. The proposed models must exhibit adaptability to accommodate this evolving lexicon and remain updated for accurate rumour detection. Detecting rumours on microblogging platform relies on understanding the context of post, prompting the use of methods to capture temporal and situational details. To address these challenges, researchers have refined predominantly BERT models using Twitter-specific data, adapting them to the platform’s unique features, such as BERTweet proposed by Nguyen et al (2020).

However, the pursuit of optimal performance continues. Despite commendable efforts, there’s an ongoing need to improve detection accuracy. Acknowledging the dynamic nature of microblogging platform and keeping up with changing linguistic patterns and user behaviours, we suggest exploring new methodologies and learning techniques, as introduced by Leonardi et al (2021); Luo et al (2021); Ali and Malik (2023) in their studies. Furthermore, in future research, it is pertinent to consider the potential efficacy of combining pre-processing methods, and model ensemble approaches with novel fine-tuning or learning techniques to enhance rumour detection on microblogging platform’s posts.

5.2 Challenges of data collection

Collecting data from microblogging platform for rumour detection poses challenges related to access restrictions, privacy policies, and the sheer volume of posts. Researchers need to navigate rate limits, address ethical considerations, and effectively manage and label large datasets. Furthermore, additional complexities need to be addressed concerning the handling of messages in various languages, the mitigation of biases, and the assurance of balanced datasets.

While existing datasets have contributed valuable insights, the existing rumour datasets lack diversity as they are mainly focused on a few emerging topics, such as breaking news events, politics, and sports. Therefore, the construction of new datasets may be necessary in cases where a study aims to develop a model for a new specific scenario or topic, such as political information during election seasons, a new outbreak event or disease, or a recent disaster or war. Additionally, the existing datasets lack the depth and specificity required for particular scenarios as conversations on microblogging platforms are dynamic and subject to rapid changes in narratives, terminology, and sentiments. For example, the use of emojis, hashtags, and retweets may vary across different time periods and affect the rumour verification task.

We propose that future research on rumour detection should not only focus on developing advanced methods to address the mentioned challenges but also emphasize the creation of datasets spanning diverse domains and events to counter the observed lack of diversity in existing datasets. To deal with the scarcity of data, we suggest to incorporate augmentation techniques, as proposed in Tian et al (2020); Do et al (2021); Khandelwal (2021); Zervopoulos et al (2022); Bozuyla and ÖZÇİFT (2022); Saikia et al (2022); Panagiotou et al (2021); Alghamdi et al (2023a); Al Obaid et al (2023). These studies integrated multiple augmentation techniques to increase data diversity and enhance the performance of the model.

Moreover, it would be beneficial to investigate the performance of few-shot learning models for rumour detection, as highlighted in (Wang et al 2020; Kochkina et al 2023). Few-shot learning offers advantages by enabling models to generalize effectively from limited labelled examples, quickly adapt to new topics and rumours with a small number of examples, and reduce the annotation effort needed for large datasets, which is particularly beneficial in the case of large microblogging platform’s data requiring manual annotation. In our literature survey, we observed that Tian et al (2021) is the only study that introduced a novel zero-shot cross-lingual transfer learning framework for rumour detection. This framework does not require any annotated data in the target language and leverages multilingual BERT and self-training to adapt the model from the source language to the target language. The study achieved state-of-the-art results on both English and Chinese rumour datasets, surpassing existing monolingual and cross-lingual models. This encourages further further research in the same direction.

5.3 Challenges of training and evaluation metrics

Implementing Transformer-based models for detecting rumours on microblogging platforms involves multiple challenges related to techniques and evaluation metrics. The first challenge is to train the models effectively, whether to leverage pre-trained Transformer models, engage in fine-tuning, or create custom architectures. Each option presents critical decisions laden with unique challenges and trade-offs. Another challenge emerges in the selection of the right Transformer model variant or architecture for effective rumour detection as discussed in Section 4.3. Understanding the nuanced differences between Transformer models and their suitability for specific tasks becomes pivotal in this context. Furthermore, the process of hyperparameter tuning, which involves optimizing parameters like learning rates, batch sizes, and dropout rates, is indispensable for optimizing model performance. This process can be demanding in terms of time and computation.

Additionally, choosing the right evaluation metrics for detecting rumours is vital. Commonly employed metrics include precision, recall, F1-score, accuracy, and AUC-ROC. However, the choice of metrics depends on the specific goals of the task. The evaluation metrics for NLP tasks are different, such as METEOR and BLUE scores. Conducting comparative evaluations, which assess various Transformer-based models, architectures, or approaches, presents challenges, requiring careful experimental design and consistent evaluation practices.

From our literature survey, we believe that there is still room for improvement in achieving the optimal accuracy of the rumour detection model. To advance in this direction, future research should prioritize refining and enhancing these aspects. Given the rapid spread of rumours across diverse social media platforms, it is crucial for future research to develop novel approaches for cross-platform evaluation. Additionally, to uphold the integrity and reliability of information shared on microblogging platforms, there is a dire need to develop a real-time rumour detection system. Such a system would enable platforms to act swiftly in addressing misinformation, thereby safeguarding individuals, organizations, and the broader community from the negative impacts of false rumours.

5.4 Challenges of computational requirements

Applying Transformer-based models is computationally challenging for most of the models. This is because Transformer models have a substantial number of parameters, especially the bigger ones like BERT_LARGE and RoBERTa. Running these models requires substantial computational resources including expensive graphics processors (GPUs) or tensor processing units (TPUs), large memory, etc. Thus, training a large number of parameters in Transformer-based models requires a long time. Therefore, it is important to make a balance between required resources and performance.

Few studies have attempted to address the computational issues by introducing lighter variant models of Transformer, such as (Lan et al 2019; Sanh et al 2019; Clark et al 2020). We also found the findings in Anggrainingsih et al (2023) to tackle the computational challenge by reducing the number of parameters in BERT while maintaining its performance. The study introduced CE-BERT, a concise and efficient BERT-based model, demonstrating its ability to effectively capture the semantic features of tweets and achieve state-of-the-art results in a computationally efficient manner.

While existing studies have made efforts to address computational issues, there is still a need for enhancements to achieve computationally efficient solutions. This endeavour aligns with the goal of making research greener, more inclusive, and cognitively plausible, as emphasized by AI researchers (Schwartz et al 2020; Strubell et al 2019; Dhar 2020). We suggest that future research should focus on three key areas: exploring novel approaches to manage computational costs effectively, enhancing model performance under resource constraints, and experimenting with different techniques for model adjustment, such as fine-tuning, to facilitate faster learning without compromising accuracy.

6 Conclusion

This survey reviews the implementation of a Transformer-based classification system for detecting rumours on microblogging platforms. In this survey, we implemented rigorous validation procedures to ensure the credibility and reliability of the findings. The initial step involved a comprehensive search strategy, which encompassed academic databases including Scopus and WoS, scholarly journals, and conference proceedings. The search terms were carefully chosen to capture the breadth and depth of the research topic, and inclusion criteria were established to select studies that met the predefined relevance and quality standards. Throughout the literature survey, the findings from different sources were synthesised and analysed to identify contributions and their strengths and weaknesses.

Additionally, we elaborate on the significance of automating rumour detection on microblogging platforms. We emphasize the pivotal role played by text embedding in transforming textual data into numerical representations. Our observation highlights the significance of attention mechanisms in this transformation process, particularly in capturing contextual information.

This survey systematically presents the current approaches of implementing Transformer-based models for rumour detection on microblogging platforms and offers insights into the associated challenges and potential avenues for future research. Although the problem is relatively new, resulting in a smaller volume of published studies compared to other methods, the surveyed literature demonstrates that Transformer-based classification models have achieved remarkable success in detecting rumours on microblogging platforms. Drawing from the findings of the surveyed articles, we are confident that promising results will continue to emerge using Transformer-based models to combat the spread of rumours on microblogging platforms.