Social media platforms are getting an important communication medium in our daily life, and their increasing popularity makes them an ideal platform for spammers to spread spam messages, known as spam problems. Moreover, messages on social media are vague and messy, so a good representation of the text may be the first step to address spam problem. While traditional weighting methods suffer from both high dimensionality and high sparsity problems, traditional word embedding methods suffer from context independence and out of vocabulary problems. To overcome these problems, in this study, we propose a novel architecture based on a context-dependent representation of text using the BERT model. The model was tested using the Twitter dataset, and experimental results show that the proposed method outperforms traditional weighting methods, traditional word embedding based methods as well as the existing state of the art methods used to detect spam on the twitter platform.
Social media are interactive computer-mediated technologies that facilitate the creation or sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. Twitter is one of the most popular social media nowadays. Twitter reported that its worldwide monetizable daily active users (mDAUs) grew by 24% to 166 million in Q1 2020. Each twitter user has, on average, 208 followers, and they post 140 million tweets daily. This popularity of the Twitter platform has made it a suitable environment for spreading spam messages, which have become a challenging problem due to the messy and ambiguity of short text messages on social media.
Social spam messages might be defined as irrelevant or unsolicited messages sent over social media such as malicious links, advertisements, or any low-quality content. Unlike long messages like e-mails, social spam messages are more sparse and ambiguous, and thus spam classification problem in social networks becomes a more challenging problem. One of the important tasks that could be utilized to handle short text on social media is word representation.
The traditional word representation methods are based on the Bag of Word (BoW) model in which each word or n-gram is linked to a vector index and marked as 0 or 1 depending on whether it occurs in a given document. Although it produces acceptable results, it suffers from some problems like high dimensionality and high sparsity. Word Embedding methods solve these problems by representing the words as dense vectors, where a vector represents the projection of the word into a continuous vector space. Word2vec is the first-word embedding model introduced by Tomas Mikolov in 2013 at Google.
There are two main training algorithms for Word2vec: Skip Gram and Continous Bag Of Words (CBOW). Fast Text, an extension of the word2vec model, is another common word embedding method developed by Facebook in 2016. In Fast Text, each word in the corpus is represented as an n-gram of characters, which helps to capture the meaning of shorter words and allows the embeddings to understand suffixes and prefixes. The other benefit of using the Fast Text model is the ability to work with the rare words that haven't seen them before during the training data. Another common model for word representation is Global Vectors (GloVe) which was developed by Pennington et al. at Stanford University. Unlike word2vec, a predictive model, GloVe is a count-based model and learns its vectors by dimensionality reduction on the co-occurrence counts matrix. The topic model is another method that might be used for representing the texts. The topic model is a statistical model and discovers the abstract "topics" that occur in a collection of documents. The most common form of the topic modeling is Latent Dirichlet Allocation (LDA), LDA aims to find topics in which a document belongs to, based on the words in it. Its main limitation is the inability to work with short text like tweets.
On the other hand, traditional word embedding models suffer from so-called the context-independent problem in which they generate only one vector for each word, regardless of its meaning and its position in the sentence, and also suffer from the out of vocabulary (OOV) problem that the models cannot process new data not previously present in the models' dictionary. To overcome these problems, we propose to use Bidirectional Encoder Representations from Transformers (BERT) to represent the tweet text. BERT is a context-dependent model that generates different word embeddings for a word depending on its meaning and its location in a sentence. Any word that does not occur in the vocabulary is broken down into sub-words to deal with the out of vocabulary problem. In short, in this study, we implement various types of deep learning-based models with the help of different word embedding methods to address the problem of spam messages on social networks and test the models on twitter and SMS datasets. Then, we propose a novel architecture based on the BERT model to detect spam messages on twitter. Finally, we compare traditional weighting methods, traditional word embedding-based methods, and some state of the art methods with our proposed method in terms of their performances. The rest of the paper is organized as follows. Section 2 presents some existing methods in the literature to detect spam on social networks. Section 3 describes the used methodologies. Section 4 is devoted to the proposed models. It presents the architecture of the models and describes models' parameters. Section 5 presents experimental results, Section 6 includes discussion, and Section 7 includes conclusion and future works.
Spam problem on social networks has gained attention in the last decade. Researchers in various areas made a great effort to address spam on social media. In this study, we will focus on spam detection on the Twitter network. To detect spam on Twitter, most of the studies were oriented on two main directions, detecting spammer accounts as in [1,2,3,4,5,6], and detecting spammed tweet messages as in [7,8,9,10,11].
The majority of the researches treated spam as messages that contain malicious links  and others like  generalized the spams’ definition by including advertisements, automatically generated content, and any low-quality content.
Due to the convincing performances provided by machine learning algorithms, many researchers have used machine learning methods to detect spam in social networks. For example in , the authors used direct features which were extracted from crawled JSON tweet to detect content polluter using random forest classifier, and in  a framework based on tweet-based and user-based features along with text-based features was proposed to detect spam on the tweet text and evaluated using some machine learning classifiers and neural network. Table 1 summarizes some studies based on machine learning algorithms to detect spam on social networks.
Along with the remarkable development of deep learning algorithms in the field of natural language processing, various types of text classifications based on deep neural networks emerged. Since neural networks are designed to learn from numerical data, word embedding methods improve the ability of networks to learn from a text by representing the words as lower-dimensional vectors. Besides, embedding methods enable us to measure the similarity between different words by computing the distance between embedding vectors. With fixed-length embedding vectors as input, deep learning-based models with word embedding methods achieve competitive results in text classification problems which prompted researchers to integrate deep learning methods to their detection architecture to achieve better performances, for example In , a recurrent convolutional neural network was used in document classification, and in , the model called “C-LSTM” which combines Recurrent Neural Network (RNN) with Convolutional Neural Network (CNN) was proposed for sentence representation and text classification. The model utilized CNN to extract a sequence of higher-level phrase representations and used a long short-term neural network (LSTM) to obtain the sentence representation. In , the authors proposed two combination models, named BLSTM-2DPooling and BLSTM- 2DCNN, where the 2D max-pooling operation was applied to obtain a fixed-length representation of the text, and 2D convolution was utilized to sample more meaningful information of the matrix. In , the authors proposed semantic Convolutional Neural Network (SCNN) for spam classification on Twitter with the help of WordNet and ConceptNet knowledge bases, and in [15, 16] the syntax of each tweet was learned through Word2Vector model and trained by deep learning, then a binary classifier was used to differentiate between spam and non-spam tweet.
Table 2 summarizes some deep learning-based methods to detect spam on social networks.
Furthermore, the topic model, especially LDA, was used significantly in text classification problems. For example in  a new algorithm based on the LDA and the Support Vector Machine was used in the Arabic texts classification, and in  an improved method based on the LDA topic model and K-Nearest Neighbor algorithm was proposed to handle the problem of short text classification. On the other hand in , a so-called Labeled-LDA to enhance the traditional LDA is proposed to integrate the class information, and based on it a new algorithm was introduced to figure out the latent topics’ quantities of each class synergistically.
Herein this study, we propose to use a novel method based on BERT encoder to digitize tweets in detecting spam on Twitter. To the best of our knowledge, it is the first time that this method has been used on the Twitter spam detection domain. Also, the most common word representation methods with different architectures were implemented to analyze the effect of word representation methods on spam detection tasks.
We used the dataset published by Chen et al. in their study  in which 100,000 tweets were collected through public streams provided by Twitter Streaming API. The original dataset contains only Tweet IDs with labels without tweets information, so we built a crawler that collects tweet information from Twitter based on Tweet ID using a Python program with the help of the Tweepy library. We couldn't access all tweets in the dataset, where only 58,159 tweets were reached, perhaps due to the deletion of these tweets by the owners or by Twitter. Since the extracted dataset was imbalanced, data augmentation was applied to the text to balance the dataset to improve the performance of classification tasks. We also used the SMS spam dataset, which was available in the UCI repository. Table 3 presents the details of the datasets. The datasets were split into 90% training and 10% testing in all models, except for the proposed model the splitting was 80% training and 20% testing. The table presents the number of spam messages, the number of non-spam messages for both twitter and SMS spam datasets, and the total for each of them.
The first step in any text classification task is the preprocessing step. The preprocessing step is very important because it helps to clean unnecessary data to improve the performance of the classification task.
The following preprocessing steps were applied to the text: removing punctuations, removing extra spaces, removing URLs, removing special characters and emojis, removing stop words, and converting the text to lower case. After that, the text was converted to a sequence of tokens and padded with zeroes to ensure equal length.
Traditional weighting–based method
In this study, we experimented with the most popular BoW based weighting methods such as count vectorizer and TF-IDF with one of the most common machine learning classifiers used in text classification problems such as Naïve Bayes (NB) classifier.
Traditional word embedding based methods
Three different deep neural network architectures were implemented and experimented with the most common word embedding methods such as word2vec, Glove, and fast Text. We chose simple architectures from each of the following types of neural networks such as a convolutional neural network (CNN), Long short term memory neural network (LSTM), and bidirectional long short term neural network (bi-LSTM). Table 4 shows the hyperparameters used in each model. Where Adam is used as an optimizer in each model, binary cross-entropy is used as a loss function, and a constant value of dropout is used in all models.
Bidirectional encoder representations from transformers
BERT stands for Bidirectional Encoder Representations from Transformers, which is Google’s neural network-based technique for natural language processing (NLP). It was created and published in 2018 by Devlin et al. in Google .
Traditional word embedding models are context-independent models output just one vector for each word, regardless of the position of the word in the sentence and regardless of the meaning of that word. In contrast, the BERT model can generate different vectors (embeddings) for the word depending on the meaning and the order of that word in the sentence. BERT is designed to pre-train deep bidirectional representations from the unlabeled text by jointly conditioning on both left and right context in all layers . Figure 1 shows the block diagram of the BERT model where E1 is the embedding representation, T1 is the final output, and Trm are the intermediate representations of the same token.
The proposed model
Figure 2 shows the proposed architecture. The model consists of an input layer, a BERT encoder layer with sequence length equal to 20 tokens, then 768 nodes in a hidden layer with Relu as an activation function, and finally one output layer with sigmoid activation function applied to classify the text to spam or ham.
Table 5 displays the hyperparameters used in the model where Adam is used as an optimizer, binary cross-entropy is used as a loss function.
Herein, we proposed a novel architecture based on the BERT model for text representation and tested on twitter dataset, then compared the results with traditional feature weighting methods and traditional most common word embedding methods. All experiments carried out using Intel Xeon E5-2680 8 cores and 64 GB of RAM.
Standard evaluation metrics were used to measure our proposed method, such as accuracy, F1-score, recall, and precision. Table 6 presents the confusion matrix for the proposed model. Here, True Positive (TP) indicates the number of tweets that classified correctly into the spam class. The meanings of other quantities like True negative (TN), False positive (FP), and False negative (FN) follows from TP. Accuracy, precision, recall, and F1-score can be computed by equations below.
In this section, we discuss the results obtained in both of the traditional weighing-based methods and neural networks models based on word embedding methods then we will compare the results with the results obtained by the proposed method.
Figure 3 shows the comparisons between the traditional word embedding methods for spam detection on the Twitter dataset. The structures based on the representation of GloVe gave the best results compared to the rest of the models. As for the neural network structure, it appears that the LSTM-based structure is the best structure among all models. As for the SMS dataset, the LSTM structure based on FastText embedding achieved the highest accuracy compared to the rest of the structures, see Fig. 4.
As for the traditional weighing methods, as shown in Table 9, the performance of the Count vectorizer with Naive Bayes (NB) classifier outperforms the other experimented methods on the Twitter data set, while word-level TF-IDF achieved the best results in the SMS data set. On the other hand, by moving to the proposed method as shown in Table 7, the proposed method outperformed all previous methods with a remarkable performance reached 98.1% of accuracy and 98.05% of F1-score (Tables 8 and 9).
Comparison with other works
Table 10 shows the performance comparison between the proposed method and studies using similar methods available in the literature. Jain et al.  proposed an LSTM structure to detect spam messages on twitter and used word2vec for text representation with the help of knowledge bases to enhance the representation of text by adding more semantic. Madisetty et al.  used different word embedding methods such as word2vec and GloVe to represent text and fed into the CNN-based architecture to detect spam messages on twitter (Figure 5). On the other side, Ameen et al.  preferred the word2vec model to represent text and linear classification methods to distinguish spam from non-spam tweets.
Conclusions and future works
With the noticeable increase in the popularity of social networks in recent years, short texts classification problem has become quite common. One of the most common problems experienced by social networks' users is spam messages. An appropriate word representation could be a good solution to handle these messages.
Traditional weighting methods and word embedding methods cannot handle the problem of short text classification effectively because of some previously mentioned limitations. Therefore, to overcome these limitations, novel context-dependent word representation methods were developed like CoVe, ELMO, and BERT.
In this study we proposed to use the BERT encoder to represent the text in the Twitter dataset, then we compared the performance with both traditional weighting methods and traditional word embedding methods. Finally, we compared the results of the proposed model with the results of some state of the art methods in the literature. We can conclude that the proposed method outperforms all other methods. to represent the text in the Twitter dataset, then we compared the performance with both traditional
As future work, we intend to improve the architecture of the proposed model and conduct an empirical study to tune the hyperparameters. Combining traditional weighting methods with contextual word representation could be a new open trend. Furthermore, other modern word representation methods could be integrated with deep learning algorithms.
Yang C, Harkreader R, Gu G (2013) Empirical evaluation and new design for fighting evolving twitter spammers. IEEE Trans Inf Forensics Secur 8(8):1280–1293
Lee S, Kim J (2014) Early filtering of ephemeral malicious accounts on Twitter. Comput Commun 54:48–57
El-Mawass, N. and S. Alaboodi, Hunting for spammers: Detecting evolved spammers on twitter. arXiv preprint arXiv:1512.02573, 2015.
Zheng X et al (2015) Detecting spammers on social networks. Neurocomputing 159:27–34
Kandasamy, K. and P. Koroth. An integrated approach to spam classification on Twitter using URL analysis, natural language processing and machine learning techniques. In: 2014 IEEE Students' Conference on Electrical, Electronics and Computer Science. 2014. IEEE
Miller Z et al (2014) Twitter spammer detection using data stream clustering. Inf Sci 260:64–73
Chen, W., et al. Real-time Twitter Content Polluter Detection Based on Direct Features. In: 2015 2nd International Conference on Information Science and Security (ICISS). 2015. IEEE
Wu, T., et al. Twitter spam detection based on deep learning. In: Proceedings of the australasian computer science week multiconference. 2017
Jain G, Sharma M, Agarwal B (2018) Spam detection on social media using semantic convolutional neural network. Int J Knowl Discov Bioinf (IJKDB) 8(1):12–26
Gupta, H., et al. A framework for real-time spam detection in Twitter. In: 2018 10th International Conference on Communication Systems & Networks (COMSNETS). 2018. IEEE
Chen W et al (2017) A study on real-time low-quality content detection on Twitter from the users’ perspective. PLoS ONE 12(8):e0182487
Lai, S., et al. Recurrent convolutional neural networks for text classification. In: Twenty-ninth AAAI conference on artificial intelligence. 2015
Zhou, C., et al., A C-LSTM neural network for text classification. arXiv preprint arXiv:1511.08630, 2015.
Zhou, P., et al., Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. arXiv preprint arXiv:1611.06639, 2016.
Wu T et al (2017) Detecting spamming activities in twitter based on deep-learning technique. Concurr Comput Pract Exp 29(19):e4209
Ameen, A.K. and B. Kaya Spam detection in online social networks by deep learning. In: 2018 International Conference on Artificial Intelligence and Data Processing (IDAP). 2018. IEEE.
Zrigui M et al (2012) Arabic text classification framework based on latent dirichlet allocation. J Comput Inf Technol 20(2):125–140
Chen Q. L. Yao, and J. Yang. Short text classification based on LDA topic model. In: 2016 International Conference on Audio, Language and Image Processing (ICALIP). 2016. IEEE
Li W, Sun L, Zhang D-K (2008) Text classification based on labeled-LDA model. Chin J Comp Chin Ed 31(4):620
Mateen M. et al. A hybrid approach for spam detection for Twitter. In: 2017 14th International Bhurban Conference on Applied Sciences and Technology (IBCAST). 2017. IEEE
Uysal AK (2018) Feature selection for comment spam filtering on YouTube. Data Sci Appl 1(1):4–8
Barushka A, Hajek P (2019) Spam detection on social networks using cost-sensitive feature selection and ensemble-based regularized deep neural networks. Neural Comput Appl 32:4239–4257. https://doi.org/10.1007/s00521-019-04331-5
Madisetty S, Desarkar MS (2018) A neural network-based ensemble approach for spam detection in Twitter. IEEE Trans Comput Soc Syst 5(4):973–984
Devlin J. et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018
The authors would like to thank Chen et al.  and UCI repository for sharing their datasets.
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Conflict of interest
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ghanem, R., Erbay, H. Context-dependent model for spam detection on social networks. SN Appl. Sci. 2, 1587 (2020). https://doi.org/10.1007/s42452-020-03374-x
- Spam detection
- Word embedding
- Bidirectional encoder representations from transformers