On Profiling Bots in Social Media

Abstract

The popularity of social media platforms such as Twitter has led to the proliferation of automated bots, creating both opportunities and challenges in information dissemination, user engagements, and quality of services. Past works on profiling bots had been focused largely on malicious bots, with the assumption that these bots should be removed. In this work, however, we find many bots that are benign, and propose a new, broader categorization of bots based on their behaviors. This includes broadcast, consumption, and spam bots. To facilitate comprehensive analyses of bots and how they compare to human accounts, we develop a systematic profiling framework that includes a rich set of features and classifier bank. We conduct extensive experiments to evaluate the performances of different classifiers under varying time windows, identify the key features of bots, and infer about bots in a larger Twitter population. Our analysis encompasses more than 159K bot and human (non-bot) accounts in Twitter. The results provide interesting insights on the behavioral traits of both benign and malicious bots.

A Predictions on Unlabeled Twitter Accounts

To facilitate our study on a larger Twitter population, we first examined how well our best classfier (i.e., LR) can predict for unlabeled data that it never sees in the (labeled) CV data. Table 4 summarizes the top K prediction results, whereby we varied K from 10 to 50 to verify the robustness of the predictions. For each class, we computed the number of correctly predicted instances (TP) as well as precision at top K, i.e., $Precision = \frac{TP}{K}$.

Table 4. Top K predictions on unlabeled 158,111 Twitter accounts

As shown in Table 4, our LR classifier produces fairly accurate and consistent predictions across different K values. With respect to human accounts, our LR classifier achieved perfect Precision for all K values. Unsurprisingly, we can expect that human accounts constitute the largest proportion of the Twitter population, and thus they should be the easiest to classify. We also obtained good results for the broadcast and consumption bots, with precision scores greater than $75\,\%$ and $95\,\%$ respectively. On the other hand, we observe rather modest Precision scores for spam bots (i.e., 40–$47.5\,\%$). We can attribute this to the insufficient number of instances for spam bots, which form only $\frac{105}{1,613} = 6.51\,\%$ of our labeled data (cf. Table 1). This may (again) be due to our data collection procedure that involved popular users as seeds and/or due to our relatively strict criteria for the characterization of spam bot accounts (cf. Sect. 7.1). Nevertheless, the Precision scores of 40–$47.5\,\%$ remain relatively good, if we compare with that of a random guess for our labeled data (i.e., $6.51\,\%$).

All in all, we find our top K predictions on unlabeled data to be satisfactory. Based on this, we can use our predictions to infer the behavioral profiles of bots in a larger Twitter population, which in this case spans the overall Singapore users. In particular, we analyze the entropy-based dynamic tweet features, namely the entropy distributions of the tweet, retweet, mention, hashtag and url activities, which constitute the majority group of the top discriminative features in Fig. 5. Figure 6 presents the cumulative distribution functions of these features. The detailed analysis of the distributions can be found in Sect. 7.3.

Fig. 6.

Distribution of entropy-based features for 158,111 Twitter accounts