SentiBench - a benchmark comparison of state-of-the-practice sentiment analysis methods

Abstract

In the last few years thousands of scientific papers have investigated sentiment analysis, several startups that measure opinions on real data have emerged and a number of innovative products related to this theme have been developed. There are multiple methods for measuring sentiments, including lexical-based and supervised machine learning methods. Despite the vast interest on the theme and wide popularity of some methods, it is unclear which one is better for identifying the polarity (i.e., positive or negative) of a message. Accordingly, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, as they are used in practice, across multiple datasets originated from different data sources. Such a comparison is key for understanding the potential limitations, advantages, and disadvantages of popular methods. This article aims at filling this gap by presenting a benchmark comparison of twenty-four popular sentiment analysis methods (which we call the state-of-the-practice methods). Our evaluation is based on a benchmark of eighteen labeled datasets, covering messages posted on social networks, movie and product reviews, as well as opinions and comments in news articles. Our results highlight the extent to which the prediction performance of these methods varies considerably across datasets. Aiming at boosting the development of this research area, we open the methods’ codes and datasets used in this article, deploying them in a benchmark system, which provides an open API for accessing and comparing sentence-level sentiment analysis methods.

Introduction

Sentiment analysis has become an extremely popular tool, applied in several analytical domains, especially on the Web and social media. To illustrate the growth of interest in the field, Figure 1 shows the steady growth on the number of searches on the topic, according to Google Trends,¹ mainly after the popularization of online social networks (OSNs). More than 7,000 articles have been written about sentiment analysis and various startups are developing tools and strategies to extract sentiments from text [1].

Figure 1

Searches on Google for the Query: ‘Sentiment Analysis’. This figure shows the steady growth on the number of searches on the topic, according to Google Trends, mainly after the popularization of online social networks (OSNs).

The number of possible applications of such a technique is also considerable. Many of them are focused on monitoring the reputation or opinion of a company or a brand with the analysis of reviews of consumer products or services [2]. Sentiment analysis can also provide analytical perspectives for financial investors who want to discover and respond to market opinions [3, 4]. Another important set of applications is in politics, where marketing campaigns are interested in tracking sentiments expressed by voters associated with candidates [5].

Due to the enormous interest and applicability, there has been a corresponding increase in the number of proposed sentiment analysis methods in the last years. The proposed methods rely on many different techniques from different computer science fields. Some of them employ machine learning methods that often rely on supervised classification approaches, requiring labeled data to train classifiers [6]. Others are lexical-based methods that make use of predefined lists of words, in which each word is associated with a specific sentiment. The lexical methods vary according to the context in which they were created. For instance, LIWC [7] was originally proposed to analyze sentiment patterns in formally written English texts, whereas PANAS-t [8] and POMS-ex [9] were proposed as psychometric scales adapted to the Web context.

Overall, the above techniques are acceptable by the research community and it is common to see concurrent important papers, sometimes published in the same computer science conference, using completely different methods. For example, the famous Facebook experiment [10] which manipulated users feeds to study emotional contagion, used LIWC [7]. Concurrently, Reis et al. used SentiStrength [11] to measure the negativeness or positiveness of online news headlines [12, 13], whereas Tamersoy [14] explored VADER’s lexicon [15] to study patterns of smoking and drinking abstinence in social media.

As the state-of-the-art has not been clearly established, researchers tend to accept any popular method as a valid methodology to measure sentiments. However, little is known about the relative performance of the several existing sentiment analysis methods. In fact, most of the newly proposed methods are rarely compared with all other pre-existing ones using a large number of existing datasets. This is a very unusual situation from a scientific perspective, in which benchmark comparisons are the rule. In fact, most applications and experiments reported in the literature make use of previously developed methods exactly how they were released with no changes and adaptations and with none or almost none parameter setting. In other words, the methods have been used as a black-box, without a deeper investigation on their suitability to a particular context or application.

To sum up, existing methods have been widely deployed for developing applications without a deeper understanding regarding their applicability in different contexts or their advantages, disadvantages, and limitations in comparison with each another. Thus, there is a strong need to conduct a thorough apple-to-apple comparison of sentiment analysis methods, as they are used in practice, across multiple datasets originated from different data sources.

This state-of-the-practice situation is what we propose to investigate in this article. We do this by providing a thorough benchmark comparison of twenty-four state-of-the-practice methods using eighteen labeled datasets. In particular, given the recent popularity of online social networks and of short texts on the Web, many methods are focused in detecting sentiments at the sentence-level, usually used to measure the sentiment of small sets of sentences in which the topic is known a priori. We focus on such context - thus, our datasets cover messages posted on social networks, movie and product reviews, and opinions and comments in news articles, TED talks, and blogs. We survey an extensive literature on sentiment analysis to identify existing sentence-level methods covering several different techniques. We contacted authors asking for their codes when available or we implemented existing methods when they were unavailable but could be reproduced based on their descriptions in the original published paper. We should emphasize that our work focus on off-the-shelf methods as they are used in practice. This excludes most of the supervised methods which require labeled sets for training, as these are usually not available for practitioners. Moreover, most of the supervised solutions do not share the source code or a trained model to be used with no supervision.

Our experimental results unveil a number of important findings. First, we show that there is no single method that always achieves the best prediction performance for all different datasets, a result consistent with the ‘there is no free lunch theorem’ [16]. We also show that existing methods vary widely regarding their agreement, even across similar datasets. This suggests that the same content could be interpreted very differently depending on the choice of a sentiment method. We noted that most methods are more accurate in correctly classifying positive than negative text, suggesting that current approaches tend to be biased in their analysis towards positivity. Finally, we quantify the relative prediction performance of existing efforts in the field across different types of datasets, identifying those with higher prediction performance across different datasets.

Based on these observations, our final contribution consists on releasing our gold standard dataset and the codes of the compared methods.² We also created a Web system through which we allow other researchers to easily use our data and codes to compare results with the existing methods.³ More importantly, by using our system one could easily test which method would be the most suitable to a particular dataset and/or application. We hope that our tool will not only help researchers and practitioners for accessing and comparing a wide range of sentiment analysis techniques, but can also help towards the development of this research field as a whole.

The remainder of this paper is organized as follows. In Section 2, we briefly describe related efforts. Then, in Section 3 we describe the sentiment analysis methods we compare. Section 4 presents the gold standard data used for comparison. Section 5 summarizes our results and findings. Finally, Section 6 concludes the article and discusses directions for future work.

Background and related work

Next we discuss important definitions and justify the focus of our benchmark comparison. We also briefly survey existing related efforts that compare sentiment analysis methods.

Focus on sentence-level sentiment analysis

Since sentiment analysis can be applied to different tasks, we restrict our focus on comparing those efforts related to detect the polarity (i.e. positivity or negativity) of a given short text (i.e. sentence-level). Polarity detection is a common function across all sentiment methods considered in our work, providing valuable information to a number of different applications, specially those that explore short messages that are commonly available in social media [1].

Sentence-level sentiment analysis can be performed with supervision (i.e. requiring labeled training data) or not. An advantage of supervised methods is their ability to adapt and create trained models for specific purposes and contexts. A drawback is the need of labeled data, which might be highly costly, or even prohibitive, for some tasks. On the other hand, the lexical-based methods make use of a pre-defined list of words, where each word is associated with a specific sentiment. The lexical methods vary according to the context in which they were created. For instance, LIWC [7] was originally proposed to analyze sentiment patterns in English texts, whereas PANAS-t [8] and POMS-ex [9] are psychometric scales adapted to the Web context. Although lexical-based methods do not rely on labeled data, it is hard to create a unique lexical-based dictionary to be used for all different contexts.

We focus our effort on evaluating unsupervised efforts as they can be easily deployed in Web services and applications without the need of human labeling or any other type of manual intervention. As described in Section 3, some of the methods we consider have used machine learning to build lexicon dictionaries or even to build models and tune specific parameters. We incorporate those methods in our study, since they have been released as black-box tools that can be used in an unsupervised manner.

Existing efforts on comparison of methods

Despite the large number of existing methods, only a limited number of them have performed a comparison among sentiment analysis methods, usually with restricted datasets. Overall, lexical methods and machine learning approaches have been evolving in parallel in the last years, and it comes as no surprise that studies have started to compare their performance on specific datasets and use one or another strategy as baseline for comparison. A recent survey summarizes several of these efforts [17] and conclude that a systematic comparative study that implements and evaluates all relevant algorithms under the same framework is still missing in the literature. As new methods emerge and compare themselves only against one, at most two other methods, using different evaluation datasets and experimental methodologies, it is hard to conclude if a single method triumphs over the remaining ones, or even in specific scenarios. To the best of our knowledge, our effort is the first of kind to create a benchmark that provides such thorough comparison.

An important effort worth mentioning consists of an annual workshop - The International Workshop on Semantic Evaluation (SemEval). It consists of a series of exercises grouped in tracks, including sentiment analysis, text similarity, among others, that put several together competitors against each other. Some new methods such as Umigon [18] have been proposed after obtaining good results on some of these tracks. Although, SemEval has been playing an important role for identifying relevant methods, it requires authors to register for the challenge and many popular methods have not been evaluated in these exercises. Additionally, SemEval labeled datasets are usually focused on one specific type of data, such as tweets, and do not represent a wide range of social media data. In our evaluation effort, we consider one dataset from SemEval 2013 and two methods that participated in the competition in that same year.

Ahmadi et al. [19] performed a comparison of Twitter-based sentiment analysis tools. They selected twenty tools and tested them across five Twitter datasets. This benchmark is the work that most approximate from ours, but it is different in some meaningful aspects. Firstly, we embraced distinct contexts such as reviews, comments and social networks aiming at providing a broader evaluation of the tools. Secondly, the methods they selected included supervised and unsupervised approaches which, in our view, could be unfair for the unsupervised ones. Although the results have been presented separately, the supervised methods, as mentioned by authors, required extensive parameter tuning and validation in a training environment. Therefore, supervised approaches tend to adapt to the context they were applied to. As previously highlighted, our focus is on off-the-shelf tools as they have been extensively and recently used. Many researchers and practitioners have also used supervised approaches but this is out of scope of our work. Finally, most of the unsupervised methods selected in the Twitter Benchmark are paid tools, except from two of them, both of which were developed as a result of published academic research. Oppositely we made an extensive bibliography review to include relevant academic outcomes without excluding the most used commercial options.

Finally, in a previous effort [20], we compared eight sentence-level sentiment analysis methods, based on one public dataset used to evaluate SentiStrength [11]. This article largely extends our previous work by comparing a much larger set of methods across many different datasets, providing a much deeper benchmark evaluation of current popular sentiment analysis methods. The methods used in this paper were also incorporated as part of an existing system, namely iFeel [21].

Sentiment analysis methods

This section provides a brief description of the twenty-four sentence-level sentiment analysis methods investigated in this article. Our effort to identify important sentence-level sentiment analysis methods consisted of systematically search for them in the main conferences in the field and then checking for papers that cited them as well as their own references. Some of the methods are available for download on the Web; others were kindly shared by their authors under request; and a small part of them were implemented by us based on their descriptions in the original paper. This usually happened when authors shared only the lexical dictionaries they created, letting the implementation of the method that use the lexical resource to ourselves.

Table 1 and Table 2 present an overview of these methods, providing a description of each method as well as the techniques they employ (L for Lexicon Dictionary and ML for Machine Learning), their outputs (e.g. −1, 0, 1, meaning negative, neutral, and positive, respectively), the datasets they used to validate, the baseline methods used for comparison and finally lexicon details, as well as the Lexicon size column describing the number of terms contained in the method’s lexicon. The methods are organized in chronological order to allow a better overview of the existing efforts over the years. We can note that the methods generate different outputs formats. We colored in blue the positive outputs, in black the neutral ones, and in red those that are negative. Note that we included LIWC and LIWC15 entries in Table 2, which represents the former version, launched in 2007, and the latest version, from 2015, respectively. We considered both versions because the first one was extensively used in the literature. This also allows to compare the improvements between both versions.

Table 1 Overview of the sentence-level methods available in the literature

Table 2 Overview of the sentence-level methods available in the literature

Adapting lexicons for the sentence level task

Since we are comparing sentiment analysis methods on a sentence-level basis, we need to work with mechanisms that are able to receive sentences as input and produce polarities as output. Some of the approaches considered in this paper, shown in Table 2, are complex dictionaries built with great effort. However, a lexicon alone has no natural ability to infer polarity in sentence level tasks. The purpose of a lexicon goes beyond the detection of polarity of a sentence [1, 56], but it can also be used with that purpose [57, 58].

Several existing sentence-level sentiment analysis methods, like VADER [15] and SO-CAL [31], combine a lexicon and the processing of the sentence characteristics to determine a sentence polarity. These approaches make use of a series of intensifiers, punctuation transformation, emoticons, and many other heuristics.

Thus, to evaluate each lexicon dictionaries as the basis for a sentence-level sentiment analysis method, we considered the VADER’s implementation. In other words, we used VADER’s code for determining if a sentence is positive or not considering different lexicons. The reasons for choosing VADER are twofold: (i) the fact it is an open source tool, allowing easy replication of the procedures we performed in our study; and (ii) VADER’s expressive results observed in previous experiments.

VADER’s heuristics were proposed based on qualitative analyses of textual properties and characteristics which affect the perceived sentiment intensity of the text. VADER’s author identified five heuristics based on grammatical and syntactical cues to convey changes to sentiment intensity that go beyond the bag-of-words model. The heuristics include treatments for: (1) punctuation (e.g. number of ‘!’s); (2) capitalization (e.g. ‘I HATE YOU’ is more intense than ‘i hate you’); (3) degree modifiers (e.g. ‘The service here is extremely good’ is more intense than ‘The service here is good’); (4) constructive conjunction ‘but’ to shift the polarity; (5) tri-gram examination to identify negation (e.g. ‘The food here isn’t really all that great’). We choose VADER as a basis for such heuristics as it is one of the most recent methods among those we considered. Moreover, it is becoming widely used, being even implemented as part of the well-known NLTK python library.⁴

We applied such heuristics to the following lexicons: ANEW_SUB, AFINN, Emolex, EmoticonsDS, NRC Hashtag, Opinion Lexicon, PANAS-t, Sentiment 140 Lexicon and SentiWordNet. We notice that those strategies drastically improved most of the results of the lexicons for sentence-level sentiment analysis in comparison with a simple baseline approach that averages the occurrence of positive and negative words to classify the polarity of a sentence. The results for the simplest usage of the above lexicons as plain methods are available in the last four tables in Additional file 1 of the electronic version of the manuscript. LIWC dictionary was not included in these adaptations due to its very restrictive license, which does not allow any derivative work based on the original application and lexicon. Table 2 has also a column (Lexicon size) that describes the number of terms contained in the proposed dictionary.

Output adaptations

It is worth noticing that the output of each method varies drastically depending on the goal it was developed for and the approach it employs. PANAS-t, for instance, associates each word with eleven moods as described in Table 1 and it was designed to track any increase or decrease in sentiments over time. Emolex lexicon provides the association of each word with eight sentiments. The word ‘unhappy’ for example is related to anger, disgust, and sadness and it is not related to joy, surprise, etc. SentiWordNet links each word with a synset (i.e. a set of synonyms) characterized by a positive and a negative score, both of them represented with a value between 0 and 1.

The aforementioned lexicons were used as dictionary input to VADER’s code. We had to adapt the way the words are processed as follows. For PANAS-t we assumed that joviality, assurance, serenity, and surprise are positive affect. Fear, sadness, guilt, hostility, shyness, and fatigue are negative affect. Attentiveness was considered neutral. In the case of Emolex, we considered two other entries released by the authors. The first one defines the positivity of a word (0 or 1) and the second characterizes the negativity (0 or 1). For SentiWordNet we calculate an overall score to the word by subtracting the positive value from negative value defined to that word. For example, the positive value for the word faithful is 0.625 while its negative score is 0.0. Then the overall score is 0.625. Finally, for ANEW_SUB we employed only the valence emotion of each word. This metric ranges from 1 to 9 and indicates the level of pleasantness of a specific word - we considered the values one to four as negative, five as neutral, and six to nine as positive.

Other lexicons included in our evaluation already provide positive and negative scores such as SentiWordNet or an overall score ranging from a negative to a positive value. After applying VADER’s heuristics for each one of these lexicons we get scores in the same way VADER’s output (see Table 2).

Other methods also required some output handling. The available implementation of OpinionFinder,⁵ for instance, generates polarity outputs (−1, 0, or 1) for each sentiment clue found in a sentence so that a single sentence can have more than one clue. We considered the polarity of a single sentence as the sum of the polarities of all the clues.

The outputs from the remaining methods were easily adapted and converted to positive, negative or neutral. SO-CAL and Pattern.en delivery float numbers greater than a threshold, indicating positive, and lesser than the threshold, indicating negative. LIWC, SenticNet, SASA, USent, SentiStrength, Umigon, VADER and Semantria already provide fixed outputs indicating one of three desired classes while Stanford Recursive Deep Model yields very negative and very positive which in our experiments are handled as negative and positive, respectively.

Paid softwares

Seven out of the twenty-four methods evaluated in this work are closed paid softwares: LIWC (2007 and 2015), Semantria, SenticNet 3.0, Sentiment140 and SentiStrength. Although SentiStrength is paid, it has a free of charge academic license. SenticNet’s authors kindly processed all datasets with the commercial version and return the polarities for us. For SentiStrength we used the Java version from May 2013 in a package with all features of the commercial version. For LIWC we acquired the licenses from 2007 (LIWC07) and 2015 (LIWC15) versions. Finally, for Semantria and Sentiment140 we used a trial account free of charge for a limited number of sentences, which was sufficient to run our experiments.

Methods not included

Despite our effort to include in our comparison most of the highly cited and important methods we could not include a few of them for different reasons. Profile of Mood States (POMS-ex) [9] is not available on the Web or under request and could not be re-implemented based on their descriptions in the original papers. The same situation occurs with the Learning Sentiment-Specific Word Embedding for Twitter Sentiment Classification [59]. NRC SVM [40] is not available as well, although the lexical resources used by the authors are available and were considered in our evaluation resulting in the methods: NRC Hashtag and Sentiment140. The authors of the Convolutional Neural Network for Modeling Sentences [60] and of the Effective Use of Word Order for Text Categorization with Convolutional Neural Networks [61] have made their source code available but the first one lacks the train files and the second one requires a GPU to execute. There are a few other methods for sentiment detection proposed in the literature and not considered here. Most of them consists of variations of the techniques used by the above methods, such as WordNet-Affect [62] and Happiness Index [63].

Datasets and comparison among methods

From Table 2 we can note that the validation strategy, the datasets used, and the comparison with baselines performed by these methods vary greatly, from toy examples to large labeled datasets. PANAS-t and Emoticons DS used manually unlabeled Twitter data to validate their methods, by presenting evaluations of events in which some bias towards positivity and negativity would be expected. PANAS-t is tested with unlabeled Twitter data related to Michael Jackson’s death and the release of a Harry Potter movie whereas Emoticons DS verified the influence of weather and time on the aggregate sentiment from Twitter. Lexical dictionaries were validated in very different ways. AFINN [29] compared its Lexicon with other dictionaries. Emoticon Distance Supervised [32] used Pearson Correlation between human labeling and the predicted value. SentiWordNet [24] validates the proposed dictionary with comparisons with other dictionaries, but it also used human validation of the proposed lexicon. These efforts attempt to validate the created lexicon, without comparing the lexicon as a sentiment analysis method by itself. VADER [15] compared results with lexical approaches considering labeled datasets from different social media data. SenticNet [55] was compared with SentiStrength [11] with a specific dataset related to patient opinions, which could not be made available. Stanford Recursive Deep Model [41] and SentiStrength [11] were both compared with standard machine learning approaches, with their own datasets.

This scenario, where every new developed solution compares itself with different solutions using different datasets, happens because there is no standard benchmark for evaluating new methods. This problem is exacerbated because many methods have been proposed in different research communities (e.g. NLP, Information Science, Information Retrieval, Machine Learning), exploiting different techniques, with low knowledge about related efforts in other communities. Next, we describe how we created a large gold standard to properly compare all the considered sentiment analysis methods.

Gold standard data

A key aspect in evaluating sentiment analysis methods consists of using accurate gold standard labeled datasets. Several existing efforts have generated labeled data produced by experts or non-experts evaluators. Previous studies suggest that both efforts are valid as non-expert labeling may be as effective as annotations produced by experts for affect recognition, a very related task [64]. Thus, our effort to build a large and representative gold standard dataset consists of obtaining labeled data from trustful previous efforts that cover a wide range of sources and kinds of data. We also attempt to assess the ‘quality’ of our gold standard in terms of the accuracy of the labeling process.

Table 3 summarizes the main characteristics of the eighteen exploited datasets, such as number of messages and the average number of words per message in each dataset. It also defines a simpler nomenclature that is used in the remainder of this paper. The table also presents the methodology employed in the classification. Human labeling was implemented in almost all datasets, usually done with the use of non-expert reviewers. Reviews_I dataset relies on five stars rates, in which users rate and provide a comment about an entity of interest (e.g. a movie or an establishment).

Table 3 Labeled datasets

Labeling based on Amazon Mechanical Turk (AMT) was used in seven out of the eighteen datasets, while volunteers and other strategies that involve non-expert evaluators were used in ten datasets. Usually, an agreement strategy (i.e. majority voting) is applied to ensure that, in the end, each sentence has an agreed-upon polarity assigned to it. The number of annotators used to build the datasets is also shown in Table 3.

Tweets_DBT was the unique dataset built with a combination of AMT Labeling with Expert validation [66]. They selected 200 random tweets to be classified by experts and compared with AMT results to ensure accurate ratings. We note that the Tweets_Semeval dataset was provided as a list of Twitter IDs, due to the Twitter policies related to data sharing. While crawling the respective tweets, a small part of them could not be accessed, as they were deleted. We plan to release all gold standard datasets in a request basis, which is in agreement with Twitter policies.

In order to assess the extent to which these datasets are trustful, we used a strategy similar to the one used by Tweets_DBT. Our goal was not to redo all the performed human evaluation, but simply inspecting a small sample of them to infer the level of agreement with our own evaluation. We randomly select 1% of all sentences to be evaluated by experts (two of the authors) as an attempt to assess if these gold standard data are really trustful. It is important to mention that we did not have access to the instructions provided by the authors. We also could not get access to small amount of the raw data in a few datasets, which was discarded. Finally, our manual inspection unveiled a few sentences in idioms other than English in a few datasets, such as Tweets_STA and TED, which were obviously discarded.

Column CK from Table 3 exhibits the level of agreement of each dataset in our evaluation by means of Cohen’s Kappa, an extensively used metric to calculate inter-anotator agreement. After a close look in the cases of disagreement with the evaluations in the Gold standard, we realized that other interpretations could be possible for the given text, finding cases of sentences with mixed polarity. Some of them are strongly linked to original context and are very hard to evaluate. Some NYT comments, for instance, are directly related to the news they were inserted to. We can also note that some of the datasets do not contain neutral messages. This might be a characteristic of the data or even a result of how annotators were instructed to label their pieces of text. Most of the cases of disagreement involve neutral messages. Thus, we considered these cases, as well as the amount of disagreement we had with the gold standard data, reasonable and expected, specially when taking into account that Landis and Koch [70] suggest that Kappa values between 0.4 and 0.6 indicate moderate agreement and values amid 0.60 and 0.8 correspond to substantial agreements.

Comparison results

Next, we present comparison results for the twenty-four methods considered in this paper based on the eighteen considered gold standard datasets.

Experimental details

At least three distinct approaches have been proposed to deal with sentiment analysis of sentences. The first of them, applied by OpinionFinder and Pattern.en, for instance, splits this task into two steps: (i) identifying sentences with no sentiment, also named as objective vs. neutral sentences and then (ii) detecting the polarity (positive or negative), only for the subjective sentences. Another common way to detect sentence polarity considers three distinct classes (positive, negative and neutral) in a single task, an approach used by VADER, SO-CAL, USent and others. Finally, some methods like SenticNet and LIWC, classify a sentence as positive or negative only, assuming that only polarized sentences are presented, given the context of a given application. As an example, reviews of products are expected to contain only polarized opinion.

Aiming at providing a more thorough comparison among these distinct approaches, we perform two rounds of tests. In the first we consider the performance of methods to identify 3-class (positive, negative and neutral). The second considers only positive and negative as output and assumes that a previous step of removing the neutral messages needs to be executed firstly. In the 3-class experiments we used only datasets containing a considerable number of neutral messages (which excludes Tweets_RND_II, Amazon, and Reviews_II). Despite being 2-class methods, as highlighted in Table 2, we decided to include LIWC, Emoticons and SenticNet in the 3-class experiments to present a full set of comparative experiments. LIWC, Emoticons, and SenticNet cannot define, for some sentences, their positive or negative polarity, considering it as undefined. It occurs due to the absence in the sentence of emoticons (in the case of Emoticons method) or of words belonging to the methods’ sentiment lexicon. As neutral (objective) sentences do not contain sentiments, we assumed, in the case of these 2-class methods, that sentences with undefined polarities are equivalent to neutral sentences.

The 2-class experiments, on the other hand, were performed with all datasets described in Table 3 excluding the neutral sentences. We also included all methods in these experiments, even those that produce neutral outputs. As discussed before, when 2-class methods cannot detect the polarity (positive or negative) of a sentences they usually assign it to an undefined polarity. As we know all sentences in the 2-class experiments are positive or negative, we create the coverage metric to determine the percentage of sentences a method can in fact classify as positive or negative. For instance, suppose that Emoticons’ method can classify only 10% of the sentences in a dataset, corresponding to the actual percentage of sentences with emoticons. It means that the coverage of this method in this specific dataset is 10%. Note that, the coverage is quite an important metric for a more complete evaluation in the 2-class experiments. Even though Emoticons presents high accuracy for the classified phrases, it was not able to make a prediction for 90% of the sentences. More formally, coverage is calculated as the number of total sentences minus the number of undefined sentences, all of this divided by the total of sentences, where the number of undefined sentences includes neutral outputs for 3-class methods.

$$\mathit{Coverage} = \frac{\#\ \mathit{Sentences} - \#\ \mathit{Undefined}}{\#\ \mathit{Sentences}}. $$

Comparison metrics

Considering the 3-class comparison experiments, we used the traditional Precision, Recall, and F1 measures for the automated classification.

Each letter in Table 4 represents the number of instances which are actually in class X and predicted as class Y, where $X; Y \in \{\mbox{positive}; \mbox{neutral}; \mbox{negative}\}$. The recall (R) of a class X is the ratio of the number of elements correctly classified as X to the number of known elements in class X. Precision (P) of a class X is the ratio of the number of elements classified correctly as X to the total predicted as the class X. For example, the precision of the negative class is computed as: $P(\mathit{neg}) = i/(c+f+i)$; its recall, as: $R(\mathit{neg}) = i/(g+h+i)$; and the F1 measure is the harmonic mean between both precision and recall. In this case, $F1(\mathit{neg})=\frac{2P(\mathit{neg})\cdot R(\mathit{neg})}{P(\mathit{neg})+R(\mathit{neg})}$.

Table 4 Confusion matrix for experiments with three classes

We also compute the overall accuracy as: $A = \frac {a+e+i}{a+b+c+d+e+f+g+h+i}$. It considers equally important the correct classification of each sentence, independently of the class, and basically measures the capability of the method to predict the correct output. A variation of F1, namely, Macro-F1, is normally reported to evaluate classification effectiveness on skewed datasets. Macro-F1 values are computed by first calculating F1 values for each class in isolation, as exemplified above for negative, and then averaging over all classes. Macro-F1 considers equally important the effectiveness in each class, independently of the relative size of the class. Thus, accuracy and Macro-F1 provide complementary assessments of the classification effectiveness. Macro-F1 is especially important when the class distribution is very skewed, to verify the capability of the method to perform well in the smaller classes.

The described metrics can be easily computed for the 2-class experiments by just removing neutral columns and rows as in Table 5.

Table 5 Confusion matrix for experiments with two classes

In this case, the precision of positive class is computed as: $P(\mathit{pos}) = a/(a+c)$; its recall as: $R(\mathit{pos}) = a/(a+b)$; while its F1 is $F1(\mathit{pos})=\frac{2P(\mathit{pos})\cdot R(\mathit{pos})}{P(\mathit{pos})+R(\mathit{pos})}$.

As we have a large number of combinations among the base methods, metrics and datasets, a global analysis of the performance of all these combinations is not an easy task. We propose a simple but informative measure to assess the overall performance ranking. The Mean Ranking is basically the sum of ranks obtained by a method in each dataset divided by the total number of datasets, as below:

$$\mathit{MR} = \frac{\sum_{j=1}^{\mathit{nd}}\mathit{ri}}{\mathit{nd}} $$

where nd is the number of datasets and ri is the rank of the method for dataset i. It is important to notice that the rank was calculated based on Macro-F1.

The last evaluation metric we exploit is the Friedman’s Test [71]. It allows one to verify whether, in a specific experiment, the observed values are globally similar. We used this test to tell if the methods present similar performance across different datasets. More specifically, suppose that k expert raters evaluated n item - the question that arises is: are rates provided by judges consistent with each other or do they follow completely different patterns? The application in our context is very similar: the datasets are the judges and the Macro-F1 achieved by a method is the rating from the judges.

The Friedman’s Test is applied to rankings. Then, to proceed with this statistical test, we sort the methods in decreasing order of Macro-F1 for each dataset. More formally, the Friedman’s rank test in our experiment is defined as:

$$F_{R} = \Biggl(\frac{12}{rc(c+1)} {\sum_{j=1}^{c}R^{2}_{j}}\Biggr) -3r(c+1), $$

where

$$\begin{aligned}& R^{2}_{j} = \mbox{square of the sum of rank positions of method }j\quad (j = 1,2,\ldots, c), \\& r = \mbox{number of datasets}, \\& c = \mbox{number of methods}. \end{aligned}$$

As the number of datasets increases, the statistical test can be approximated by using the chi-square distribution with $c-1$ degrees of freedom [72]. Then, if the $F_{R}$ computed value is larger than the critical value for the chi-square distribution the null hypothesis is rejected. This null hypothesis states that ranks obtained per dataset are globally similar. Accordingly, rejecting the null hypothesis means that there are significant differences in the ranks across datasets. It is important to note that, in general, the critical value is obtained with significance level $\alpha= 0.05$. Synthesizing, the null hypothesis should be rejected if $F_{R} > X^{2}_{\alpha}$, where $X^{2}_{\alpha}$ is the critical value verified in the chi-square distribution table with $c-1$ degrees of freedom and α equals 0.05.

Comparing prediction performance

We start the analysis of our experiments by comparing the results of all previously discussed metrics for all datasets. Table 6 and Table 7 present accuracy, precision, and Macro-F1 for all methods considering four datasets for the 2-class and 3-class experiments, respectively. For simplicity, we choose to discuss results only for these datasets as they come from different sources and help us to illustrate the main findings from our analysis. Results for all the other datasets are presented in Additional file 1. There are many interesting observations we can make from these results, summarized next.

Table 6 2-classes experiments results with 4 datasets

Table 7 3-classes experiments results with 4 datasets

Methods prediction performance varies considerably from one dataset to another: First, we note the same social media text can be interpreted very differently depending on the choice of a sentiment method. Overall, we note that all the methods yielded with large variations across the different datasets. By analyzing Table 6 we can note that VADER works well for Tweets_RND_II, appearing in the first place, but it presents poor performance in Tweets_STF, Comments_BBC, and Comments_DIGG, achieving the eleventh, thirteenth and tenth place respectively. Although the first two datasets contain tweets, they belong to different contexts, which affects the performance of some methods like VADER. Another important aspect to be analyzed in this table is the coverage. Although SentiStrength has presented good Macro-F1 values, its coverage is usually low as this method tends to classify a high number of instances as neutral. Note that some datasets provided by the SentiStrength’s authors, as shown in Table 3, specially the Twitter datasets, have more neutral sentences than positive and negative ones. Another expected result is the good Macro-F1 values obtained by Emoticons, specially in the Twitter datasets. It is important to highlight that, in spite of achieving high accuracy and Macro-F1, the coverage of many methods, such as PANAS, VADER, and SentiStrength, is low (e.g. below 30%) as they only infer the polarity of part of the input sentences. Thus, the choice of a sentiment analysis is highly dependent on the data and application, suggesting that researchers and practitioners need to take into account this tradeoff between prediction performance and coverage.

The same high variability regarding the methods’s prediction performance can be noted for the 3-class experiments, as presented in Table 7. Umigon, the best method in five Twitter datasets, felt to the eighteenth place in the Comments_NYT dataset. We can also note the lower Macro-F1 values for some methods like Emoticons are due to the high number of sentences without emoticons in the datasets. Methods like Emoticons DS and PANAS tend do classify only a small part of instances as neutral and also presented a poor performance in the 3-class experiments. Methods like SenticNet and LIWC were not originally developed for detecting neutral sentences and also achieved low values of Macro-F1. However, they also do not appear among the best methods in the 2-class experiments, which is the task they were originally designed for. This observation about LIWC is not valid for the newest version, as LIWC15 appears among the top five methods for 2-class and 3-class experiments (see Table 8).

Table 8 Mean rank table for all datasets

Finally, Table 9 presents the Friedman’s test results showing that there are significant differences in the mean rankings observed for the methods across all datasets. It statistically indicates that in terms of accuracy and Macro-F1 there is no single method that always achieves a consistent rank position for different datasets, which is something similar to the well-known ‘no-free lunch theorem’ [16]. So, overall, before using a sentiment analysis method in a novel dataset, it is crucial to test different methods in a sample of data before simply choose one that is acceptable by the research community.

Table 9 Friedman’s test results

This last results suggests that, even with the good insights provided by this work about which methods perform better in each context, a preliminary investigation needs to be performed when sentiment analysis is used in a new dataset in order to guarantee a reasonable prediction performance. In the case in which prior tests are not feasible, this benchmark presents valuable information for researchers and companies that are planning to develop research and solutions on sentiment analysis.

Existing methods let space for improvements: We can note that the performance of the evaluated methods are ok, but there is a lot of space for improvements. For example, if we look at the Macro-F1 values only for the best method on each dataset (see Table 6 and Table 7), we can note that the overall prediction performance of the methods is still low - i.e. Macro-F1 values are around 0.9 only for methods with low coverage in the 2-class experiments and only 0.6 for the 3-class experiment. Considering that we are looking at the performance of the best methods out of 24 unsupervised tools, these numbers suggest that current sentence-level sentiment analysis methods still let a lot of space for improvements. Additionally, we also noted that the best method for each dataset varies considerably from one dataset to another. This might indicate that each method complements the others in different ways.

Most methods are better to classify positive than negative or neutral sentences: Figure 2 presents the average F1 score for the 3-class experiments. It is easier to notice that twelve out of twenty-four methods are more accurate while classifying positive than negative or neutral messages, suggesting that some methods may be more biased towards positivity. Neutral messages showed to be even harder to detect by most methods.

Figure 2

Average F 1 score for each class. This figure presents the average F1 of positive and negative class and as we can see, methods use to achieve better prediction performance on positive messages.

Interestingly, recent efforts show that human language have a universal positivity bias ([73] and [74]). Naturally, part of the bias is observed in sentiment prediction, an intrinsic property of some methods due to the way they are designed. For instance, [32] developed a lexicon in which positive and negative values are associated to words, hashtags, and any sort of tokens according to the frequency with which these tokens appear in tweets containing positive and negative emoticons. This method showed to be biased towards positivity due to the larger amount of positivity in the data they used to build the lexicon. The overall poor performance of this specific method is credited to its lack of treatment of neutral messages and the focus on Twitter messages.

Some methods are consistently among the best ones: Table 8 presents the mean rank value, detailed before, for 2-class and 3-class experiments. The elements are sorted by the overall mean rank each method achieved based on Macro-F1 for all datasets. The top nine methods based on Macro-F1 for the 2-class experiments are: SentiStrength, Sentiment140, Semantria, OpinionLexicon, LIWC15, SO-CAL, AFINN and VADER and Umigon. With the exception of SentiStrength, replaced by Pattern.en, the other eight methods produce the best results across several datasets for both, 2-class and 3-class tasks. These methods would be preferable in situations in which any sort of preliminary evaluation is not possible to be done. The mean rank for 2-class experiments is accompanied by the coverage metric, which is very important to avoid misinterpretation of the results. Observe that SentiStrength and Sentiment140 exhibited the best mean ranks for these experiments, however both present very low coverage, around 30% and 40%, a very poor result compared with Semantria and OpinionLexicon that achieved a worse mean rank (4.61 and 6.62 respectively) but an expressive better coverage, above 60%. Note also that SentiStrength and Sentiment140 present poor results in the 3-class experiments which can be explained by their bias to the neutral class as mentioned before.

Another interesting finding is the fact that VADER, the best method in the 3-class experiments, did not achieve the first position for none of the datasets. It reaches the second place five times, the third place twice, the seventh three times, and the fourth, sixth and fifth just once. It was a special case of consistency across all datasets. Tables 10 and 11 present the best method for each dataset in the 2-class and 3-class experiments, respectively.

Table 10 Best method for each dataset - 2-class experiments

Table 11 Best method for each dataset - 3-class experiments

Methods are often better in the datasets they were originally evaluated: We also note those methods perform better in datasets in which they were originally validated, which is somewhat expected due to fine tuning procedures. We could do this comparison only for SentiStrength and VADER, which kindly allowed the entire reproducibility of their work, sharing both methods and datasets. To understand this difference, we calculated the mean rank for these methods without their ‘original’ datasets and put the results in parenthesis. Note that, in some cases the rank order changes towards a lower value but it does not imply in major changes. We also note those methods often perform better in datasets in which they were originally validated, which is somewhat expected due to fine tuning procedures. We could do this comparison only for SentiStrength and VADER, which kindly allowed the entire reproducibility of their work, sharing both methods and datasets. To understand this difference, we calculated the mean rank for these methods without their ‘original’ datasets and put the results in parenthesis. Note that, in some cases the rank order slightly changes but it does not imply in major changes. Overall, these observations suggest that initiatives like SemEval are key for the development of the area, as they allow methods to compete in a contest for a specific dataset. More important, it highlight that a standard sentiment analysis benchmark is needed and it needs to be constantly updated. We also emphasize that is possible that other methods, such as paid softwares, make use of some of the datasets used in this benchmark to improve their performance as most of gold standard used in this work is available in the Web or under request to authors.

Some methods showed to be better for specific contexts: In order to better understand the prediction performance of methods in types of data, we divided all datasets in three specific contexts - Social Networks, Comments, and Reviews - and calculated mean rank of the methods for each of them. Table 12 presents the contexts and the respective datasets.

Table 12 Contexts’ groups

Tables 13, 14 and 15 present the mean rank for each context separately. In the context of Social Networks the best method for 3-class experiments was Umigon, followed by LIWC15 and VADER. In the case of 2-class the winner was SentiStrength with a coverage around 30% and the third and sixth place were Emoticons and PANAS-t with about 18% and 6% of coverage, respectively. This highlights the importance to analyze the 2-class results together with the coverage. Overall, when there is an emoticon on the text or a word from the psychometric scale PANAS, these methods are able to tell the polarity of the sentences, but they are not able to identify the polarity of the input text for the large majority of the input text. Recent efforts suggest these properties are useful for combination of methods [20]. Sentiment140, LIWC15, Semantria, OpinionLexicon and Umigon showed to be the best alternatives for detecting only positive and negative polarities in social network data due to the high coverage and prediction performance. It is important to highlight that LIWC 2007 appears on the 16th and 21th position for the 3-class and 2-class mean rank results for the social network datasets and it is a very popular method in this community. On the other side, the newest version of LIWC (2015) presented a considerable evolution obtaining the second and the fourth place in the same datasets.

Table 13 Mean rank table for datasets of social networks

Table 14 Mean rank table for datasets of comments

Table 15 Mean rank table for datasets of reviews

Similar analyses can be performed for the contexts Comments and Reviews. SentiStrength, VADER, Semantria, AFINN, and Opinion Lexicon showed to be the best alternatives for 2-class and 3-class experiments on datasets of comments whereas Sentiment140, SenticNet, Semantria and SO-CAL showed to be the best for the 2-class experiments for the datasets containing short reviews. Note that for the last one, the 3-class experiments have no results since datasets containing reviews have no neutral sentences nor a representative number of sentences without subjectivity.

We also calculated the Friedman’s value for each of these specific contexts. Even after grouping the datasets, we still observe that there are significant differences in the observed ranks across the datasets. Although the values obtained for each context were quite smaller than Friedman’ global value, they are still above the critical value. Table 16 presents the results of Friedman’s test for the individual contexts in both experiments, 2 and 3-class. Recall that for the 3-class experiments, datasets with no neutral sentences or with an unrepresentative number of neutral sentences were not considered. For this reason, Friedman’s results for 3-class experiments in the Reviews context presents no values.

Table 16 Friedman’s test results per contexts

Concluding remarks

Recent efforts to analyze the moods embedded in Web 2.0 content have adopted various sentiment analysis methods, which were originally developed in linguistics and psychology. Several of these methods became widely used in their knowledge fields and have now been applied as tools to quantify moods in the context of unstructured short messages in online social networks. In this article, we present a thorough comparison of twenty-four popular sentence-level sentiment analysis methods using gold standard datasets that span different types of data sources. Our effort quantifies the prediction performance of the twenty-four popular sentiment analysis methods across eighteen datasets for two tasks: differentiating two classes (positive and negative) and three classes (positive, negative, and neutral).

Among many findings, we highlight that although our results identified a few methods able to appear among the best ones for different datasets, we noted that the overall prediction performance still left a lot of space for improvements. More important, we show that the prediction performance of methods vary largely across datasets. For example, LIWC 2007, is among the most popular sentiment methods in the social network context and obtained a bad rank position in comparison with other datasets. This suggests that sentiment analysis methods cannot be used as ‘off-the-shelf’ methods, specially for novel datasets. We show that the same social media text can be interpreted very differently depending on the choice of a sentiment method, suggesting that it is important that researchers and companies perform experiments with different methods before applying a method.

As a final contribution we open the datasets and codes used in this paper for the research community. We also incorporated them in a Web service from our research team called iFeel [21] that allow users to easily compare the results of various sentiment analysis methods. We hope our effort can not only help researchers and practitioners to compare a wide range of sentiment analysis techniques, but also help fostering new relevant research in this area with a rigorous scientific approach.