Lexicon-Based Sentiment Analysis in Behavioral Research

Abstract

A complete science of human behavior requires a comprehensive account of the verbal behavior those humans exhibit. Existing behavioral theories of such verbal behavior have produced compelling insight into language’s underlying function, but the expansive program of research those theories deserve has unfortunately been slow to develop. We argue that the status quo’s manually implemented and study-specific coding systems are too resource intensive to be worthwhile for most behavior analysts. These high input costs in turn discourage research on verbal behavior overall. We propose lexicon-based sentiment analysis as a more modern and efficient approach to the study of human verbal products, especially naturally occurring ones (e.g., psychotherapy transcripts, social media posts). In the present discussion, we introduce the reader to principles of sentiment analysis, highlighting its usefulness as a behavior analytic tool for the study of verbal behavior. We conclude with an outline of approaches for handling some of the more complex forms of speech, like negation, sarcasm, and speculation. The appendix also provides a worked example of how sentiment analysis could be applied to existing questions in behavior analysis, complete with code that readers can incorporate into their own work.

Appendix 1

For this worked example, we assume only a basic familiarity with the R programming language and the tidyverse suite of packages within it (Wickham & RStudio, 2017). We have intentionally written the code for maximum readability (sometimes at the cost of brevity), so even readers without this background should still be able to read along. Readers interested in brushing up on R and the tidyverse are encouraged to work through any of the excellent and freely available tutorials available online (Wickham & Grolemund, 2017). Readers interested in a more copy/paste-able format of this appendix can find the annotated raw code in our supplemental file here: https://osf.io/sp6mx/?view_only=cdcd6ff0df71417590672e34386e6beb .

A basic behaviorally informed sentiment analysis involves several steps, which we now demonstrate in order.

• 1. Select a previously validated lexicon or create a new one

• 2. Acquire raw verbal data (documents)

• 3. Tokenize your documents and wrangle them into a “tidy” format.

• 4. Remove stop words / stop tokens

• 5. Use the lexicon to score each token

• 6. Compute summary statistics (e.g., proportion of positive words)

• 7. Analyze with standard behavior analytic methods (e.g., regression, visual analysis)

We will implement these steps to perform an analysis reminiscent of McDowell and Caron’s (2010) work connecting rule-break talk to received praise, in accordance with the GML. Except in this case, we will be examining whether two U.S. politicians—vice presidents Mike Pence and Kamala Harris—post tweets in accordance with the GML.

• Step 1: Acquire an Appropriate Lexicon

Technically, steps 1 and 2 can be conducted out of order. We begin with the lexicon in this discussion simply because we needed to begin somewhere. When acquiring a lexicon, a researcher has two options. They can either utilize a prevalidated lexicon from previous research or create a new one. We encourage anyone new to sentiment analysis to use a pre-validated lexicon, which is both safer and faster. The lexicon you choose can come from a range of sources (Khoo & Johnkhan, 2018). The easiest to use will be those already available in an R package like tidytext (Silge & Robinson, 2022), which includes a helper function to download the lexicons displayed in Table 1. For most of the sentiment analysis a researcher would want to conduct in behavior analysis, these will be sufficient because they include several emotional categories that will get a researcher through their first few studies. By the time a researcher completes their first few studies with these well-known lexicons, they should already have a sense of the kind of things they would want in their next lexicon.

One other lexicon behavioral researchers should be aware of right away, however, is the Linguistic Inquiry and Word Count (LIWC; Boyd et al., 2022; Tausczik & Pennebaker, 2010). This lexicon was created for psychological research and has been evaluated and revised several times. It is especially valuable for its comprehensiveness, including many more word categories than is common in other lexicons (e.g., words related to cognitive processes, social processes, hierarchy). For this reason, LIWC has already been used extensively to study the connection between subjects’ linguistic content and a range of psychological topics and in a number of languages (Brandt & Herzberg, 2020; Cutler et al., 2021; Lumontod, 2020). Researchers who find themselves saying “I feel like the basic lexicons aren’t enough, I wish I had a lexicon that covered my niche topic” should immediately check whether LIWC covers their particular case.

In this case, we will use the National Research Council Word-Emotion Association Lexicon (NRC) lexicon, which was built up from a range of sources, including the preexisting WordNet affective lexicon and 8,000 terms from the General Inquirer (Mohammad & Turney, 2010, 2013). Previous work has used it specifically to study Twitter tweets, including the identification of suicide-related posts, predicting Affordable Care Act enrollment, and to evaluate global pandemic reactions (Dubey, 2020; Sarsam et al., 2021; Wong et al., 2015). This diversity of topics, including one using the NRC to predict overt behavior (e.g., insurance enrollment), all increase the plausibility this lexicon tracks behaviorally meaningful verbal content. It has the added advantage of being included in the tidytext R package, so we can load it directly like this.

With the NRC lexicon loaded into memory, we can narrow down the kind of sentiment we want to study in this analysis. Here, we retain only words that are related to the dimension of trust / mistrust. We expect this dimension is especially relevant to the occupational success of our two subjects, so it is likely to be a function of some salient reinforcer—like the number of “likes” from Twitter followers. Below, we also provide a random sample of the remaining trust-related words.

• Step 2: Acquire Raw Verbal Data

Most researchers will already be aware of verbal data sources relevant to their research (e.g., intervention session transcripts), so we will avoid repeating some of the most common sources here. Instead, we point out that there are likely a few data sources that researchers have not previously considered. For example, some video conference platforms (e.g., Zoom) have built-in support for automatic transcription of recorded meetings. And readers will be pleased to learn that these transcriptions are both accurate and arrive in a standardized format. In a yet unpublished study, our own research group has already taken advantage of these resources, finding that the latency, duration, and content of speech is associated with intervention satisfaction, recall, and self-reported adoption at 1-month follow-up (manuscript in preparation).

Another example is Project Gutenberg, which provides digital versions of public domain literature. Although this is outside the scope of most modern behavior studies, we mention it to interested readers who might want to follow in Skinner’s early footsteps, which actually began with an analysis of alliteration in the works of William Shakespeare (Skinner, 1939).

The last approach—and the one we use for our worked example—is to use a REST API.³ Usually shortened to just “API,” this is a system for communicating with a web server via code, rather than a point-and-click interface. This process requires some initial effort, but is often simpler than it sounds and is a quick way to access a substantial amount of data. One of the most well-known APIs in research is the Twitter API, which allows people outside of Twitter to access a substantial amount of granular data on the activity of Twitter’s users. To give readers a sense of the scope, the first author was able to gather 64 million tweets from 17 million different for a recent study—all for free (Cero & Witte, 2020).

Although a comprehensive introduction to APIs is beyond the scope of the current discussion, Twitter’s own tutorial is a great introduction and will remain up-to-date whenever they implement changes (Twitter, 2022). In practice, the process involves filling out a brief application to Twitter, who will then provide a set of tokens that function like a username and password. Researchers can then pass these tokens and a search query to an R package (“academictwitteR”) that knows how to handle the Twitter API (Barrie et al., 2022), doing most of the work under the hood.

For example, to save the roughly 30,000 posts Pence and Harris have produced from 2016 through 2021, a researcher simply provides their bearer token from Twitter, a formatted search query for tweets from Pence’s and Harris’s accounts, the dates to search through, and a data path (folder) in which to save the results.

The get_all_tweets() function unfortunately saves tweet and user data in separate places, so we’ll need to load and merge them ourselves. For the loading, we can use the bind_tweets() function from the academictwitteR package to bring the tweets and user data into memory. Along the way, we extract (unnest(public_metrics)) some information about each tweet, including the like_count—our putative reinforcer in this mini matching study. We’ll also filter out retweets (which always start with an “RT”), retaining only the tweets generated by Pence and Harris themselves. By coincidence, this leaves exactly 18,000 tweets in total.

We can then use the left_join() function of the tidyverse package to add user information to each tweet.

• Step 3: Tokenize Your Documents and Wrangle Them into a “Tidy” Format

In its current form, our full_df dataframe stores each tweet as a line of text. Although it is easy to read, this makes it hard for our code to access each individual word and compare it to the entries in our NRC lexicon. To get around this, we need to tokenize all of our tweets, so that each row of our dataframe will represent a single word. This is called the tidy format in R. Fortunately, the tidytext package makes this process easy, providing us with the unnest_tokens() function that handles everything automatically. We simply tell it we want a new column named word, which is made up of the individual words from the old text column. Careful readers will thus notice the first several entries in the word column of the tokenized_df now represent the first several words of the first text in the text column of the full_df.

• Step 4: Remove Stopwords from the Dataset

Stop words or stop tokens (in the case of multiword n-grams) are those that occur so often that they are uninformative to the meaning of a text (e.g., “of,” “and,” “the”). Fortunately, just by loading the tidytext package, we have already loaded a precompiled list of stopwords called stop_words in the background. Thus, the quickest way to get these stopwords out of our tokenized_df dataframe is simply to anti_join() them. In an anti-join or anti-merge, only the records from the first dataset (tokenized_df) that DO NOT match anything in the second dataset (stop_words) are retained.

While we are removing unhelpful tokens, we’ll also filter out “t.co” and “https.” Visual inspection of our tokenized_df revealed these are both fragments of web links Pence and Harris posted in some of their tweets, which were accidentally included during the tokenization process (unnest_tokens() thought they were words worth retaining). Because our lexicon does not cover them, we can explicitly filter them out here too.

• Step 5: Use the Lexicon to Score Each Token

We expect this likely sounds as though it will be the most intensive part of sentiment analysis. After all, we estimated for McDowell and Caron’s group to hand-score a much smaller sample of text likely took over 140 person hr. Scoring all the words from 18,000 tweets must be quite laborious, right? In fact, all of our words are effectively scored with just two lines of code, which join the words from our observed dataset to the values in our NRC trust lexicon.

A minor snag is that our nrc_trust lexicon only includes words that are trust-related. It produces missing values for everything on which the lexicon is silent (i.e., nontrust words). To simplify our upcoming analysis, we’ll compute a new true/false column called trust_word, which will indicate whether a given word in our dataset is a trust word, based on the values in the adjacent sentiment column.

As a quick sanity check, we’ll now peak at a random sample of trust and nontrust words from both subjects.

Here, we get a quick sense of the kinds of trust- and nontrust-related words each subject might be using. These randomly selected words are overall somewhat banal, but they are consistent with what we would expect. Words like “system” imply something that needs to be relied on, so they exist somewhere along a dimension of trustworthiness. Words like “fair” are morally salient, but do not imply something related to reliance and thus not scored as trust-related. The same is true of words like persecution, which is unfair to be sure, but does not indicate a dimension of trust.

• Step 6: Compute Summary Statistics

For our upcoming matching analysis, we’ll want to know whether each subject produces tweets with trust-related words in proportion to the likes those tweets received. To get this far, we needed to break up (“tokenize”) whole tweets into individual words, so that we could score those words with a lexicon. Now that they have been scored in the trust_word column, we need to start going in reverse. We need to recombine words into tweets and summarize each tweet by whether any of its words is a trust word.

Once individual tweets have been scored in the is_trust_tweet column, we arrange tweets by their ID numbers (which are strictly in order of date produced) and assign them to blocks of 50 people. This final line, block = floor(row_number() - 1) / 50, is just a shorthand way of saying “take the row number of each tweet, divide by 50, and round down to the nearest integer, and treat that as its block number.”

With blocks assigned, we simply compute the familiar matching statistics. One trick to note is that R will treat TRUE and FALSE values as 1 and 0 when they are forced into mathematical computations. Thus, sum(like_count*is_trust_tweet) can be read “the sum of likes produced when is_trust_tweet is true.” We also proactively filter() to retain only cases where the log_b and log_r are still finite, which in this case is all of them because there were no blocks with 0 trust/non-trust tweets or 0 likes for either of those cases.

• Step 7: Analyze with Standard Behavior-Analytic Methods (e.g., Visual

• Analysis, Regression)

At this point, all that is left to do is perform a matching analysis. Because we have two subjects who will need separate regressions, we use the group-nest-map-tidy-unnest approach. It is probably overkill for only two regressions, but in the common case that a matching analysis includes a half-dozen or more subjects to regress, this strategy is both faster and safer than copy-pasting code.

Unnesting the regression coefficients reveals some interesting results. Both subjects are highly sensitive to the likes associated with trust-related words.

What is even more interesting are these substantial bias terms, which suggest that even if trust-related tweets produced likes in equal proportion to nontrust-related tweets, both subjects would still produce trust-related tweets in substantial excess. In particular, even if the likes received for each tweet type were perfectly balanced, Harris would be expected to produce 2^0.278 = 1.21x more trust-related tweets than nontrust related ones—and Pence would produce 1.34x more.

Examining the efficacy of the matching model to explain such behavior, note that the R-squared values for both subjects are significant, but much higher for Pence. Combined with a sensitivity very near 1.0 for this subject, such a finding suggests this learning model is a compelling (if as yet, nonexperimental) account of his verbal behavior over many years.

We can see this by visually examining the log behavior and log reinforcement rates for each subject for each block, observing that Pence’s blocks conform much more closely to the theoretically perfect matching (the dashed line). (Fig. 1).

Fig. 1

Matching analyses. Note. R output for the log2 behavior and reinforcement rates by block for both subjects. Solid lines represent empirically observed regression slopes and gray ribbons represent confidence bounds. Dashed lines represent theoretically perfect matching