Psychological Text Analysis in the Digital Humanities

Abstract

In the digital humanities, it has been particularly difficult to establish the psychological properties of a person or group of people in an objective, reliable manner. Traditionally, the attempt to understand an author’s psychological makeup has been primarily (if not exclusively) accomplished through subjective interpretation, qualitative analysis, and speculation. In the world of empirical psychological research, however, the past two decades have witnessed an explosion of computerized language analysis techniques that objectively measure psychological features of the individual. Indeed, by using modern text analysis methods, it is now possible to quickly and accurately extract information about people—personalities, individual differences, social processes, and even their mental health—all through the words that people write and speak. This chapter serves as a primer for researchers interested in learning about how language can provide powerful insights into the minds of others via well-established and easy-to-use psychometric methods. First, this chapter provides a general background on language analysis in the field of psychology, followed by an introduction to modern methods and developments within the field of psychological text analysis. Finally, a solid foundation to psychological text analysis is provided in the form of an overview of research spanning hundreds of studies from labs all over the world.

Appendix

When preparing and processing texts using psychological text analysis, there are some widely accepted (but often unspoken) guidelines that are quite commonplace. These guidelines serve to ensure accurate insights both while processing texts and in subsequent statistical analyses. This appendix can be thought of as a basic “need to know” reference and primer for technical considerations when performing psychological text analysis. Feel free to treat this appendix as a “cheat sheet” for the methods covered in this chapter—one that should help to give you a head start in the world of research with language.

Preparing Texts for Analysis

One of the most common questions that people ask during their first romp into the world of psychological text analysis is this: “How should I prepare my texts?” Ultimately, there is no answer to this question that will apply in all cases, as guidelines will vary as a function of text source, research questions, and goals. However, there are some basic considerations that apply to virtually all cases. As a general rule when it comes to the psychological analysis of text, “good enough” really is “good enough” for most purposes. One could literally spend years preparing a collection of text files so that they are 100% perfect for analysis, however, the conceptual (and, more importantly, statistical) gains from doing so are often nil.

Spelling and Spelling Variations

It is tempting to worry about correcting texts so that all words that could potentially be captured by a dictionary are successfully recognized. Note, however, that word distributions follow what is known as a Zipf distribution (see Piantadosi 2014), wherein a relatively small number of words constitute the majority of words actually seen in texts, verbalizations, and so on. Translated into practical terms, what this means is that unless a very common word is misspelled with high frequency, it is unlikely to have a measurable impact on LIWC-based and MEM-based measures of psychological processes. For example, if a single text has the misspelling “teh” two times, yet contains 750 uses of other articles in the whole text, the measured percentage of articles in a text will differ from the actual occurrence of articles by such a small amount as to be negligible.

Texts with several high-frequency misspellings, however, may benefit from correction. Multiple software programs exist that allow users to automatically batch-correct text files to avoid the tedious job of manual spelling correction (e.g., GNU Aspell; http://aspell.net). While most of these applications are useful only for the technically savvy, other options exist that allow users to find specific misspellings and replace them with corrections, such as Find and Replace (FAR; http://findandreplace.sourceforge.net). Relatedly, regional spelling variants may benefit from standardization, depending on the nature of one’s research question. For example, given that the MEM looks for co-occurrences of words, we might expect the words “bloody,” “neighbor,” and “color” to co-occur more often than “bloody,” “neighbor,” and “color.” Unless we are interested in identifying culture-specific word co-occurrences, we would want to standardize regional variants to have parallel spellings across all texts, ensuring more accurate psychological insights.

Finally, certain special circumstances arise when analyzing transcribed verbal exchanges, particularly when using software such as LIWC. Several categories of “utterances” (such as nonfluency words like “uh” and “um,” and filler words like “like” and “y’know”) are psychologically meaningful (Laserna et al. 2014), but are often transcribed idiosyncratically according to certain traditions. The word “like” is particularly problematic given its various meanings as a result of homography (e.g., expressing evaluation—“I like him”—or filling spaces—“I, like, love sandwiches”). Primarily in the case of verbal transcriptions, many utterances must be converted to software-specific tokens that are recognized explicitly as filler words to improve text analysis accuracy (LIWC, for example, uses “rrlike” for filler word recognition).

Working with the MEM and MEM Results

When performing any type of topic modeling procedure, including the MEM, several decisions must be made by the researcher performing the analysis. These typically include answering questions such as “What is the correct number of themes to extract?” and “How do I know what label to give each of these themes?” Topic modeling results are occasionally dismissed as purely subjective, however, this is seldom the case. Indeed, while some steps occasionally require some (arguably) arbitrary decisions, such decisions are typically made by relying on domain expertise. The author’s recommendation is that, when in doubt, the best course of action is to consult with experts familiar with the type of research being conducted.

How to Extract/Understand Themes Using MEM

While the heart of the MEM is a statistical procedure known as a Principal Components Analysis (PCA), most of the typical guidelines that are recommended for a PCA do not extend to the modeling of language patterns. Instead, it may be more useful to think of the MEM as co-opting a statistical procedure to reach an end-goal, rather than the PCA being the goal itself.

When extracting themes using a PCA, a researcher must specify a k parameter, with k being the number of themes for which a PCA must solve. What is the ideal k parameter? In other words, how many themes should be extracted? Is it 10? 15? 50? This is a contentious issue in virtually every field that uses some form of topic modeling. The best answer at this time comes in the form of another question: “What makes sense given your research question?”

When attempting to perform a psychological analysis of text, domain expertise on psychological constructs is immensely helpful. The primary recommendation for determining the optimal k parameter is to test multiple k’s, settling on the k parameter that appears to best represent the problem space. The ideal k parameter is also directly influenced by the amount of data that you are processing. If you have an extremely large number of wordy texts, you will be able to extract many, many more themes than if you are analyzing 100 posts made to Twitter.

Even with extremely large datasets, however, the smallest number of themes that can be coherently extracted is typically the most optimal, particularly in cases of psychological research. Whereas it is not uncommon to see sophisticated studies that extract hundreds (or even thousands) of topics from a corpus, 95% of these topics will end up being an uninterpretable and highly inter-correlated grab-bag of words that does not represent anything in particular. Extracting large numbers of topics may still be highly useful in predictive modeling, however, this practice is often problematic from a social sciences/digital humanities perspective and also leads to serious concerns about the replicability of one’s findings.

Finally, when it comes to labeling themes that have been extracted using MEM or other topic models, there are no hard and fast rules. Labels assigned to word clusters should primarily be treated as a shorthand for referring to word clusters and should not treated as objective operationalizations during research in most cases. For example, one researcher may see the words “happy,” “cry,” and “shout” cluster together in a MEM analysis and label this as a broad “emotion” theme. Another researcher may see the same word clusters and see a “joyous” theme, believing that the words “cry” and “shout” are being used in a positive manner in conjunction with happiness. Most of the time, the labels are fairly simply to agree upon (e.g., a word cluster comprised of “college,” “study,” “test,” and “class” is unlikely to be anything other than a broader “school” theme). However, when in doubt, one of the best ways to understand or interpret a questionable theme is to look for texts that score high for that theme, then read them closely to see how the theme-related words are used.

How to Score Texts for MEM Themes

We will typically want to score each text in our corpus for each theme that has been extracted. Once themes have been extracted using the MEM, there are two primary ways to make this information useful for subsequent statistical modeling. In the first method, “factor scores” can be calculated using a statistical approach, such as regression scoring. In this approach, each word is weighted for its “representativeness” of a given theme—these weights correspond to the “factor loadings” of each word to each theme. For example, if a theme is extracted pertaining to animals, the word “cat” may have a factor loading of 0.5, the word “bird” has a loading of 0.25, and so on. These scores can be used in a multiple linear regression model:

y = (ax) + (bz) + … etc., or

Animal Theme = (0.5*cat) + (0.25*bird)… etc.

This procedure is used to score a text for the “animal” theme, and so on for all other themes extracted during the MEM. Some statistical software, such as IBM’s SPSS, have options that allow users to perform this scoring method automatically.

If using the regression approach to scoring texts for MEM themes, it is extremely important to note that the MEM typically requires that one performs their PCA with something called a VARIMAX rotation. Without going too far into the details, an orthogonal axis rotation such as VARIMAX ensures that all themes are mathematically 100% independent of each other. In practical terms, this means that regression scores for themes will be perfectly uncorrelated, which is often not an accurate reflection of psychological processes. This method is occasionally used in the literature but is not recommended for most purposes unless you have a well-articulated reason for doing so.

The second method for scoring MEM themes in texts is conceptually much simpler and strongly recommended. Essentially, this alternative method uses the MEM as a means of theme discovery or, in other words, simply determining what common themes exist in a corpus, as well as which words are indicative of which themes. Following this, a return to the word counting approach used by LIWC is used to score texts for each theme. For example, imagine that your use of the MEM uncovers two broad themes in your texts: a color theme (including the words “red,” “blue,” “green,” and “brown”) and a clothing theme (including the words “dress,” “shirt,” “shorts,” and “jeans”). The next step is to create a custom dictionary that places the “color” and “clothing” words into separate categories. You would then rescan your texts with software like LIWC to calculate the percentage of words pertaining to color and clothing, respectively. Unlike with the regression method of scoring texts (which would force a perfect zero correlation between the two themes), one would likely find that use of color words and clothing words shows modest bivariate correlation, which makes sense from both and intuitive and psychological perspective.