Moved by words: Affective ratings for a set of 2,266 Spanish words in five discrete emotion categories
The two main theoretical accounts of the human affective space are the dimensional perspective and the discrete-emotion approach. In recent years, several affective norms have been developed from a dimensional perspective, including ratings for valence and arousal. In contrast, the number of published datasets relying on the discrete-emotion approach is much lower. There is a need to fill this gap, considering that discrete emotions have an effect on word processing above and beyond those of valence and arousal. In the present study, we present ratings from 1,380 participants for a set of 2,266 Spanish words in five discrete emotion categories: happiness, anger, fear, disgust, and sadness. This will be the largest dataset published to date containing ratings for discrete emotions. We also present, for the first time, a fine-grained analysis of the distribution of words into the five emotion categories. This analysis reveals that happiness words are the most consistently related to a single, discrete emotion category. In contrast, there is a tendency for many negative words to belong to more than one discrete emotion. The only exception is disgust words, which overlap least with the other negative emotions. Normative valence and arousal data already exist for all of the words included in this corpus. Thus, the present database will allow researchers to design studies to contrast the predictions of the two most influential theoretical perspectives in this field. These studies will undoubtedly contribute to a deeper understanding of the effects of emotion on word processing.
KeywordsDiscrete emotion categories Affective norms Emotional effects on word processing
Interest in the study of emotional effects on word processing has grown exponentially during the last decade. A significant number of studies have revealed that these effects are pervasive. Indeed, modulations of the processing of words by their emotional content have been documented not only in tasks involving affective decisions (Estes & Verges, 2008; González-Villar, Triñanes, Zurrón, & Carrillo-de-la-Peña, 2014; Herbert, Kissler, Junghöfer, Peyk, & Rockstroh, 2006), but also in experimental situations in which the emotional content was irrelevant to the task (González-Villar et al., 2014; Hinojosa, Méndez-Bértolo, & Pozo, 2010; Kousta, Vinson, & Vigliocco 2009; Schacht & Sommer, 2009; Scott, O’Donnell, & Sereno 2014).
Most of the research above has been conducted from a dimensional theoretical perspective. Dimensional theories propose that humans’ emotional experience can be described in terms of continuous variations of a few dimensions, the most significant being valence and arousal. Valence indicates the hedonic value of a specific emotion, ranging from unpleasant to pleasant, whereas arousal indicates the degree of activation experienced, ranging from calming to exciting (Bradley & Lang, 2000; Russell, 2003). Research into emotion and language has taken advantage of the fact that words can be characterized according to these dimensions (Bradley & Lang, 1999). Hence, studies in the field have focused on testing the effects of the valence and arousal of words with different experimental paradigms and tasks, such as the detection of meaningful words in streams of stimuli (Hinojosa et al., 2010), the lexical decision task (Citron, Weekes, & Ferstl, 2013; Kousta et al., 2009; Recio, Conrad, Hansen, & Jacobs, 2014), the affective categorization task (Estes & Verges, 2008), or the emotional Stroop task (González-Villar et al., 2014), among others.
It should be noted, however, that a number of lexical and sublexical variables influence word processing, such as frequency, age of acquisition, the number of orthographic neighbors (González-Nosti, Barbón, Rodríguez-Ferreiro, & Cuetos, 2014), syllable frequency (Carreiras, Alvarez, & De Vega, 1993), and the number of letters or syllables (Acha & Perea, 2008). In addition, word recognition is affected by semantic dimensions such as concreteness, imageability, number of associates, and semantic ambiguity (see Pexman, 2012, for an overview). Thus, to obtain reliable conclusions about emotion effects on processing, stimuli are needed that are not only well-characterized in terms of their emotionality, but also can be carefully controlled with respect to those variables. To address such a need, several affective norms have been published. The first was the Affective Norms for English Words (ANEW, Bradley & Lang, 1999), containing ratings for 1,034 words. This corpus has been adapted to other languages, including Spanish (Redondo, Fraga, Padrón, & Comesaña, 2007), European Portuguese (Soares, Comesaña, Pinheiro, Simões, & Frade, 2012), and Italian (Montefinese, Ambrosini, Fairfield, & Mammarella, 2014). However, as researchers have tried to control for more relevant variables, it has become apparent that larger stimulus sets are needed. To that end, other norm lists have been published in different languages, including English (Warriner, Kuperman, & Brysbaert, 2013), German (Võ, Jacobs, & Conrad, 2006; Võ et al., 2009), French (Monnier & Syssau, 2014), Finnish (Eilola & Havelka, 2010), Dutch (Moors et al., 2013), and Spanish (Ferré, Guasch, Moldovan, & Sánchez-Casas, 2012; Guasch, Ferré, & Fraga, 2015; Hinojosa et al., 2016; Stadthagen-González, Imbault, Pérez Sánchez, & Brysbaert, 2016).
Thanks to these normative databases, main effects of valence and arousal on word processing have been well-documented in different languages (Kousta et al. 2009; Scott et al., 2014; see also Jacobs et al., 2015, for an overview), and their electrophysiological and neuroanatomical correlates have been described (González-Villar et al., 2014; Herbert, Junghöfer, & Kissler, 2008; Hinojosa et al., 2010; Recio et al., 2014; Schacht & Sommer, 2009; see also Citron, 2012, for an overview). This research has revealed an advantage in processing for positive relative to neutral words (e.g., Bayer, Sommer, & Schacht, 2011; Kissler & Koessler, 2011; Kousta et al., 2009; Kuchinke, Võ, Hofmann, & Jacobs, 2007). Concerning negative words, a processing advantage has been found in some studies (e.g., Kanske & Kotz, 2007; Kousta et al., 2009; Larsen, Mercer, Balota, & Strube, 2008), but not in others (Algom, Chajut, & Lev, 2004; Estes & Verges, 2008; Hinojosa, Albert, López-Martín, & Carretié, 2014). Such mixed results with negative words might be explained by an interaction with arousal (Citron et al., 2013). Indeed, Robinson, Storbeck, Meier, and Kirkeby (2004) proposed that stimuli with negative valence or high arousal elicit a withdrawal orientation because they represent a possible threat, whereas stimuli with positive valence or low arousal elicit an approach orientation because they are perceived as safe. Thus, facilitative processing would be expected for positive low-arousing words and negative high-arousing words, but not for negative low-arousing words. The few studies that have manipulated valence and arousal orthogonally, however, have not reported conclusive findings concerning the interaction of these variables (e.g., Citron et al., 2013; Recio et al., 2014).
The research reviewed above reveals that, although the modulation of word processing by the words’ emotional content is firmly established, inconsistencies in the exact patterns of findings have been apparent, in terms of the effects of valence and arousal manipulations. Perhaps some of these inconsistencies might be explained if we take into account that the emotional content of words can be described not solely in terms of valence and arousal. Indeed, other aspects should be considered when describing the emotional content of words in particular, and human emotional experience in general. This is the proposal of the so-called discrete-emotion theories, the second major theoretical approach to the description of the affective space. These theories assume a limited number of discrete emotions with specific characteristics, physiological correlates, behavioral action tendencies, and associated emotional experiences (e.g., Ekman, 1992; Panksepp, 1998). Although there is no agreement as to the number of discrete emotions that actually exist, at least five—happiness, sadness, anger, fear, and disgust—have been consistently identified (Briesemeister, Kuchinke, & Jacobs, 2011a). Discrete-emotion theories have been supported mainly by research focused on emotion recognition from facial expressions (e.g., Campbell & Burke, 2009; Elfenbein, Beaupré, Lévesque, & Hess, 2007). In contrast, research using this approach that has looked at words is very scarce. Only recently have a small number of studies appeared (Briesemeister, Kuchinke, & Jacobs, 2011a, 2011b, 2014; Briesemeister, Kuchinke, Jacobs, & Braun, 2015; Silva, Montant, Ponz, & Ziegler, 2012). These have been made possible thanks to the publication of normative data from which the experimental stimuli have been obtained (Briesemeister et al., 2011b; Hinojosa et al., 2016; Stevenson, Mikels, & James, 2007). However, the number of published datasets developed from the discrete-emotion approach is much lower than the number from the dimensional perspective. Hence, an effort should be made to fill this gap. This is the aim of the present study, in which we provide ratings for a large set of Spanish words in five discrete emotion categories.
To our knowledge, only five corpora are currently available in which words have been rated according to discrete emotions. Two of these include English words (Stevenson et al., 2007; Strauss & Allen, 2008), and the others focus on German (Briesemeister et al., 2011b), French (Ric, Alexopoulos, Muller, & Aubé, 2013), and Spanish (Hinojosa et al., 2016). In some cases, the approach followed by the authors has been to supplement previously published databases that included ratings of valence and arousal by collecting ratings of the same words in discrete emotions. Thus, both the ANEW (Stevenson et al., 2007) and the Berlin Affective Word List (BAWL; Briesemeister et al., 2011b) have already been characterized in terms of discrete emotions. In other cases, the authors have collected discrete-emotion ratings for new sets of words (Hinojosa et al., 2016; Ric et al., 2013; Strauss & Allen, 2008).
The availability of these databases is crucial for research on discrete emotions. Such research will undoubtedly provide a more comprehensive picture of emotion effects on language processing, as it has been demonstrated that the information provided by discrete emotion categories does not overlap with that provided by valence and arousal. For instance, Stevenson et al. (2007) run regression analyses using the dimensional data of each word to predict the categorical data and vice versa, finding a lack of homogeneity in the capacity of one type of rating to predict the other. According to the authors, these results reveal that the two sets of ratings provide different information about the stimuli. Taking these results into account, Stevenson et al. (2007) pointed out that if researchers wish to control for as many variables as possible when selecting their stimuli, both dimensional and categorical data should be taken into account. Similar conclusions were reached by Briesemeister et al. (2011b) and by Hinojosa et al. (2016).
In support of the above claim, recent research has revealed that discrete emotions affect word recognition above and beyond valence and arousal. In particular, Briesemeister et al. (2011a) identified the best predictive variables for lexical decision performance with a multiple regression analysis conducted with response times (RTs) contained in the English Lexicon Project (Balota et al., 2007). They found that happiness, fear and disgust ratings were significant predictors of RTs. In particular, high-happiness scores as well as high fear scores led to faster RTs. On the contrary, high disgust scores were related to slower RTs. It is worth mentioning here that, although both high fear-related words and disgust-related words have a negative valence, they behave differently in the lexical decision task.
Further evidence has emerged from a series of studies conducted by Briesemeister and his colleagues with German words selected from the BAWL. They directly contrasted the dimensional and the discrete-emotion models by performing factorial manipulations of discrete-emotion content while controlling for valence and arousal. For instance, Briesemeister et al. (2011b) compared high and low fear-related words and high and low happiness words, finding that high fear-related words and high-happiness words facilitated lexical decisions, with respect to their low conditions’ counterparts. This is a significant finding if we consider that high and low conditions were matched in both valence and arousal. In a similarly oriented study, Briesemeister et al. (2011a) tested five sets of words related to either happiness, disgust, fear, anger, or those not related to any discrete emotion (i.e., neutral) in a lexical decision task. Results showed that happiness-related words were responded to faster than neutral words. Importantly, although RTs were slower in the disgust, fear, and anger conditions than in the happiness condition, there were also differences among the three negative conditions; namely, disgust-related words took more time to be recognized than either fear or anger-related words.
Taken together, the results obtained with the different types of negative words (Briesemeister et al., 2011a, 2011b) suggest that the composition of the stimulus list in studies conducted from a dimensional perspective may have contributed to the inconsistent pattern of findings reported with these words (Algom et al., 2004; Estes & Verges, 2008; Kanske & Kotz, 2007; Kousta et al., 2009; Larsen et al., 2008). Indeed, if negative words can facilitate or slow down processing depending on the discrete emotion to which they are related, the proportions of the different negatively valenced discrete-emotion words included in a particular study can determine the obtained effects (Briesemeister et al., 2011a).
Importantly, the reviewed findings also indicate that discrete-emotion content influences word processing beyond the well-known effects of valence and arousal. Therefore, both dimensional and discrete-emotion information should be taken into account in research about emotion and language processing. The same conclusion was reached in two recent studies in which happiness and positivity were orthogonally manipulated. In particular, Briesemeister et al. (2014) investigated the temporal dynamics of affective processing using event related potentials (ERPs), finding that discrete-emotion effects occurred earlier than dimensional effects. In a similarly oriented fMRI study, Briesemeister et al. (2015) reported that the brain areas affected by the manipulation of positivity were not the same as those affected by the manipulation of happiness. According to these authors, such results cannot be accounted for by either discrete-emotion theories or dimensional theories alone.
As can be seen from the above, the effects of discrete emotion categories on word recognition processes seem to be well established, at least with German words selected from BAWL and lexical decision tasks. However, more research has to be done involving other experimental tasks and paradigms, as well as other languages, in order to arrive at a more complete understanding of the impact of discrete emotion categories on language processing. Furthermore, some experimental approaches, such as the use of ERPs, require large sets of items. Therefore, it is crucial to develop corpora in different languages, as has been done from the dimensional approach.
The aim of the present study was to provide affective norms for a set of 2,266 Spanish words in five discrete emotion categories: happiness, anger, fear, disgust and sadness. This will be the largest corpus to date using a discrete-emotion approach. We aimed at collecting discrete-emotion ratings of words already rated from a dimensional perspective in three affective norms published in Spanish: The Spanish adaptation of the ANEW (Redondo et al., 2007) and the corpora of Ferré et al. (2012) and Guasch et al. (2015). Therefore, the new data concerning the characterization of these words in discrete emotion categories will complement their valence and arousal data. In this way, the current dataset will provide researchers with a large set of words to be used in studies aimed at taking into account both the dimensional and the discrete-emotion perspectives in the study of emotion in language processing.
We collected ratings from 1,380 Spanish speakers (1, 076 females, 304 males). Their mean age was 21.70 (SD = 5.98), ranging from 18 to 61 years old. The majority of the participants were undergraduate or master’s students from three Spanish universities: the University Rovira i Virgili in Tarragona (URV; 42 % of the sample), Complutense University of Madrid (UCM; 42 % of the sample), and the University of Santiago de Compostela (USC; 16 % of the sample). URV and USC are both in bilingual communities, and the participants from these universities also knew the other official language (Catalan and Galician, respectively). Participants took part in the study voluntarily or in exchange for academic credits. None of them reported a major history of psychiatric or neurological disorder.
Materials and procedure
The final set used included 2,266 Spanish words. The criterion for selecting the words was quite straight forward: we aimed at collecting discrete-emotion ratings of words already rated from a dimensional perspective. Therefore, we focused on the Spanish adaptation of the ANEW (Redondo et al., 2007), containing 1,034 words; on the corpus of Ferré et al. (2012), including ratings for 380 words grouped by the semantic categories of animals, objects and people; and on the norms from Guasch et al. (2015), with ratings for 1,400 words. After excluding repeated words, the sum of the three datasets was of 2,419 different words. We then checked for duplicates in the only corpus of discrete emotions published to date in Spanish (Hinojosa et al., 2016), finding 153 words. These words were excluded from the final set.
The selected words were divided randomly into 46 lists of 50 words each (except the two final lists, which were somewhat shorter). All the words included in a particular list were rated on the scales of the five discrete emotions. The surveys were created and distributed using SurveyMonkey, with different URLs for each list, which were distributed randomly among the respondents. Each survey was completed by exactly 30 participants.
We kindly ask you to rate each of the following words according to the five emotional categories presented: happiness, anger, sadness, fear and disgust. Please, answer all of the words in the questionnaire by selecting the score that you consider appropriate on a scale of 1 to 5, with 1 being “nothing at all” and 5 “extremely.”
On the following screens participants were presented with the words, one word per screen, centered in the middle of the display and written in lower case Helvetica font, 14-point bold. Participants used scales displayed below the word for their ratings. It was compulsory to answer on each scale before proceeding to the next word. However, participants had the option to mark a word as unknown. Finally, the order of the words, as well as the order of the five scales for each word, was randomized in all the questionnaires.
Results and discussion
The final dataset is available as Supplementary material. It is presented in an Excel spreadsheet including the 2,266 words sorted alphabetically in Spanish. English translations are also provided. Ratings of the five discrete emotions are presented in the following order: happiness (Hap), anger (Ang), sadness (Sad), fear (Fea), and disgust (Dis). For each variable, we include the mean value of the ratings (Mean) and the standard deviation (SD). An additional column shows the number of ratings for each word (N). Although each list was responded to by 30 participants, they had the option to mark a word as unknown. Of the words in the dataset, 88.66 % had 30 ratings; the remaining 11.34 % received from eight to 29 ratings (M = 26.98, SD = 3.63). We provide this information because it might be useful to other researchers in selecting those words for their experiments that participants would be most likely to know.
Descriptive statistics of the ratings for the five discrete emotions (upper section), and other relevant affective and psycholinguistic indices (lower section)
Reliability and validity of the measurements
First, we explored the interrater reliability of the ratings for each of the 46 versions of the questionnaire with a split-half procedure. Since 30 participants completed each questionnaire, we divided them into two subgroups of 15 participants. Then, we computed the Pearson correlations between both sub-groups applying the Spearman–Brown correction. The results showed very high and positive correlations for the five discrete emotions. The mean values were r = .97 for happiness (ranging from .93 to .99), r = .96 for anger (ranging from .89 to .99), r = .97 for sadness (ranging from .91 to .99), r = .96 for fear (ranging from .88 to 1), and r = .95 for disgust (ranging from .90 to .98). As can be seen, the reliability barely changed among emotional categories. It is worth noting that these values were almost identical to those reported by Hinojosa et al. (2016) for a different set of words rated in the same discrete emotions.
We also aimed at assessing the validity of our ratings by correlating them with the scores obtained in other studies. However, as we previously explained, no words overlapped with the only existing corpus for discrete emotions published in Spanish (Hinojosa et al., 2016). For that reason, we focused on English, German, and French norms. Concerning English, we relied on the norms of Stevenson et al. (2007). In all, the datasets had 1,012 words in common (out of the 1,034 words contained in ANEW). The Pearson correlations were r = .90 for happiness, r = .86 for anger, r = .87 for both sadness and fear, and r = .83 for disgust. With respect to German, 590 words overlapped between our dataset and the norms of Briesemeister et al. (2011b). The Pearson correlations were r = .78 for happiness, r = .77 for anger, r = .73 for sadness, r = .80 for fear, and r = .74 for disgust. Finally, our dataset had only 98 words in common with the French norms of Ric et al. (2013). The Pearson correlations were r = .94 for happiness, r = .83 for anger, r = .88 for sadness, r = .85 for fear, and r = .83 for disgust. Overall, these correlations were as high, for example, as the correlations reported for arousal ratings across different datasets in the same language (e.g., Guasch et al., 2015). This is particularly interesting given that we were comparing ratings of words in different languages. In a similar vein, Redondo et al. (2007) calculated the correlations of valence and arousal ratings between the Spanish and the American versions of ANEW (Redondo et al., 2007, and Bradley & Lang, 1999, respectively). The value of the correlation coefficients were r = .92 for valence and r = .75 for arousal. When comparing these correlations to those obtained in the present study, it seems that ratings obtained from the discrete-emotion approach are at least as stable across cultures as ratings obtained from a dimensional perspective.
Means and standard deviations (in parentheses) of the ratings in the five discrete emotions, by gender
The Pearson correlations between males and females for each discrete category were r = .86 for happiness, r = .80 for anger, r = .82 for sadness, r = .80 for fear, and r = .79 for disgust. These correlations are positive and significant (all ps < .001), as had previously been reported by Hinojosa et al. (2016) in their analysis of gender effects on discrete-emotion ratings. Thus, it seems that male and female participants rated the words in similar veins, in line with past studies reporting a high level of agreement between the sexes in their dimensional ratings of valence and arousal (Redondo et al., 2007; Soares et al., 2012).
To explore in more detail the possible effects of gender, we ran paired t tests for each emotional category. On average, male participants rated words significantly higher than females for happiness (t = 3.44, p < .01), anger (t = 7.26, p < .01), and disgust (t = 6.52, p < .01). On the contrary, women rated words significantly higher than men for fear (t = 8.72, p < .01). No sex differences were found in ratings for sadness (t = 1.62, p > .05).
Finally, we also explored how generalized were gender differences across the dataset. Some words did indeed show extreme differences between genders in their ratings. For instance, the word suegra (“mother-in-law”) was rated higher in happiness by women than by men (with a difference of 1.81 points). On the contrary, men rated the word tanga (“thong”) as more related to happiness than women did (with a difference of 1.80 points). To examine the frequency of words that showed sex differences, we ran t tests on ratings for each individual word for all five discrete emotional categories. With a level of significance of .05, some 9.75 % of words differed between the sexes for happiness, 9.36 % for anger, 9.31 % for sadness, 7.63 % for fear, and 10.06 % for disgust. Thus, only a limited number of words (9.22 % on average) showed sex differences in discrete-emotion ratings, in line with past studies involving similar analyses (Stevenson et al., 2007). Taken together, the results of the analyses by sex suggest that although there is high consistency between the ratings of males and females, some gender differences exist. It should be noted, however, that these differences were minimal (see Table 2), ranging in magnitude from 0.04 to 0.10 points. Of note, caution should be taken in drawing any firm conclusions from these analyses, considering the disproportionate number of females in the sample (only 22 % of the respondents were males).
Relationship between discrete emotions
Pearson correlations among the discrete emotions of happiness (Hap), anger (Ang), sadness (Sad), fear (Fea), and disgust (Dis), as well as correlations between the discrete emotions and the dimensions of valence and arousal
Happiness ratings showed negative correlations with those of negative emotions, whereas the latter correlated positively among themselves. Anger exhibited high correlations with the other negative emotions, and sadness and fear were also highly correlated. Finally, disgust was the category showing the lowest correlations with the other negative emotions, even though they were still significant.1 It is worth noting that the overall pattern of correlations is very similar to those found by Hinojosa et al. (2016) in Spanish and by Briesemeister et al. (2011b) in German. That is, just as in the present study, happiness was also negatively correlated with the negative emotions in these two databases. Moreover, both studies reported high correlations between fear and the emotions of anger and sadness, as well as a lower correlation of disgust with the other negative emotions.
Relationship between discrete emotions and affective dimensions
We also analyzed the pattern of correlations between discrete emotions and the affective dimensions (i.e., valence and arousal). To compute these correlations, we took the values of valence and arousal from the database of Guasch et al. (2015), the Spanish adaptation of the ANEW (Redondo et al., 2007), and the database of Ferré et al. (2012; see Table 3). The ratings for happiness and valence had a highly significant positive correlation, whereas the ratings for the negative emotions showed negative correlations with valence. On the contrary, the correlation with arousal was negligible for happiness, but moderate and positive for the four negative emotions. Among these emotions, fear and anger had the highest correlations with arousal, followed by sadness and disgust.2 Of note, the present findings are comparable to those reported by Hinojosa et al. (2016) and Briesemeister et al. (2011b). Namely, in both databases valence was positively correlated with happiness and negatively correlated with the negative emotions. Furthermore, the correlation of arousal with the negative emotions was highest for fear, followed by anger and sadness in both studies. Finally, disgust was the negative emotion that correlated least with arousal in all cases. This pattern of correlations is logical if we consider the characteristics of the words included in each discrete category; that is, words denoting high levels of anger (e.g., cruelty) or fear (e.g., murder) are often used to refer to highly arousing events or situations. In contrast, one can easily think of many events and stimuli eliciting high levels of happiness (e.g., peace), sadness (e.g., lonely), or disgust (e.g., trash) that are not arousing at all.
Distribution of words among the five discrete emotions
Words belonging to each emotional category, considering the highest rating for each word: Number and percentage of words included in that category, average ratings of these words in that category, and valence and arousal means (standard deviations in parentheses)
Comparing the percentages of words falling under each category with the data of Hinojosa et al. (2016), the pattern is highly consistent, in that those authors also found a large pool of words associated with happiness (35.1 % of their database), and lower percentages of words belonging to each negative emotion (i.e., anger, 9.6 %; sadness, 9.9 %; fear, 13 %; disgust, 3.4 %). Our distribution also agrees with that observed by Hinojosa et al., if we sort the negative emotions by the percentage of words belonging to each one. Indeed, fearful words are the most numerous negative-emotion stimuli in both datasets, whereas the number of words related to disgust is the lowest.
Numbers and percentages of words with high values in one emotional category and low values in all other categories (“pure” words)
As can be seen in Table 5, 80.27 % of the stimuli in the dataset can be considered “pure” words, either because they were mainly assigned to a single emotion (42.49 %) or because they were neutral with respect to the five categories (37.78 %). Concerning the former, words denoting happiness were the largest set (31.07 % of the database). In fact, 95.52 % of happy-related words were “pure.” We would note that the number of “pure” words in the dataset not only was larger for happiness than for the other emotions, but also that happiness-related words were the most frequent. Indeed, we examined whether differences in lexical frequency depended on the discrete emotion category through a one-way analysis of variance. This analysis showed a significant effect, F(5, 1818) = 7.03, p < .01. Bonferroni-corrected comparisons revealed that happiness-related words were more frequent (M = 55.32) than either disgust (M = 5.94) or neutral (M = 26.26) words. These results are in agreement with literature showing a positivity bias in language (Augustine, Mehl, & Larsen, 2011; Dodds et al., 2015).
The pattern of results observed for happiness-related words contrasts with that observed for words assigned to negative emotions. Indeed, negative words with high values in only one category (i.e., “pure” words) accounted for only 11.43 % of the total. Thus, the majority of negative words showed high scores in more than one discrete emotion. This held true for anger-, sadness-, and fear-related words. The only exception in the negative domain was disgust, with more “pure” words—65.71 %—than mixed ones.
Turning now to mixed words (i.e., the words that had high values in more than one discrete emotion), these represented 19.73 % of the total database (N = 447). Among them, only 40 (1.77 % of the total) were words with values higher than 2.5 in both happiness and at least one of the four negative emotions (e.g., the word pasado “past,” with high ratings for happiness and sadness). Thus, 407 of the words (17.96 % of the total) mixed only negative emotions (i.e., anger, sadness, fear, or disgust). Taken together, these results suggest that approximately equal numbers of positive (32.37 %), neutral (37.59 %), and negative (30.04 %) words are in the corpus, but that they are distributed differently between “pure” and mixed words in different categories. Thus, although large sets of the words have “pure” happy connotations or no emotional connotations (i.e., neutral words with regard to the five discrete emotions), the negative words are more heterogeneous. That is, negative words do not group around four definite clusters of “pure” anger, sadness, fear, or disgust, since only 11.43 % of the total words can be classified in a single negative category.
To explore the question of whether there was a consistent pattern to the discrete negative emotions that went together, we conducted an additional analysis with negative words. To this end, we removed neutral words, “pure” happy words, and words whose highest rating was for happiness. The remaining set comprised 673 words that had their highest rating in one of the four negative emotions. Then we applied the Proxscal procedure (Busing, Commandeur, & Heiser, 1997) from multidimensional scaling (MDS). MDS is an exploratory data analysis technique that facilitates the visualization of the similarity relations among data points in complex datasets. In our case, each word could be represented as a single point in a five-dimensional space (each dimension corresponding to each of the five discrete emotions). However, even a three-dimensional representation of the data was difficult to understand. MDS algorithms allow the reduction of the number of dimensions of any n-dimensional space, while also preserving the possible distances among the objects represented. Therefore, our initial five-dimensional space could be approximated and plotted on a two-dimensional graph by using an MDS procedure. After applying this procedure, the resulting space had a Kruskal’s stress value of .20. Although this value of adjustment is somewhat too high, it is far from the value that would be expected by chance (Sturrock & Rocha, 2000). Hence, the resulting space is appropriate to explore the internal structure of the data.
As can be seen, the words with a high rating in anger (panel A) concentrate on the right side of the space. Some anger words are “pure” (mainly at the bottom), but the remaining anger words seem to be evenly mixed with the other three emotions. Concerning sadness and fear (panels B and C, respectively), words related to these two emotions are similarly distributed in the space (i.e., concentrated to the left of the vertical axis). There are some “pure” sad words (panel B, bottom), as well as “pure” fearful words (panel C, bottom). Regarding the “nonpure” words, it is worth noting that the mixed sad words combine mainly with fear on the upper-left side (panel B), whereas the mixed fear words combine mainly with sadness in that quadrant (panel C). Thus, it seems from the graphs that sadness and fear are two closely related emotions. Focusing now on fear (panel C), it is worth noting that a significant pool of fearful words have second-highest ratings for anger. Finally, disgust words (panel D) have their own area in the space (bottom right). Interestingly, these words are mainly “pure” rather than mixed, suggesting that disgust is the least overlapped emotion.
The findings reported in this section reveal differences among the five discrete emotion categories in the extents to which words are exclusively related to each of the categories (i.e., “pure” words), as well as with respect to the mixing patterns that emerge (i.e., the discrete emotion categories that correlate with each other). Namely, happiness words are the most consistently related to a single discrete category. Focusing on negative emotions, anger is the discrete emotion that contains the lowest percentage of “pure” words. In fact, it correlates most with the other negative discrete categories, including similar numbers of words that involve sadness, fear, and disgust. The other two categories with low percentages of “pure” words are fear and sadness. Interestingly, the high correlation between these two categories suggests that many words have high ratings for both emotions. Finally, disgust is the negative emotion that contains the highest percentage of “pure” words and that correlates the least with the other discrete categories. Even though some caution is needed when interpreting results on the basis of 30 ratings per word, and though it would be desirable to conduct further studies by increasing the number of ratings per word as well as the number of rated words, we believe that these findings have relevant implications for research on emotion and language.
Before drawing any conclusions concerning the properties of affectively valenced words, we should first rule out the possibility that the findings above are the result of either the particular set of words included in the present database or the characteristics of our sample of respondents. To this end, we analyzed in detail the datasets of Briesemeister et al. (2011b), Hinojosa et al. (2016), and Stevenson et al. (2007). By using the same criterion as in the present study, we classified the words included in these databases as either “pure” or mixed. The pattern of results was very similar across databases. In all cases, happiness was the most homogeneous category (98.13 %, 99.31 %, and 99.64 % of the happiness-related words in the databases of Hinojosa et al., 2016; Stevenson et al., 2007; and Briesemeister et al., 2011b, respectively, were “pure” words), whereas “disgust” was always the negative category with the highest percentage of “pure” words (48.39 %, 48.53 %, and 76.36 % for Hinojosa et al., 2016; Stevenson et al., 2007; and Briesemeister et al., 2011b, respectively). These results suggest that the findings of the present study generalize to other word sets, languages, and populations.
Taking all of the above results into consideration, we can conclude that the discrete emotion categories are consistently different in the extents to which the words related to them are “pure” or mixed. It is worth discussing this point here in more detail. Concerning “pure” words, the highest percentages are found in the “happiness” and “disgust” categories. With respect to happiness, it should be noted that it was the only positive category. Thus, it is not surprising that it has the highest percentage of “pure” words, since there were no other positive categories to which these words could be attached. This contrasts with negative words, which could be related to four different emotions. Thus, before drawing firm conclusions concerning happiness related words, it would be interesting to conduct further studies in which more positive categories are available for word ratings. Also of interest is the high homogeneity found in disgusting words. In order to interpret this result, we first asked whether other characteristics of the words, those non-related to their emotional content, could be affecting the ratings. When we looked for the concreteness and imageability values of our words in the available databases (Duchon et al., 2013; Guasch et al., 2015), we found that disgust-related words were more concrete (M = 5.33 in a 1–9 scale) than words related with anger, sadness and fear (Ms = 4.48, 4.62, and 4.96, respectively; all ps < .05). Similarly, disgust-related words were easier to imagine (M = 5.13 in a 1–9 scale) than words related with anger and sadness (Ms = 4.30 and 4.63, respectively; all ps < .01). Thus, disgust-related words might have been rated more consistently as associated with disgust because they are more concrete and the things in the world they refer to are easier to imagine (e.g., trash). There is also an alternative explanation, pertaining to disgust as an emotion itself. It has been suggested that disgust has its origins in an old food rejection system based on distaste (Rozin, Haidt, & McCauley, 2008), this emotion having then been shaped by evolution and expanded to a wider range of elicitors. These come mainly from nine domains: food, body products, animals, sexual behavior, contact with death or corpses, violations of the exterior envelope of the body, poor hygiene, interpersonal contamination, and certain moral offenses (Haidt, McCauley, & Rozin, 1994). Although these are disparate domains, all are related to appraisals of either a sense of potential oral incorporation, a sense of offensiveness or a potential for contamination (Rozin & Fallon, 1987). If we look at the above list, we can see that many disgust elicitors are highly delimited and not likely to elicit other kinds of emotions. This fact, together with the higher concreteness and imageability of disgust-related words, could be the cause of the large amount of “pure” words in this emotional category.
Finally, we would focus on the categories of words that go together, and thus that produce high percentages of mixed words. There may be two reasons for this result. On the one hand, it may be that different groups of participants rated the same word with high values in different emotions. On the other hand, it is possible that certain participants rated these words with high values in more than one discrete emotion. To decide which of the two options is more plausible, we looked at the analysis of the interrater reliability of the ratings. Since this analysis was computed by dividing the ratings into two subgroups of participants (see the Reliability and Validity of the Measurements section), the very high and positive correlations observed for the five discrete emotions obtained revealed high consensus between the two subgroups of participants; otherwise, the correlations would have been lower (e.g., if one subgroup of participants, but not the other, had rated “past” as high in happiness, whereas the other subgroup had rated the same word as high in sadness). Thus, we speculate that the classification of a large number of words in the database as “mixed” is probably due to the second possibility—namely, that certain participants consider these words as having high values in more than one discrete emotion. In our opinion, what these results reveal is that many events and life circumstances elicit mixed emotions.
Our results show that anger-related words are those that mixed the most with other categories. Also interesting is the proximity of the words related to both sadness and fear. For instance, the word “rape,” the one rated highest in anger in the dataset (M = 4.87), also has very high ratings in sadness (M = 4.47), fear (M = 4.57), and disgust (M = 4.73). This is logical if we think about the meaning of this word and the kind of emotions that such a situation can elicit. The same holds true if we focus on the overlap between fear-related and sadness-related words. Many words in the dataset have high ratings in both discrete emotions (e.g., death, cancer, homicide, and torture, among others), suggesting that when people think about the meaning of a particular word, different aspects related to disparate emotions can be activated. For instance, people are afraid of death but they also feel sad when a loved one dies. In our opinion, the high percentage of mixed words found in the present study shows that people’s ratings reflect the heterogeneity of the emotions produced by many life situations. According to Oatley and Johnson-Laird (1996), people react to events by making multiple cognitive evaluations, which may, in turn, elicit different emotional reactions simultaneously or in rapid alternation. This would be in agreement with the literature that has shown the existence of mixed emotions, even when these are opposite in valence (Berrios, Totterdell, & Kellett, 2015; Larsen & McGraw, 2014; Trampe, Quoidbach, & Taquet, 2015). Of note, it has been demonstrated recently that these emotionally ambivalent states can be captured by people’s ratings of emotionally charged words (Briesemeister, Kuchinke, & Jacobs, 2012).
This study presents normative ratings for a large set of words in five discrete emotion categories. Our main aim was to provide an instrument that facilitates experimental research into the effects of discrete emotions on word processing. Furthermore, as the words in the dataset were previously rated for valence and arousal in other normative studies (Ferré et al., 2012; Guasch et al., 2015; Redondo et al., 2007), this corpus will enable researchers to select words taking into account both the dimensional and the discrete-emotion perspectives. Recent behavioral reports of discrete-emotion effects on word processing that cannot be accounted for by valence and arousal (Briesemeister et al., 2011a, 2011b), as well as electrophysiological evidence of temporal differences between dimensional and discrete-emotion processing (Briesemeister et al., 2014), suggest that these norms can be very useful in this field of research.
To provide a detailed description of the dataset, we analyzed the pattern of relationships among affective ratings. Our results agreed with those found in other studies conducted in Spanish (Hinojosa et al., 2016) and German (Briesemeister et al., 2011b). On the one hand, we found positive correlations among the four negative emotions and negative correlations between happiness and the four negative emotions. On the other hand, discrete-emotion ratings correlated higher with valence than with arousal. Furthermore, valence correlated positively with happiness ratings and negatively with ratings for negative emotions. The convergence of results across studies was also evident when we examined the pattern of correlations of a subset of words for which ratings were also available in English (Stevenson et al., 2007), German (Briesemeister et al., 2011b) or French (Ric et al., 2013). The results of this comparison revealed that ratings obtained from the discrete-emotion approach are highly stable across cultures.
We also present a detailed description of the distribution of words into the five emotional categories. We observed that the dataset contains approximately equal numbers of positive, negative, and neutral words. Among them, positive words (i.e., words related to happiness) were the most consistently related to a single, discrete category. In contrast, negative words tended to be mixed, including many words that belonged to more than one discrete emotion. In particular, the pattern of associations between ratings for negative emotions suggests that a high number of fear-related words also scored high in sadness, whereas disgust-related words were those that overlapped least with the others.
In summary, the present database provides subjective ratings for 2,266 Spanish words in discrete emotion categories. The analyses carried out confirm the reliability and consistency of the data. Moreover, the results concerning the relationships among affective ratings are consistent with the patterns found in previous studies. Finally, the fine-grained analysis of the distribution of words into the emotional categories revealed clear differences regarding the extents to which the words could be assigned to a single, discrete emotion category. For these reasons, these norms may be considered an appropriate tool for the design of experiments that seek to contrast the predictions of the dimensional and discrete-emotion perspectives.
To decide whether or not these magnitudes differed significantly, we computed the 95 % confidence intervals of the correlations using Fisher’s z transformation. If there is no overlap between the confidence intervals of a pair of correlations, we can conclude that significant differences distinguish them. This analysis revealed no overlap between the confidence intervals of the comparisons in which disgust was involved and those in which disgust was not involved. This result supports our statement that disgust is the least-correlated negative emotion.
As for the correlations between discrete emotions, we computed the 95 % confidence intervals of the correlations between discrete emotions and affective dimensions using Fisher’s z transformation. The analysis showed no overlap between the intervals, revealing that the differences among the correlation coefficients of the five discrete emotions and arousal were significant.
This work was supported by the Ministerio de Economía y Competitividad of Spain and the Fondo Europeo de Desarrollo Regional (MINECO/FEDER) (Grant Nos. PSI2015-63525-P, PSI2015-68368-P, and PSI2015-65116-P), as well as Ref. 2014PFR-URV-B2-37 from the University Rovira i Virgili, Ref. GRC2015/006 from the Comunidad Autónoma de Galicia, and Ref. H2015/HUM-3327 from the Comunidad de Madrid. The authors thank Ana Alonso and Mercedes González for translating the database to German, in order to compare the word ratings across languages.
- Balota, D. A., Yap, M. J., Hutchison, K. A., Cortese, M. J., Kessler, B., Loftis, B., . . . Treiman, R. (2007). The English lexicon project. Behavior Research Methods, 39, 445–459. doi:10.3758/BF03193014
- Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words (ANEW): Instruction manual and affective ratings (Technical Report C-1). Gainesville, FL: University of Florida, NIMH Center for Research in Psychophysiology.Google Scholar
- Bradley, M. M., & Lang, P. J. (2000). Measuring emotion: Behavior feeling, and physiology. In R. D. Lane & L. Nadel (Eds.), Cognitive neuroscience of emotion (pp. 242–276). New York, NY: Oxford University Press.Google Scholar
- Briesemeister, B. B., Kuchinke, L., & Jacobs, A. M. (2011a). Discrete emotion effects on lexical decision response times. PLoS ONE 6, e23743. doi:10.1371/journal.pone.0023743
- Busing, F. M. T. A., Commandeur, J. J., & Heiser, W. J. (1997). PROXSCAL: A multidimensional scaling program for individual differences scaling with constraints. In W. Bandilla & F. Faulbaum (Eds.), Softstat ’97: Advances in Statistical Software (pp. 237–258). Stuttgart, Germany: Lucius.Google Scholar
- Dodds, P. S., Clark, E. M., Desu, S., Frank, M. R., Reagan, A. J., Williams, J. R., . . . Megerdoomian, K. (2015). Human language reveals a universal positivity bias. Proceedings of the National Academy of Sciences, 112, 2389–2394. doi:10.1073/pnas.1411678112
- Guasch, M., Ferré, P., & Fraga, I. (2015). Spanish norms for affective and lexico-semantic variables for 1,400 words. Behavior Research Methods. Advance online publication. doi:10.3758/s13428-015-0684-y
- Hinojosa, J. A., Martínez-García, N., Villalba-García, C., Fernández-Folgueiras, U., Sánchez-Carmona, A., Pozo, M. A., & Montoro, P. R. (2016). Affective norms of 875 Spanish words for five discrete emotional categories and two emotional dimensions. Behavior Research Methods, 48, 272–284. doi:10.3758/s13428-015-0572-5 CrossRefPubMedGoogle Scholar
- Jacobs, A. M., Võ, M. L. H., Briesemeister, B. B., Conrad, M., Hofmann, M. J., Kuchinke, L., . . . Braun, M. (2015). 10 years of BAWLing into affective and aesthetic processes in reading: What are the echoes? Frontiers in Psychology, 6, 714. doi:10.3389/fpsyg.2015.00714
- Moors, A., De Houwer, J., Hermans, D., Wanmaker, S., van Schie, K., van Harmelen, A. L., . . . Brysbaert, M. (2013). Norms of valence, arousal, dominance, and age of acquisition for 4,300 Dutch words. Behavior Research Methods, 45, 169–177. doi:10.3758/s13428-012-0243-8
- Oatley, K., & Johnson-Laird, P. N. (1996). The communicative theory of emotions: Empirical tests, mental models, and implications for social interaction. In L. L. Martin & A. Tesser (Eds.), Striving and feeling: Interactions among goals, affect, and self-regulation (pp. 363–393). Mahwah, NJ: Erlbaum.Google Scholar
- Panksepp, J. (1998). Affective neuroscience: The foundations of human and animal emotions. New York, NY: Oxford University Press.Google Scholar
- Pexman, P. M. (2012). Meaning-based influences on visual word recognition. In J. S. Adelman (Ed.), Visual word recognition: Vol. 2. Meaning and context, individuals and development (pp. 24–43). Hove, UK: Psychology Press.Google Scholar
- Rozin, P., Haidt, J., & McCauley, C. R. (2008). Disgust. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 757–776). New York, NY: Guilford Press.Google Scholar
- Stadthagen-González, H., Imbault, C., Pérez Sánchez, M. A., & Brysbaert, M. (2016). Norms of valence and arousal for 14,031 Spanish words. Behavior Research Methods. doi:10.3758/s13428-015-0700-2
- Trampe, D., Quoidbach, J., & Taquet, M. (2015). Emotions in everyday life. PLoS ONE, 10, e0145450. doi:10.1371/journal.pone.0145450