The primary aim of this study was to validate a list of sentences that would extend current literature (Ben-David et al., 2011; Russ et al., 2008) to include 10 additional emotion categories. Previous studies have focused on sentences conveying only four emotions (i.e., angry, fearful, happy, sad) alongside semantically neutral sentences. The inclusion of a broader range of emotions, including seven additional positively-valenced emotions in the current study, significantly contributes to literature by attempting to validate sentences that include meaningful verbal content and convey more complex emotions. Although the number of sentences tested was not as expansive as lists validated by Ben-David et al. (2011) and Russ et al. (2008), the sentences were assessed by a significantly larger group of participants from a number of different Western countries, thus results are robust. Moreover, participants had a larger number of response options to choose from reducing the potential for inflated agreement due to chance and increasing the potential for varied interpretations and responses. Our use of diversity ratings to identify sentences that more discretely represented emotions is a strength of this study because it accounts for variations that may have been present in participants from different Western countries. As shown in Table 5, a total of 38 sentences were labelled with a low degree of diversity across the 10 emotion categories targeted in this study. The number of sentences within each category varied, ranging from only one sentence identified as representing pride, to six sentences for irritation.
Across the two studies, only six sentences were labelled as representing one of the five emotion categories included in Ben-David et al. (2011) and Russ et al. (2008), despite the fact all five terms were included as potential response options. None of these achieved a low diversity score, but two were labelled were with a moderate degree of diversity—one sentence in Study 1 was labelled as fear (Stop! I think there’s something in there) with a diversity score of 0.60, and one sentence in Study 2 was labelled as angry (If you do that again, I’m going to lose it) with a diversity score of 0.49. It appears that our conceptual knowledge of emotion may be better represented by a broader set of words that capture the complexity and social aspects of language used during human interactions (Turkstra et al., 2017). However, it should be noted that we did not generate sentences to specifically target the four emotion categories previously studied, nor did we include stimuli from Ben-David et al. (2011) or Russ et al. (2008). This decision was made to maximise the number of stimuli targeting complex emotions while still ensuring a reasonable number of total stimuli for participants, but it does limit the conclusions we can draw regarding people’s conceptual knowledge of emotion. To address this, future iterations of this work should include sentences previously validated by Ben-David et al. (2011) and Russ et al. (2008) as representing happy, sad, angry, fearful, and neutral to explore whether any of these sentences would be better represented by the broader, more complex emotion categories included in the current study. Comparison of those sentences to the ones included in the current study suggest that this might be the case. For instance, a number of the sentences listed as representing happiness in Ben-David et al.’s (2011) list may have been labelled as pride if the option were available (e.g., ‘I won an award’; ‘Good job, the crowd loved you’), or even contentment (e.g., ‘It’s a beautiful day outside’; ‘I really love nature’). Similarly, sentences in Russ et al. (2008) reported to represent angry may be alternatively labelled as irritation (e.g., ‘Why are you always testing my patience?’; ‘That noise is getting really annoying’) or even disgust (e.g., ‘He always acts like he’s better than everyone’; ‘Do you know how unjust that is?’). It is also possible that sentences constructed to convey neutral may be otherwise categorised if given response emotions that represent less intense emotions. For example, in Russ et al. (2008), participants categorised the sentence, ‘I wonder what that is about’ as neutral. However, based on similarly constructed sentences used in the studies presented here, this sentence might be identified as interest or even anxiety if these options were available. Further exploring the influence of response options on categorisation of emotion sentences would provide further insight into our conceptual knowledge of emotion.
Although the current study included a large number of potential response options, a number of sentences were distinctly identified as representing a single emotion as indicated by low Simpson diversity scores. Sentences conveying disgust and interest appeared to be more readily categorised and easier to achieve consensus across participants, with the majority of sentences in these categories achieving low diversity scores. For instance, of the nine sentences labelled as disgust, six of these were identified as with a low degree of diversity across participant responses. Similarly, of the eight sentences identified as interest, six of these achieved low diversity scores.
It is possible that the six low diversity disgust sentences were more readily identified because they conveyed examples of physical disgust (e.g., spoiled food) which are more easily distinguishable than examples of moral disgust. Sentences we had anticipated would be labelled as (moral) disgust based on violation of social or moral norms (Chapman & Anderson, 2012) had a diverse array of responses including disgust, anger, and irritation (e.g., ‘They just kept picking on her’; ‘Who would do something like that?’). Research has shown that people often use disgust and anger interchangeably when describing social-moral norm violations (Gutierrez et al., 2011). Thus, response patterns for these sentences at least partially support the explanation that the content conveyed norm violations. Gutierrez et al. (2011) suggest that distinguishing between disgust and anger in response to social-moral norm violations is contingent on the type of moral violation. Since our sentences were presented without situational context, participants would not have had the information required to make this distinction which may explain the large variability in response to these sentences.
Interest is said to be generated in contexts where people are prompted to pay attention and/or want to further explore a situation or experience (Fredrickson, 2000, 2004). This description applies to all six of the sentences identified as interest with low diversity. Three of these sentences included the use of question words (e.g., how, what) and the remaining three used statements of attentiveness (e.g., ‘I can’t wait to hear more’). While interest was included in the current study as one of the seven positively-valenced emotions, alternative responses to sentences identified as Interest suggest that this emotion category does not have a clear valence. These results support our previous findings in an emotion elicitation study in which there were similar positive and negative valence ratings in response to film clips targeting Interest (Zupan & Eskritt, 2020).
Sentences identified as pride and contentment tended to garner greater diversity in responses overall; only one pride sentence and two contentment sentences achieved a low diversity score. Pride is typically classified as a positive emotion and is said to occur in situations where someone has surpassed a standard or expectation, which leads to increased feelings of self-worth (Lewis, 2008). However, some may consider expressions of pride as negative since these expressions can be perceived as self-serving and aimed to elevate one’s status at the cost of others (Oveis et al., 2010). Evaluation of responses to all sentences identified as pride seemed to support this complexity in interpretation since the emotion that received the second highest proportion of responses varied widely and included both positive and negative emotion labels (see Appendices 1 and 2 for response patternsFootnote 4). It also possible that the characteristics or our participant sample contributed to this result. More than 95% participants in the current studies were from Australia, Canada, the United States, or United Kingdom. People from individualist, westernized cultures are less likely to associate pride with scenarios that describe others’ successes (Neumann et al., 2009). Although the sentences we constructed to target pride tended to begin with ‘I’, participants were still essentially interpreting emotion in response to a description about something external to themselves. This may at least partially explain the limited success we had in identifying sentences that represented pride with low diversity.
The response pattern for sentences labelled as contentment was more consistent than for pride, with happy found to have the second highest proportion of responses for seven of the eight sentences labelled as contentment. The emotion word most frequently listed in the free-text responses for all eight of these sentences was also happy. Although these results suggest that happy and contentment may not be well differentiated from one another, 70% of participants indicated that their preferred interpretation of the sentence was contentment for four of the eight sentences. This proportion of consensus far exceeds what would be expected by chance, even if one considers only the two response options (happy, contentment). Across the two studies, only two sentences were labelled as happy. Together, these results highlight the importance of including more than one positive response option in emotion studies since happy was not a preferred response option.
The results of this study add to the field by extending the list of potential sentences researchers can use with English speakers when examining recognition of emotion across different modalities. These sentences may be particularly useful for research investigating the complex interaction between emotional prosody and verbal content. This complex interaction has received little attention in emotion research despite the fact people manipulate the congruency between these cues to convey intention and meaning during social interactions. The lack of attention to this area of emotion recognition and inference may be due in part to the challenge of constructing sentences that do not include emotion words but are still capable of conveying a distinct emotion when presented with no context. The availability of validated sentences that extend beyond semantically neutral content and content that conveys only four basic emotions should contribute to furthering research in this area.
While we have identified 38 sentences that appear to distinctly represent 10 different emotion categories, the studies presented here have some limitations that need to be identified. First, participants were recruited via the research team’s social media pages and university list serves so the sample may not be a representative one. In addition, since the studies focused on the interpretation of language-based stimuli, we should have included education as a potential confounding variable. Finally, while we asked participants to identify the country they were from, we didn’t ask them to identify their cultural background. Although all participants reported English as their primary language and more than 95% were from Australia, Canada, United States, or United Kingdom, their cultural background may have been quite varied. However, the perception of emotion has been shown to accommodate mainstream culture for people using English in their everyday interactions after only a few years of living in an English-speaking country (Liu et al., 2017). Given that our speakers needed to identify English as their primary language to participate in this study, cultural influences in responses were likely minimal. Still, future studies should examine cultural influences on the interpretation of complex emotion sentences more carefully. This will be particularly important for researchers using these sentences to explore multisensory emotion perception, including the interaction between emotional prosody and verbal content, because cultural differences have been reported to influence the weight people place on different cues when identifying the emotion portrayed (Liu et al., 2015; Tanaka et al., 2010).
Although not a limitation per se, the number of sentences identified to represent each of the targeted emotion categories was limited. Future validation research aimed to increase the number of stimuli across these 10 categories, and particularly those categories for which we identified only a few distinct stimuli (i.e., amusement, anxiety, contentment, pride, surprise), would be beneficial. Future research might also consider validating sentences that target additional basic (e.g., contempt, joy), or complex, social emotions (e.g., shame, guilt). The role of context in the interpretation of these emotion sentences should also be explored. The words used as response options provide inherent context and constraint in perception (Feldman Barrett & Kensinger, 2010). Further validating these sentences, and those identified by Russ et al. (2008) and Ben-David et al. (2011), using different response options and formats would provide additional insight into the consistency in which these sentences represent their specified emotion category.
The sentences presented here were validated in isolation. In addition to exploring the influence of different response options, future research might explore how situational context provided via an introductory sentence or vignette may influence participants’ categorisation of the meaning the sentence conveys. Though still quite different to way in which we experience and talk about emotion in everyday life, it would provide us some initial insight into how context influences emotion perception and whether or not people are able to make use of language-based social and/or emotional cues. The majority of studies that have considered the influence of context on emotion perception have focused on the interpretation of facial emotion expressions in the presence of congruent/incongruent words (Brooks et al., 2017; Fugate, 2013; Gendron et al., 2012), vocal emotion expressions (Pell et al., 2011; Pell & Kotz, 2011), or evaluative statements (e.g., He thinks you are competent) (Schwarz et al., 2013). These studies all report an influence of context on participant responses. Further evaluating sentences such as these in context is valuable given society’s expanding use of text-based communication. Embedding these sentences within a body of text might also allow for computerized text analysis. Although traditionally used to analyse underlying intentions, emotions, and motivations in natural language samples (Tausczik & Pennebaker, 2010), computerized text analysis might be applied to bodies of text containing these sentences to provide insight into how particular patterns in word use and language structure might influence people’s perception and identification of emotion.