Since the applicability of the collected affective norms in experimental studies is highly dependent on their reliability, we addressed this issue by applying split-half reliability estimation, following descriptions provided in the literature (Monnier & Syssau, 2014; Montefinese, Ambrosini, Fairfield, & Mammarella, 2014; Moors et al., 2013). The whole sample was split into halves in order to form two groups with the odd and even experiment entrance ranks. Within each group, the mean ratings of each basic emotion were calculated for each picture. Pairwise Pearson’s correlation coefficients of these means between the two groups were then calculated and adjusted using the Spearman–Brown formula. All correlations were significant (p < .01). The obtained reliability coefficients were high and comparable to the values obtained in other datasets of standardized stimuli (Bradley & Lang, 2007; Imbir, 2014; Monnier & Syssau, 2014; Moors et al., 2013)- namely, r = .97 for happiness, r = .98 for sadness, r = .93 for fear, r = .94 for surprise, r = .95 for anger, r = .97 for disgust, r = .93 for arousal, and r = .98 for valence.
Ratings of the affective variables
For each picture, we obtained from 39 to 44 ratings (M = 41.33, SD = 2.06) on each scale from the 124 participants of the study. In order to further explore the present data, we divided the whole set of pictures by their valence classes into negative, neutral, and positive pictures, according to the criteria introduced in previous studies (e.g., Ferré, Guasch, Moldovan, & Sánchez-Casas, 2012; Kissler, Herbert, Peyk, & Junghofer, 2007). These criteria were based on the mean valences for negative, neutral, and positive pictures, which usually took values around 2, 5, and 7, respectively. Therefore, we classified pictures with values of valence ranging from 1 to 4 as negative (M = 3.10, SD = 0.58), pictures with values ranging from 4 to 6 as neutral (M = 5.02, SD = 0.55), and pictures with values ranging from 6 to 9 as positive (M = 6.52, SD = 0.39). These criteria resulted in the following proportions in the present database: 148 negative pictures (28.6 %), 203 neutral pictures (40.8 %), and 159 positive pictures (30.6 %). The following distributions of negative, neutral, and positive pictures were observed in the different content categories: animals (25.5 % negative, 43.9 % neutral, 30.6 % positive), faces (26.1 % negative, 30.4 % neutral, 43.5 % positive), landscapes (12.2 % negative, 38.8 % neutral, 49.0 % positive), objects (19.6 % negative, 69.6 % neutral, 10.8 % positive), people (55.0 % negative, 21.0 % neutral, 24.0 % positive).
The distributions of all of the basic emotions, as collected for each picture and with pictures divided by their valence classes, are depicted in Fig. 2. We split the full range of the basic emotions (1–7 on the rating scales) into seven bins. For each bin, the number of means falling within the bin range was calculated for each basic emotion separately. Numbers obtained in this way (normalized by dividing them by the number of pictures in a particular valence class) are plotted for each valence class separately in each of the panels of Fig. 2.
The distributions of all the basic-emotion intensity ratings among negative pictures seem to be skewed, with a strong bias toward the low range of the scale. Only 31 % and 23 % of the pictures were rated above the middle value of the rating scales (=4) for sadness and disgust, respectively. All of the other basic emotion intensities were almost always rated lower. This low-intensity bias, which stands for a relative lack of pictures presenting high-intensity values of basic emotions, is strongest for happiness and surprise and weakest for sadness and disgust. All of the basic emotion intensities were rated low among the neutral pictures, with the highest median value being for happiness (Mdn = 2.15). In the positive picture group, the distribution of happiness covers the middle of the rating scale, Mdn = 4.15.
The analysis above shows that the majority of images do not express just one discrete emotion, but rather are associated with several different emotional states. Therefore, from the practical point of view it might be important to select stimuli representing one particular emotion much more than any other. Such images will be very useful for further studies in which an emotional category is considered an important factor (Briesemeister et al., 2015; Chapman, Johannes, Poppenk, Moscovitch, & Anderson, 2012; Costa et al., 2014; Croucher, Calder, Ramponi, Barnard, & Murphy, 2011; Flom, Janis, Garcia, & Kirwan, 2014; Schienle et al., 2014; van Hooff, van Buuringen, El M’rabet, de Gier, & van Zalingen, 2014). Importantly, several methods of stimulus classification according to the basic emotion categories available in the literature (Briesemeister et al., 2011b; Mikels et al., 2005) can be employed, depending on the specific interest of the researcher. One of the most popular is based on the overlapping of confidence intervals (CIs; Mikels et al., 2005). Using this method, the 85 % CI was constructed around the mean intensity of each basic emotion for a given picture, and a category membership was determined according to the overlap of the CIs. A single emotion category was ascribed to a given picture if the mean of one emotion was higher than the means of all of the other emotions, and if the CI for that emotion did not overlap with the CIs for the other five emotional categories. An image was classified as blended if two or three means were higher than the rest and if the CIs of those means overlapped only with each other. Finally, if the CIs of more than three means overlapped, such an image was classified as undifferentiated (Mikels et al., 2005).
The aforementioned procedure was used to find images that elicited one discrete emotion more than the others. As a result, 510 images used in the study were divided into six categories: happiness (n = 240), anger (n = 2), sadness (n = 62), fear (n = 11), disgust (n = 51), and surprise (n = 2), giving a total number of 368 pictures that were matched to specific basic emotions. The other pictures were classified as blended, including two (n = 21) or three (n = 22) emotions, or were classified as undifferentiated, eliciting similar amounts of four, five, or six emotions (n = 20, 25, and 54 pictures, respectively). Some example images from the animals category are presented in Fig. 3.
We computed a series of one-way analyses of variance solely on the pictures classified with the CI method (Mikels et al., 2005) as eliciting single basic emotions. For each group of pictures classified with a particular basic emotion, we compared the intensity ratings of this basic emotion in these pictures and in the pictures classified with all the other basic emotions. We obtained a significant effect of the basic-emotion classification in each case—namely, for happiness, F(5, 362) = 200.43, p < .001; sadness, F(5, 362) = 449.92, p < .001; fear, F(5, 362) = 147.10, p < .001; surprise, F(5, 362) = 44.19, p < .001; anger, F(5, 362) = 138.02, p < .001; and disgust, F(5, 362) = 350.14, p < .001.
The frequencies of each basic emotion among the pictures classified as single, blended, and undifferentiated basic emotions are presented in Fig. 4. It is noteworthy that the three panels of this figure cannot be compared with regard to the sums of the pictures, since in the middle and right panels the same image contributed to several bars, whereas the number of pictures equals the sum of the bars in the first panel. The bars should be interpreted only in terms of the single bars informing us how often a particular emotion was represented as single, blended, or undifferentiated.
In order to provide researchers with an overview of the groups of pictures distinguished with the CI classification method, descriptive statistics for the basic emotions and affective dimensions are presented in Table 1.
As was mentioned in previous studies (e.g., Mikels et al., 2005), alternative methods could be used to investigate the data. For instance, the CI method would classify images rated by one discrete emotion as having significantly higher ratings than the others, even though the intensity of this single rating was lower than those for other images that elicit blended or undifferentiated emotions. Following this, we provide a conservative classification method (Briesemeister et al., 2011b), according to which pictures were assigned to a specific discrete emotion category if the mean rating in one discrete emotion was more than one standard deviation higher than the ratings for other discrete emotions. Finally, the most liberal classification criterion was applied (Briesemeister et al., 2011b), according to which all of the pictures that received a higher mean rating in a particular discrete emotion were labeled as being related to this emotion. The results of all three classification methods are presented in Table 2.
Since all of the methods of classification are based on means and CIs, the picture classifications of our data did not differ substantially across the three methods described above. No pictures were classified with different basic emotions according to the different methods. The only difference was the obtained numbers of pictures classified as expressing specific basic emotions. Table S2 includes the results of each classification method for each single picture.
Relationship between basic emotions and affective dimensions
An exploration of the relationships between the basic emotions and affective dimensions showed that these variables were highly intercorrelated, as is demonstrated in Table 3.
Additionally, regression analyses were computed using the discrete emotional category ratings in order to examine the extent to which these variables could predict the ratings of valence and arousal (Bradley & Lang, 1999). We performed four separate analyses using the six emotional category ratings to predict valence and arousal within the three valence classes distinguished in the previous sections (negative, neutral, and positive), in line with analyses reported in literature (Montefinese et al., 2014; Stevenson et al., 2007; Stevenson & James, 2008).
After removing the insignificant coefficients, we repeated the regressions; all four models turned out to fit the data, and the basic-emotion intensities explained a large percentage of the variance of valence [F(5, 142) = 172.41, p < .001, R
2 = .86, for negative pictures; F(5, 197) = 413.99, p < .001, R
2 = .91, for neutral pictures; and F(3, 155) = 182.33, p < .001, R
2 = .78, for positive pictures] and of arousal [F(6, 141) = 93.47, p < .001, R
2 = .80, for negative pictures; F(6, 196) = 157.52, p < .001, R
2 = .83, for neutral pictures; and F(6, 152) = 59.14, p < .001, R
2 = .70, for positive pictures].
Standardized β coefficients were calculated for all six emotional categories. As for negative pictures, valence was strongly related to sadness, disgust, happiness, fear, and anger, yet it was not related to surprise. Arousal, in turn, was related to fear, disgust, and sadness, but not to anger and surprise. In the case of neutral pictures, valence was most strongly related to happiness, sadness, disgust, and fear, and additionally to surprise, but not to anger. Arousal was also not related to anger, yet it was related to fear, happiness, sadness, disgust, and surprise. As far as positive pictures were concerned, valence was related to happiness, sadness, and disgust only. Arousal was related to fear, disgust, anger, and sadness (but only fear was significant).
However, partial correlations (representing the unique influence of one predictor relative to the part of the variance of a dependent variable unexplained by the other predictors) revealed that discrete emotions contributed to valence and arousal in different ways (Ric, Alexopoulos, Muller, & Aubé, 2013) (Table 4). The ratings of affective dimensions were predicted particularly well by the level of happiness among positive pictures; by the levels of happiness, sadness, and fear among neutral pictures; and by the levels of sadness, fear, and disgust among negative pictures. The distribution of the ratings of pictures classified as eliciting particular discrete emotions on the basis of the CI criterion is presented in the affective space of valence and arousal in Fig. 5.
The regressions calculated using the dimensional ratings to predict emotional category ratings were similar to the previous ones, also showing a lack of homogeneity in their relationships (beta weights and a statistical analysis are presented in Table S1 in the supplementary materials).
Relations between the affective variables and the content categories
Subsequently, we performed a MANOVA including the five Content Categories (animals, faces, landscapes, objects, and people) and the three classes of Picture Valence (negative, neutral, and positive) as between-object factors, and the ratings of the six basic emotions intensities as well as the ratings of the two affective dimensions as dependent variables. Before that, we tested the assumption of the absence of multicollinearity between the dependent variables. The variance inflation factor (VIF) showed that multicollinearity might be a problem (Myers, 1990) for valence (VIF = 17.56) and happiness (VIF = 11.29). Therefore, we removed valence as a dependent variable from the analysis. Additionally, conducting collinearity diagnostics checked for interdependence of the independent variables. The obtained tolerance and VIF values were not considered problematic (tolerance > 10 and VIF < 10; Myers, 1990).
As for the between-object effects, we found significant main effects of content category [F(28, 1968) = 7.99, p < .001, η
2 = .10] and valence class [F(14, 980) = 121.25, p < .001, η
2 = .63], as well as a significant effect of the interaction between the two [F(56, 3465) = 4.73, p < .001, η
2 = .07]. Further analysis of this interaction showed interesting patterns specific to each basic emotion. This interaction was further interpreted through an analysis of the simple main effects of content category performed separately for each valence class, and the results are depicted in Fig. 6. There were significant differences in the mean basic-emotion intensities among the pictures of different valence classes, depending on their content category. To start listing all of them, the ratings of happiness were lower for objects than for landscapes for both neutral and positive pictures. As far as sadness was concerned, among negative pictures the ratings were significantly higher for faces and lower for objects than for the other categories. The ratings of fear were higher than those of the other categories for people (among both negative and neutral pictures) as well as animals (among neutral pictures). As for surprise, among positive pictures these ratings were lower for animals and people than among the neutral pictures. Anger among the negative pictures was rated significantly higher for landscapes than for the other categories. Finally, disgust among the negative and neutral pictures was rated higher for objects and lower for faces than for the other content categories. All of the significant differences (p < .05) are marked with an asterisk in Fig. 6.