Introduction

The correct interpretation of emotion expressions is essential for constructive social interaction. Smiles facilitate social evaluations such as inferences of personal traits (Krumhuber et al., 2007; Reis et al., 1990), and provide insights into others’ motives and affective states (Deutsch et al., 1987; Kappas et al., 2013). Smiles that are direct readouts of an underlying positive emotional state (e.g., pleasure, enjoyment) have been referred to as authentic, genuine, and spontaneous (Ekman & Friesen, 1982). A smile is typically perceived as spontaneous when it involves the Duchenne marker (Ekman et al., 1990): a lifting of the cheeks, narrowing of the eye opening, and wrinkles around the eye socket, colloquially called crow’s feet (Duchenne, 1862/1990). However, not all smiles are readouts of positive internal states. Smiles can also be employed for the purpose to establish and maintain positive social interactions with others (Ekman, 1989; Rychlowska et al., 2017). These more affiliative displays, often shown to reassure others, are typically described as posed smiles. Posed smiles are deliberately posed displays produced in the absence of an underlying affect, hence are unrelated to a positive emotional experience. People smile frequently (Frank & Ekman, 1993), thereby showing posed smiles in a wide variety of everyday social situations, ranging from polite interactions with strangers to collaborative tasks involving trust and cooperation (Frank, 1988).

Studies have shown that laypeople differentiate between spontaneous and posed smiles, with most of the existing work focusing on perceptual-descriptive features of the expression, particularly the Duchenne marker (e.g., Frank et al., 1993; Krumhuber & Manstead, 2009). Stimulus features (e.g., facial markers) are important for deciphering the emotional meaning of a display, i.e., when deciding with whom to cooperate or share personal resources (Johnston et al., 2010; Krumhuber et al., 2007). For example, there is converging evidence that the Duchenne marker results in favorable emotional and interpersonal responses (e.g., Frank et al., 1993; Harker & Keltner, 2001; Krumhuber & Manstead, 2009). However, contextual cues and prior knowledge about the situation of the expresser may similarly guide the extraction of information from visual objects such as faces (Sowden & Schyns, 2006). According to the conceptual act model of emotion (Barrett, 2006; Barrett et al., 2007), information about the social context is combined with facial information at early stages of stimulus processing. In this vein, emotion perception is an active inferential process in which conceptual knowledge about the social situation is used to surmise the specific meaning of a display. Context in the form of visual or verbal scenarios may therefore be inherent to how we perceive facial expressions. Supportive evidence comes from neuroimaging studies pointing towards the automatic binding of both informational sources during the structural encoding of expressions (Righart & De Gelder, 2008), memory consolidation and retrieval (Hayes et al., 2010; for a review see Wieser & Brosch, 2012).

Early behavioral work demonstrated that social context is as influential as facial information in emotion perception (Goodenough & Tinker, 1931; Munn, 1940), often overriding the visual information of the expression itself (e.g., Carroll & Russell, 1996; Wallbott, 1988). More recent studies suggest that context can make identical facial displays to be labelled as different emotions. For example, Kayyal et al. (2015) found that athletes’ expressions were judged as more positive (e.g., happy) or negative (e.g., sad or angry) depending on whether they supposedly won or lost a race. In a study by Koji and Fernandes (2010), valence attributions of neutral and emotional expressions significantly changed with the type of visual background scenes. While the effect of context has been consistently demonstrated (for reviews, see e.g. Aviezer et al., 2017; Gendron et al., 2013; Greenaway et al., 2018), in the majority of research the recognition of emotion categories has been the primary focus of interest. This is surprising given that emotions seldom involve clear-cut signals but are typically expressed in subtle and non-prototypical forms (Parkinson, 2013).

Until now, little evidence is available as to how contextual information influences the interpretation of emotions, such as whether an expression appears to be spontaneous or posed. To our knowledge, only two studies have been directly concerned with this question, showing that authenticity ratings of smiles varied with the situational description of the context scenario (Maringer et al., 2011; Mui et al., 2020). Interestingly, Maringer et al. (2011) found that this effect only applied when mimicry of the facial expression was inhibited, thereby making the smile interpretation more challenging. Mui et al. (2020) showed that smiles paired with a positive situation were rated as more genuine than those in a negative situation, but not more genuine than smiles presented without context. In both studies, however, smile displays were relatively intense, involving the Duchenne marker which may itself function as a perceptual-descriptive feature of expression genuineness (see Orlowska et al., 2020, for further discussion). Unless expressive information is complex or difficult to encode, perceivers may not be contextually sensitive when evaluating emotion authenticity (Aviezer et al., 2008; Wallbott, 1988). The present research aims to employ posed smiles due to their subtle and ambiguous nature, and test whether social context significantly affects their perceived emotional meaning.

Besides the interpretation of emotion, we investigate the role of context in the recognition memory of smile expressions. Situation-specific information may be particularly likely to be remembered when the goal is to perceive emotion (Barrett & Kensinger, 2010), thereby biasing how visual features of the expression are stored and retrieved (Hugenberg & Sacco, 2008). For example, Woll and Martinez (1982) showed that the presence of an emotion label led to biased recollection of facial expressions in the direction of the label. Using emotional background music, Woloszyn and Ewert (2012) found that happy displays were more likely to be recalled as sad ones when sad music was previously paired with the face. Given that emotions are often represented as categorical prototypes in people’s mind (Shaver et al., 1987), it is possible that (ambiguous) facial expressions are perceptually assimilated to those categories (Hugenberg et al., 2011). Providing contextual information could contribute to categorical assimilation effects in which facial expressions are recollected as more typical of the emotion category indicated by the context. In support of this notion, Halberstadt and Niedenthal (2001) showed that emotion concepts (i.e., anger) used for explaining ambiguous target expressions (i.e., angry-happy blends) led participants to subsequently remember the faces as more angry in appearance than they actually were. While these findings point toward categorical influences on recognition memory, no research to date exists in the domain of emotion authenticity. Smiles with the Duchenne marker may function as iconic signs or idealized prototypes of spontaneous enjoyment. When testing for context-based memory distortions, assimilation effects could make people recall the features of an expression in accordance with their beliefs of what a spontaneous smile looks like. In other words, smiles might be misremembered as having more of the physical attributes (i.e., Duchenne marker) associated with spontaneous enjoyment when such inference is implied by the context.

The Present Research

Until now, research on social context and facial expressions has been concerned mainly with the recognition of emotion categories, i.e. the classification of a display as belonging to one emotion or another. Whilst some empirical evidence suggests a possible role for context in emotion interpretation (Maringer et al., 2011; Mui et al., 2020), prior studies used relatively intense and unambiguous displays of smile expressions, which may itself convey strong emotional meaning. The present work aims to fill that gap by employing posed smiles, thereby investigating how context affects the interpretation of facial displays within an emotion family (i.e., enjoyment). Moreover, we seek to explore whether contextual biases occur at the recall stage of emotional faces.

To this end, participants viewed smile expressions presented together with a happy or polite social context scenario, and rated the degree to which the expresser “wanted to be nice and express positive intentions” or “was feeling happy and content”. After a filler task, recognition memory was assessed through an unexpected face recollection test, in which participants had to accurately select the original expression among several foils that consisted of morphed smile images with varying degrees of the Duchenne marker. We expected to confirm former findings that contextual knowledge regarding the given social situation exerts a significant influence on emotion perception and recognition memory (e.g., Halberstadt & Niedenthal, 2001; Kayyal et al., 2015). Specifically, smiles seen in a happy context should be judged as more spontaneous than the same smiles presented in a polite context. Also, smiles should be misremembered as having more of the physical attributes (i.e., Duchenne marker) associated with spontaneous enjoyment when they appear in the happy than polite context condition.

Method

Participants and Design

Ninety-six participants (68 women, 28 men), aged 18–34 years (Mage = 21.15, SD = 2.71), were recruited at the university campus and completed the study without remuneration. Only White participants were chosen since the target faces were of this ethnicity, thereby avoiding potential cross-race effects (Krumhuber et al., 2015). A priori power analysis using G*Power 3.1 (Faul et al., 2007) indicated that this sample size was sufficient to detect a medium-sized effect of context (Cohen’s d = 0.58) with 80% statistical power (α = 0.05). The experiment had a one-factorial between-subject design, with participants being randomly allocated to either the polite (n = 48) or happy (n = 48) context condition. Ethical approval was granted by the departmental ethics committee, and subjects provided written informed consent prior to participation.

Stimulus Material

The facial stimuli featured 12 White women with direct gaze and in frontal view. Each stimulus identity expressed a deliberately posed non-enjoyment smile unrelated to a positive emotional experience (i.e., produced upon instructions to pose a smile), and a spontaneous enjoyment smile that occurred as part of a positive emotional experience (i.e., while viewing pleasurable emotive pictures). The stimuli were taken from the image sets developed and validated by Johnston et al. (2010) and McLellan et al. (2010). In morphological terms (Facial Action Coding System, Ekman et al., 2002), spontaneous enjoyment smiles corresponded to the description of a smile with the Duchenne marker (Frank et al., 1993) in that they involved the Lip Corner Puller (Action Unit 12) and Cheek Raiser (AU6). In contrast, posed smiles did not involve Action Unit 6.

To create intermediate images along the visual continuum, we morphed in 25% increments between the posed and spontaneous smiles of the same stimulus identity using FantaMorph software (©Abrosoft). This resulted in five images for each target face: the original posed smile, three morphed smiles with varying levels of the Duchenne marker (25%, 50%, 75%), and the original spontaneous smile. For the sake of simplicity, we re-labelled the face continuum using a scale from 0% (spontaneous smile) to 100% (posed smile). All stimuli were displayed in colour on a plain background and measured 360 × 480 pixels.

The context information consisted of two verbal context labels, describing either a polite (“smiling when asking a stranger for the time”) or happy situation (“smiling upon hearing that she will be able to graduate”). Pilot-testing with a separate group of participants (N = 66) confirmed that the polite context was judged as more indicative of a situation in which a posed smile would occur (M = 80.98, SD = 23.29, 0–100 scale) compared to the happy context description (M = 19.64, SD = 24.96), t(65) = 22.77, p < .001, d = 2.80.

Procedure

Participants were tested individually on computers running Qualtrics, a web-based software (Provo, UT). Upon arrival, they were informed that the purpose of the study was to prepare materials for people who would like to improve their social skills. Their task was to rate smiles, as making appropriate facial expressions can greatly contribute to one’s success in social relations. The following brief definitions of each smile type, informed by previous research (Ekman & Friesen, 1982; Frank & Ekman, 1993), were provided: (a) spontaneous smile: “a smile displayed when someone is happy or amused and is truly feeling the emotion”, b) posed smile: “a smile that is intentional in the sense that someone wants to be nice and express positive intentions but does not feel the respective emotion”. For the manipulation of social context, half the participants were informed that the images they would see featured people who were “smiling when asking a stranger for the time” (polite condition). The other half was told that the person in the image was “smiling upon hearing that she will be able to graduate” (happy condition).

At the beginning of the experiment, with the aim to measure a-priori beliefs about the meaning of a smile expression, participants answered questions about the likelihood of a spontaneous or posed smile being shown in a specific context. For this, they were asked to imagine a person smiling “when asking a stranger for the time” or “upon hearing that she will be able to graduate”, consistent with the condition to which they had been assigned. They then rated the extent to which they think that the imagined smile would communicate that the person “wants to be nice and express positive intentions (posed smile)” (from 0 to 100%) and “is feeling happy and content (spontaneous smile)” (from 0 to 100%). If participants thought that one answer category fully applies, they could choose 100% for that answer category and 0% for the other. If they thought that both answer categories would fit that question, they could choose the ratings for each category. Ratings on the two dimensions were complementary and had to sum up to 100%. For the sake of simplicity, we focus in all analyses on the “feeling happy and content” dimension, with higher scores reflecting greater perceived smile spontaneity.

The main smile evaluation task involved the presentation of posed smiles by 12 face identities in combination with one of the two social context descriptions (polite or happy). The context information was shown at the top of the screen together with each stimulus (see Fig. 1a). Presentation order was randomized, with every smile expression being displayed for 3 s. Following each stimulus, participants judged the extent to which the smile would communicate that the person “wanted to be nice and express positive intentions (posed smile)” (from 0 to 100%) and “was feeling happy and content (spontaneous smile)” (from 0 to 100%), with complementary ratings on both scales again. Similar to the first task, if participants thought that one answer category fully applies, they could choose 100% for that answer category and 0% for the other. If they thought that both answer categories would fit that question, they could choose the ratings for each category. All further analyses focus on the “feeling happy and content” dimension, with higher scores reflecting greater perceived smile spontaneity.

Fig. 1
figure 1

a Example of face-context stimulus featuring a posed smile in combination with a polite or happy context label (between-subjects). Image courtesy of Tracey McLellan. b Mean spontaneity ratings of the smile expression as a function of social context. Higher scores indicate greater levels of perceived spontaneity. Error bars indicate standard errors of the means

Full size image

Next, participants engaged in a 5–8 min filler task which required them to indicate their mood on a 7-point Likert scale (1 - very negative, 7 - very positive) as well as to complete the 40-item Empathy Quotient (EQ. 40; Baron-Cohen & Wheelwright, 2004). The EQ. 40 is a self-report questionnaire designed to assess empathy in a range of social situations. Using a 4-point Likert scale, participants judge how strongly they agree or disagree with each of the 40 statements, such as: “I find it easy to put myself in somebody else’s shoes” or “I am good at predicting how someone will feel.” To control for various constructs, mood and empathy were included as covariates in the analyses.

The filler task was followed by an unexpected face recollection test. Participants were told that the facial displays seen earlier will be presented along several other smile expressions by the same face identity. On every trial, they saw the posed smile from the first part of the experiment, alongside four distractor stimuli of the same person (250 × 310 pixels): a spontaneous enjoyment smile and three morphed smiles with varying levels of the Duchenne marker (25%, 50%, 75%; see Fig. 2). Participants’ task was to accurately select the original image out of five images (display resolution: 250 × 310 pixels) presented simultaneously in a scattered random order on the screen (for a similar method, see Epley & Whitchurch, 2008; Walker & Keller, 2019; Wang et al., 2018). Thus, posed smiles were always the correct response, whereas spontaneous and morphed smiles were foils. Both the target and distractor images remained on the screen until participants submitted their response. Only one image could be selected, and no feedback was provided. Trial and image order were randomized for each of the 12 stimulus identities. Finally, participants were thanked and debriefed.

Fig. 2
figure 2

Example of image response options from the face recollection test. Images ranged from 0% (spontaneous smile) to 100% (posed smile), including morphed images with varying levels of the Duchenne marker (25%, 50%, 75%). Image order was randomized in the experiment, and neither spontaneous/posed labels nor morph percentage levels were shown to participants

Full size image

Results

We first investigated participants’ a-priori beliefs regarding the likely occurrence of a spontaneous or posed smile in the specific contexts. An independent samples t-test revealed that participants expected smiles to be more spontaneous in their emotional meaning when they appeared in the happy context (M = 90.54, SD = 13.84) than the polite context condition (M = 19.12, SD = 17.72), t(94) = −22.00, p < .001, 95% CI [−77.86, −64.97], d = −4.49.

Next, spontaneity ratings and memory scores were analyzed with mixed-effects models including random intercepts for stimulus identity and participants using the lmer 4 package (Bates et al., 2015) as part of the R environment (R3.4.3. GUI 1.70). This allowed us to estimate the fixed coefficient by directly modelling the shared variance that arises among participants as well as the stimulus identities. Results revealed that spontaneity ratings were significantly higher for smiles seen in the happy context (M = 48.66, SD = 13.04) than in the polite context condition (M = 33.93, SD = 15.22), β = 14.73, SE = 2.89, 95% CI [9.03, 20.43], t(94) = 5.09, p < .001, d = 1.04 (see Fig. 1b). Moreover, memory for and identification of previously seen faces was biased in a context specific manner. Specifically, smiles were misremembered as being more spontaneous in their physical appearance (i.e., containing more of the Duchenne marker) when they had been viewed in the happy context (M = 63.28, SD = 15.90) than the polite context condition (M = 69.57, SD = 13.03), β = −6.29, SE = 2.97, 95% CI [−12.11, −0.47], t(94) = −2.12, p = .037, d = 0.43.

There was no significant difference between participants in the polite and happy context condition for measures of mood (Mpolite = 4.10, SD = 1.43 vs. Mhappy = 4.33, SD = 1.34), t(94) = −0.81, p = .421, 95% CI [−.79, .33], d = 0.17, and empathy (Mpolite = 48.19, SD = 11.08 vs. Mhappy = 43.98, SD = 11.42), t(94) = 1.83, p = .070, 95% CI [−.35, 8.77], d = 0.37. Adding mood and empathy as covariates in an ANCOVA did not significantly change the pattern of results. Also, there were no effects of the covariates on a-priori beliefs, spontaneity ratings, and recognition memory: mood, Fs < 3.90, ps > .051; empathy, Fs < 2.39, ps > .125.

Discussion

Smiles can have a variety of meanings which make them complex in interpretation (Krumhuber & Skora, 2016). Beyond the perceptual analysis of the expression, prior knowledge about the situation may help to disambiguate a sender’s message. In real life, emotion expressions are never perceived in isolation, as faces are typically seen attached to a body and against the backdrop of some scenery (Aviezer et al., 2008, 2017; Kret et al., 2013; Lecker et al., 2020). Previous research has shown that considering this contextual information is critical for accurately classifying emotion displays, i.e. whether they belong to one emotion category or another (Aviezer et al., 2017; Gendron et al., 2013).

In this study, we investigated whether social context influences the interpretation of smile expressions and the way smiles are visually represented in memory. Unlike other emotion categories which are well demarcated theoretically in one’s mind, there is a conceptual overlap among smile types as positive facial displays (Abel, 2002). We focused on posed smiles, known for their ambiguous meaning, and presented them alongside a polite or happy context description. In line with predictions, participants expected more spontaneous smiles when the context implied a happy situation. Moreover, spontaneity ratings were significantly higher for smiles seen in the happy than polite context condition. The findings replicate and extend those of previous research (Maringer et al., 2011; Mui et al., 2020), showing that situational descriptions can fundamentally change how an expression is evaluated. By using posed smiles, facial information in the form of visually discriminative features of emotion genuineness (i.e., Duchenne marker) was limited. This allowed for contextual information to play a moderating role in determining whether the smile was perceived as direct readout of internal states or voluntary signal displayed for social and communicative purposes.

Besides affecting emotion interpretation, we found that knowledge about the situation led to biased recall of smile expressions. Specifically, smiles were likely to be misremembered as spontaneous smiles with a Duchenne marker when they had been previously paired with a happy (compared to a polite) context scenario. As suggested by past research, social factors can provoke assimilation of a face to a categorical prototype, with people’s memory being distorted towards the more typical exemplar of the category (Hugenberg & Wilson, 2013; Wang et al., 2018). Halberstadt and Niedenthal (2001) demonstrated categorical assimilation effects in the recognition of emotions, such that ambiguous expressions (i.e., angry-happy blends) conceptualized in terms of anger were recollected as physically angrier than they actually were. The present study is to our knowledge the first in showing that smile expressions viewed in combination with a category label (i.e., spontaneous enjoyment) can lead to the activation of physical attributes associated with that category (i.e., Duchenne marker), thereby influencing the memory retrieval of respective expression.

In this work, we wanted to measure mood, which can affect people’s veracity judgements (Forgas & East, 2008), and control for trait empathy. Whereas Dawel et al. (2015) showed that higher levels of empathy increased the ability to judge the authenticity of smile expressions, no such association was found by McLellan and McKinlay (2013). Here, the pattern of results remained unchanged, suggesting that the impact of social context was sufficiently stable when accounting for individual differences. Nonetheless, a promising field for future research would be the investigation of dispositional factors (i.e., emotional expressivity, personality traits, Fox & Zougkou, 2012; Kring et al., 1994) and their role in emotion interpretation. Toward this end, it might be interesting to test for gender differences. Past research, for example, indicates that target and participant gender exert a significant influence on how genuine vs. posed a smile expression appears to be (Krumhuber et al., 2007; McKeown et al., 2015). In this study, we kept the target gender constant; also, the majority of participants were female. However, both variables might interact in driving contextual effects.

Studying the impact of context on emotion perception is essential, as presenting participants with face stimuli lacking contextual information creates an experimental setting that has low ecological validity. In real life, observers process contextual information to accurately interpret the meaning of emotion displays, especially given that displays observed in naturalistic emotional situations are nuanced and include subtleties (Hyniewska et al., 2019) which make them challenging to univocally categorize per se (Hess et al., 1997; Sato et al., 2019).

Given the specificity of the task used in this research, the generalizability of our findings is limited to the clear-cut scenarios used and the format of our face stimuli. Further research is necessary regarding the perception and recall of stimuli that are more representative of social encounters. Knowing who produces a facial expression (Wang et al., 2018) and towards whom this expression is directed (Quadflieg & Penton-Voak, 2017), for example, can radically change our understanding of a communicative facial cue and how it is remembered. In addition, a single spontaneity rating scale may be used to measure participants’ perception of happy feelings versus positive intentions. At the moment, these were treated as two complementary scales which could lead to confusion in smile judgments.

While the present work targeted smile expressions, research is needed to explore whether the findings replicate with other emotion displays, i.e., sadness, fear, and surprise. There is evidence indicating that the role of context might vary between emotions (e.g., Carroll & Russell, 1996; Woll & Martinez, 1982). Furthermore, it is reasonable to expect that posed and spontaneous expressions are easier to discriminate for some emotions than others (McLellan et al., 2010). To this end, a detailed morphological and dynamic analysis of facial displays is essential for determining their relative contribution to emotion interpretation. Akin to most studies, we resorted to images of smile expressions due to their simplicity and the ability to control for the presence/absence of the Duchenne marker. Clearly, those static displays do not capture the realistic nature of facial expressions encountered in everyday life. Furthermore, they may impede coherence in the identification of facial affect compared to dynamic displays (Zloteanu et al., 2018; for a review see Krumhuber et al., 2013). It falls to future research to address those questions. By going beyond the recognition of emotion categories, the current results underline the importance of situational context in the interpretation and recall of facial expressions. We hope that this line of work contributes to a fuller understanding of the circumstances under which emotion displays are differentially judged and remembered.