Facial identity and emotional expression as predictors during economic decisions

Abstract

Two sources of information most relevant to guide social decision making are the cooperative tendencies associated with different people and their facial emotional displays. This electrophysiological experiment aimed to study how the use of personal identity and emotional expressions as cues impacts different stages of face processing and their potential isolated or interactive processing. Participants played a modified trust game with 8 different alleged partners, and in separate blocks either the identity or the emotions carried information regarding potential trial outcomes (win or loss). Behaviorally, participants were faster to make decisions based on identity compared to emotional expressions. Also, ignored (nonpredictive) emotions interfered with decisions based on identity in trials where these sources of information conflicted. Electrophysiological results showed that expectations based on emotions modulated processing earlier in time than those based on identity. Whereas emotion modulated the central N1 and VPP potentials, identity judgments heightened the amplitude of the N2 and P3b. In addition, the conflict that ignored emotions generated was reflected on the N170 and P3b potentials. Overall, our results indicate that using identity or emotional cues to predict cooperation tendencies recruits dissociable neural circuits from an early point in time, and that both sources of information generate early and late interactive patterns.

Faces represent complex visual patterns that convey a wide array of information about individuals, such as their identity, emotional expression, sex, and age (e.g., Bruce & Young, 1986; Palermo & Rhodes, 2007; Pizzagalli et al., 2002). Among these invariant and changeable facial properties, facial identity and emotional expression are essential to meet social demands (Frijda, 1988; Frijda & Mesquita, 1994; Todorov, Said, Engell, & Oosterhof, 2008). Under this rationale, the aim of this study was to employ electroencephalographic measurements to evaluate how the strategic use of identity and emotional expression as relevant cues for guiding economic decisions influences different stages of information processing and whether these two sources of information are analyzed in an independent or interactive manner.

Models of face perception indicate that identity and emotion are processed by relatively separate brain regions (Bruce & Young, 1986; Calder, 2011; Calder & Young, 2005; Haxby, Hoffman, & Gobbini, 2000). The ability to extract information from faces is supported by a specialized and distributed neural network composed by a core system of visual extrastriate areas that acts in concert with complementary regions of an extended system (Haxby & Gobbini, 2011; Haxby et al., 2000). There are, however, discrepancies as to whether these properties are processed by fully separated routes or whether they interact to some extent (e.g., Baudouin, Gilibert, Sansone, & Tiberghien, 2000; Calder, 2011; Stolier & Freeman, 2016; see also Calder & Young, 2005). Haxby and colleagues (2000) proposed that perceptual operations on invariant and changeable facial properties are anatomically segregated, given their potential to interfere with each other (Haxby & Gobbini, 2011). In a complementary way, Calder (2011) proposed, based on principal components techniques, that face decoding may depend on a ventral temporal route (which includes the fusiform gyrus) engaged in the analysis of visual forms associated to both identity and expression, and an additional separate route (including the superior temporal sulcus) involved in the processing of dynamic facial information.

Some studies have provided evidence supporting the mutual influence of facial properties under certain circumstances. For example, Schweinberger and Soukup (1998) investigated whether variations in irrelevant stimulus dimensions influenced judgments about identity, emotional expression, and facial speech. They showed that identity judgments were unaffected by the other two dimensions, but emotion and facial speech judgments were influenced by identity. Hence, identity may modulate the decoding of emotion whereas emotion per se may not change the way in which identity is processed (Schweinberger & Soukup, 1998). In contrast, Baudouin et al. (2000) showed that emotional expression influenced familiarity judgments of faces. Smiling faces increased the degree of perceived familiarity for both famous and unknown identities (Baudouin et al., 2000). Also, Fisher, Towler, and Eimer (2016) recently observed, during a sequential matching procedure, that identity and emotional expression influence each other at the stage where perceptual face representations are matched with stored memories. In this line, Ganel and Goshen-Gottstein (2004) proposed a structural-reference hypothesis, in which the structure of a face (i.e., its identity) can be employed as a reference image to guide the decoding of idiosyncratic emotional expressions. In addition, recent discoveries from a series of fMRI experiments by Stolier and Freeman (2016) have shown that stereotypical and other categorical knowledge can bias face perception through top-down channels. This evidence, in sum, favors the intertwined nature of these social category representations, including emotional expressions (see also, Todorov, Mandisodza, Goren, & Hall, 2005; Todorov et al., 2008). Thus, personal identity, at least in part, is constructed using emotional content, which suggests some sort of interaction between invariant and changeable facial properties.

Understanding how identity and emotional expression are processed under the influence of each other is of paramount importance because they signal intentionality during interpersonal interactions (e.g., Frijda & Mesquita, 1994; Todorov et al., 2008; Tooby & Cosmides, 1990). Crucially, these sources may lead to contradictory predictions. For example, attributions of intentionality based on identity (e.g., someone is described or known as trustworthy) may create expectations of positive future outcomes (e.g., Oosterhof & Todorov, 2008), but if the same individual looks angry, then intentions naturally associated with emotions generate negative predictions (Frijda, 1988; Frijda & Mesquita, 1994). In this line, Tortosa, Strizhko, Capizzi, and Ruz (2013) showed that when emotions are explicitly employed to predict other people’s behavior, participants need less time to learn natural associations (e.g., “happiness–trustworthy”) than nonnatural ones (e.g., “happiness–untrustworthy”; see also Ruz & Tudela, 2011). In addition, when participants are required to attend to facial identity, defined as the relevant cue, while the emotional expression is explicitly described as nonpredictive, the automatic interference of the emotional content seems to be unavoidable despite its irrelevance (Alguacil, Tudela, & Ruz, 2015; Tortosa, Lupiáñez, & Ruz, 2013).

The evidence reviewed so far suggests that emotional information generates an unavoidable influence during identity judgments. Given the rapid processing of identity and emotional information (e.g., Barrett & Bliss-Moreau, 2009), this emotional influence could take place from the early visual inspection of facial elements. The description of the temporal dynamics of these processes is essential to their understanding (Batty & Taylor, 2003; Bentin, Allison, Puce, Perez, & McCarthy, 1996; Pizzagalli et al., 2002).

According to previous literature, a number of evoked potentials (ERP) could reflect the use of identity and emotion as cues to predict outcomes and also the interference between them. The earliest ERP responses to face stimuli appear approximately one hundred milliseconds after stimulus onset at the posterior bilateral P1 and the central N1 (Eimer & Holmes, 2002; Hilimire, Mienaltowski, Blanchard-Fields, & Corballis, 2014; Yang, Gu, Guo, & Qiu, 2011). At this level, previous research has reported early orientation of attention to low-level facial characteristics (black vs. white faces; e.g., Ito & Urland, 2003). Similarly, it has been suggested that these potentials reflect a fast global extraction of salient emotional information before fine-grained face decoding (e.g., Dennis, Malone, & Chen, 2009; Eimer & Holmes, 2002; Hilimire et al., 2014; Pitcher, Walsh, Yovel, & Duchaine, 2007; Pizzagalli et al., 2002; Pourtois, Dan, Grandjean, Sander, & Vuilleumier, 2005).

Of special interest are the N170, with posterior bilateral distribution, and the vertex positive potentials (VPP), with a central location, which appear around 170 milliseconds after face onset (Bentin et al., 1996; Rossion et al., 2000), although earlier timings have been reported (e.g., Rossion & Caharel, 2011). Both potentials, which seem to reflect similar mental operations (Eimer, 2011; Joyce & Rossion, 2005; Yang et al., 2011), have been linked to the decoding of configural facial properties (Eimer, 2011; Jeffreys, 1989; Joyce & Rossion, 2005). They are sensitive to the categorization of facial identity and also to emotional expressions (Eimer, 2011; Vuilleumier & Pourtois, 2007). Even more, emotional expressions affect their amplitude depending on the race of the individual (Tortosa, Lupiáñez, et al., 2013). A similar interaction effect has been found in an emotional face–word Stroop task in which participants responded to emotional expressions or word meaning (Zhu, Zhang, Wu, Luo, & Luo, 2010; see also Hinojosa, Mercado, & Carretié, 2015).

The frontocentral N2 has been related to cognitive control and conflict processing (e.g., Folstein & Van Petten, 2008; Kopp, Rist, & Mattler, 1996). Existing empirical evidence supports its sensitivity to conflict when nonemotional elements lead to incongruent representations (Folstein & Van Petten, 2008) and also to the affective background during the resolution of cognitive conflict (see, e.g., Kanske & Kotz, 2010a). In contexts of reward-outcome evaluation, the N2 potential is impacted by participants’ loss aversion when no reward is anticipated (e.g., Potts, 2011; see also Pornpattananangkul, & Nusslock, 2015). In the field of study of social cognition, other paradigms employing socially interactive contexts have not observed emotional conflict at the N2 (Alguacil, Tudela, & Ruz, 2013; Ruz, Madrid, & Tudela, 2013; Tortosa, Lupiáñez, et al., 2013). On the other hand, previous studies have shown that the N2 is also sensitive to the processing of social expectations extracted from identity (Derks, Stedehouder, & Ito, 2015; Tortosa, Lupiáñez, et al., 2013) and from expressed emotions (Derks et al., 2015; Tortosa, Lupiáñez, et al., 2013). Also, Ruz and colleagues (2013) found an interaction between executive attention and emotion, reflected in a heightened N2 amplitude for trustworthy identities expressing anger.

Finally, the P3b (Polich, 2007) is influenced by facial identities and emotions (see, e.g., Campanella et al., 2000, 2013; see also Campanella, Quinet, Bruyer, Crommelinck, & Guerit, 2002). Variations in its amplitude have been associated to the discrimination of facial expressions (Luo, Feng, He, Wang, & Luo, 2010; Tortosa, Lupiáñez, et al., 2013) and facial predictive identities of trustworthy or untrustworthy behaviors (Ruz et al., 2013). Its amplitude also changes with outcome evaluation and reward processing during economic exchanges, as, for example, in gambling tasks (Wu & Zhou, 2009). Stimuli with high motivational significance, like those that signal personal benefits, seem to enhance this potential (Moser, Gaertig, & Ruz, 2014; Ruz et al., 2013; Yeung & Sanfey, 2004). It is also usually larger for congruent than for incongruent trials (Neuhaus et al., 2010; Valle-Inclán, 1996).

In summary, although previous studies have explored how identity and emotion are decoded, these have mostly employed simple categorization tasks (Atkinson, Tipples, Burt, & Young, 2005; Fisher et al., 2016; Schweinberger & Soukup, 1998) or paradigms where only identity or emotion were the relevant elements for predicting outcomes (Alguacil et al., 2015; Ruz et al., 2013; Ruz & Tudela, 2011; Tortosa, Lupiáñez, et al., 2013). Furthermore, most studies to date have employed paradigms fully devoid of social context, which represents a drawback given the innate social nature of emotional phenomena (e.g., Parkinson, 1996). We are in constant interaction with other people, and in many cases we use social signals such as others’ identity and emotion to predict their likely behavior. Thus, the strategic use of these cues to obtain reward in a social context could further enhance their potential interactions along different processing stages. Thus, in this study we employed a modified version of the trust game, where identity and emotion were used as cues for predicting the prospective behavior of others. The trust game is particularly well suited for our purposes as individuals maximize reward by correctly anticipating their partners’ behavior (i.e., their trustworthiness). In our manipulation, during the game, participants earned money by predicting the most likely behavior of alleged partners, represented by facial photographs displaying emotional expressions. In separate blocks, they had to use the identity, the emotion or the color of a frame bordering the target pictures as cues to predict the most likely trial outcome (being the two other dimensions nonpredictive). This procedure allowed, in line with our goals, to contrast identity and emotional properties as cues for predicting future outcomes and making decisions during economic exchanges as well as to examine their potential interaction across different processing stages.

We predicted, at the behavioral level, that participants would mostly guide their decisions based on the predictive cues, and that they would need more time to decide when identity and ignored facial emotion led to incongruent predictions about outcomes in a trial (see Alguacil et al., 2015). At the electrophysiological level, we expected that attending to emotion would increase P1 and N1 amplitudes (Carretié, Hinojosa, Martín-Loeches, Mercado, & Tapia, 2004; Eimer, 2011; Vuilleumier & Pourtois, 2007). We also expected to observe an interaction between emotion and identity on the N170 potential reflecting incongruence, given its sensitivity to top-down control (Aranda, Madrid, Tudela, & Ruz, 2010; Tortosa, Lupiáñez, et al., 2013; Zhu et al., 2010; although see Fisher et al., 2016). We hypothesized that the N2 potential would be heightened for stimuli with a negative outcome prediction (e.g., Derks et al., 2015; Pornpattananangkul, & Nusslock, 2015; Potts, 2011; Ruz et al., 2013; Tortosa, Lupiáñez, et al., 2013), and that the P3b would be enhanced for motivationally salient information signaling positive outcomes (Moser, et al., 2014; Ruz et al., 2013; Wu & Zhou, 2009; Yeung & Sanfey, 2004), and reduced for incongruent situations (Neuhaus et al., 2010; Valle-Inclán, 1996) where expectations from identity and facial expression conflict.

Method

Participants

Twenty-eight healthy volunteers were recruited from the University of Granada in exchange for course credits. All of them had normal or corrected-to-normal vision. Sixteen were female, their mean age was 23 years (range: 18–40 years), and two of them were left-handed. After removing four females (see Electrophysiological Recording and Analysis section), the final sample was composed by 24 participants (12 female). All participants received payment according to their earnings during the experimental task (the sum of the winnings in three rounds randomly selected; ranging between 3.5 and 7.5 EUR), and they all signed a consent form approved by the local Ethics Committee.

Task

In separate blocks, participants played three different multiple-round adaptations of the trust game (Berg, Dickhaut, & McCabe, 1995): identity, emotion, and color. In all of them, participants received 1 EUR at the beginning of each trial. Then they decided whether to keep it (which resulted in 1 EUR payoff) or to bet it. If participants decided to bet, the initial sum was multiplied by five (5 EUR), and then a feedback symbol informed them of the win (2.5 EUR) or loss (0 EUR) as a result of the specific reward value linked to the target element that they responded to. In each block, different cues predicted the most likely trial outcome if the participant decided to bet the money.

In the identity block, two facial identities cued a likely win and the other two a likely loss. In the emotion block, the photographs of two happy and two angry faces cued a most likely win/loss, respectively. In the color block, the color of a frame on pixelated faces served as the cue for win or loss (see Fig. 1). The relevant cues were valid on 83% of the trials.¹ Irrelevant properties of each block (identity and color in the emotion block, emotion and color in the identity block, and identity and emotion in the color block) were nonpredictive (50% validity); therefore, participants were required to ignore them.

Fig. 1

Sequence of events in a trial (Color figure online)

Participants were instructed that the photographs of the faces in the game were alleged partners that behaved according to the patterns showed by real people in previous experiments that had been performed in collaboration with other universities. They were also asked to make their decision when the face was on the screen by pressing one of two buttons with their left or right index fingers.

Stimuli and procedure

Forty-eight faces (12 identities, six females) displaying happy or angry emotional expressions (half of them with the mouth closed) were taken from the NimStim set (Tottenham et al., 2009). Two different female and two male identities were used for each of the blocks (counterbalanced across participants). Adobe Photoshop CS 6 (Adobe Systems Inc., San Jose, CA) was employed to pixelate the original pictures (5 × 5 pixels) for the color block. The color, nonsocial, block included pixelated faces to provide a pure nonsocial condition where the automatic processing of facial properties could not interfere with judgments (e.g., Alguacil et al., 2015; Egner, Etkin, Gale, & Hirsch, 2008). However, data from the color block were not included in the final electrophysiological analysis (see Electrophysiological Recording and Analysis section).

Pictures were color framed in all blocks. In total, six colors were employed for this purpose, in two distinct tones each (dark and light blue, green, orange, purple, red, and yellow). Pairs of two different colors were used in each block (counterbalanced across participants).

The order of the blocks, the association between hand and response, the feedback symbols, and their color were fully counterbalanced across participants. The task was implemented using E-Prime 2.0 Professional software (Schneider, Eschman, & Zuccolotto, 2002). All stimuli were centrally displayed in a 17-inch CRT monitor, against a gray background (see Fig. 1). A trial started with a fixation cross (+, 0.5° on average) lasting 3350 ms on average (random 3100–3600 ms), followed by a framed picture of a face (7.15° on average) for 1500 ms, another fixation cross for 200 ms on average (random 100–300 ms), and finally the feedback symbol (*, 0.67°, or #, 0.57°) for 200 ms. On average, a trial lasted 5250 ms, and each participant completed 576 of them (192 for each of the three blocks) for a total of about 55 minutes. At the beginning of the session, participants responded to 60 practice trials (20 per block) where the cue was 100% predictive of the outcome. This minimal practice enabled us to train the participants with the specific associations between cues and likely outcomes and the procedure of the paradigm.

Electrophysiological recording and analysis

Participants were seated in front of the computer monitor in an electrically shielded room and were instructed to avoid eye blinks and movements during stimulus presentation and responses. EEG was recorded with a high-density 128-channel EEG system Geodesic Sensor Net (Tucker, 1994), referenced to the vertex channel. The head coverage included sensors lateral to and below both eyes to monitor horizontal and vertical eye movements (HEOG and VEOG). The EEG net was connected to an AC-coupled, high-input impedance amplifier (200 MΩ). At the beginning of the recording session, impedance was measured for each channel and was set below 50 kΩ, as recommended for Electrical Geodesics high-input impedance amplifiers. Amplified analog voltages (0.1–200-Hz band pass) were digitized at 1000 Hz² (12 bits A/D converter and 0.02 μV minimum resolvable voltage), and the continuous EEG was filtered offline using a 40-Hz low-pass filter. After that, the EEG was segmented 200 ms before and 800 ms after target onset and processed for artifact detection. Trials containing eye blinks or eye movements (electrooculogram channel differences greater than 70 μV) or more than 20% of bad channels were excluded. Data from bad channels were later replaced using a spherical interpolation algorithm (Perrin, Pernier, Bertrand, & Echallier, 1989). ERPs were re-referenced offline to the average. A 200-ms prestimulus interval was used as baseline.

A minimum criterion of 30 artifact-free trials per subject and condition was established to maintain an acceptable signal-to-noise ratio. On average, each condition included approximately 42 observations (range: 40–45). Voltage analyses were performed on the spatiotemporal windows that captured the grand-average peaks of the P1, central N1, N170, VPP, N2, and P3b potentials. The selected electrodes were those where the components of interest were maximally distributed (see Figs. 2, 4, 5, and 6) and also those in agreement with previous relevant literature (e.g., Eimer 2000, 2011; Luo et al., 2010; Pizzagalli et al., 2002; Polich, 2007; Pourtois et al., 2005; Rossion et al., 2000; Rossion, Joyce, Cottrell, & Tarr, 2003). The temporal windows employed were always centered on the peak of the potentials in the grand-average waveforms.

Fig. 2

Spatiotemporal distribution of electrodes used for the ERP analyses. LH = left hemisphere; RH = right hemisphere

Trials without a response were not considered either in the behavioral or ERP analyses (2.44%), and four (female) participants from the initial sample were excluded because of excessive artifacts during recording. Thus, behavioral and ERP analyses were carried out in a final group of 24 participants (12 female). An initial observation of the grand-average data revealed that the ERPs of the color block showed marked differences in latency and shape with the other two blocks, most likely due to the pixilation of the faces (see Fig. 1). Because of this, to reduce the complexity of the ERP analyses and given that our core hypothesis concerned the comparison of the emotion and identity blocks, data from the color block were only considered in the behavioral analysis.

Two separate analyses were carried out to address our research questions. The first one, which will be referred to as overall analysis, compared the key stages that varied according to whether personal identities or displays of emotion were used as cues to predict trial outcomes. To do this, the mean amplitude of face-locked ERPs, averaged over the selected channels and time windows, were submitted to repeated-measures ANOVA with Cue (identity vs. emotion) and Cued-Outcome (win vs. loss) as factors. Note that of the four conditions that participants received in the identity block (win/happy, win/angry, loss/happy, and loss/angry), two were not presented in the emotion block to avoid predictive conflict (win/angry and loss/happy; see Ruz & Tudela, 2011). These conditions had to be included in the identity block to make emotions nonpredictive and prevent the use of this cue to predict trial outcomes. Hence, to make data comparable across blocks, these two conditions were removed from the identity average. To equate the number of observations across conditions, 50% of the trials of the emotion block were randomly removed (for a total of 42 trials on average).

The second analysis, labeled as emotional conflict analysis, sought to evaluate the stages of processing affected by emotional conflict during identity judgments (see Alguacil et al., 2015). This set of ANOVAs employed the same electrode montages and time windows as the previous analysis, but only with data from the identity block (in which emotions were ignored). Factors entered were Cued-Outcome (win vs. loss) and Facial Expression (happy vs. angry). To avoid unnecessary duplication in the report of results, for this analyses we focused on the interaction between the Cued-Outcome and Facial Expression factors, so no other effects will be reported.

Given the relevance of lateralization for face processing, as reported by previous evidence (e.g., Campanella et al., 2000; Luo et al., 2010; Pizzagalli et al., 2002; Wager, Phan, Liberzon, & Taylor, 2003), Hemisphere was included as relevant factor for the P1, the N170, and the P3b potentials in both analyses (see Fig. 2).

Results

Behavioral

On average, participants betted their money on 49% of the trials. As revealed by the overall analysis, participants’ mean bet rates yielded a main effect of Cued-Outcome, F(1, 23) = 442.41, p < .001, η_p ² = .95, as they risked more money when they anticipated win (M = 0.88, SE = 0.28) than when they expected loss (M = 0.10, SE = 0.16). For the emotional conflict analysis, besides the Cued-Outcome effect, the effect of Facial Expression approached significance, F(1, 23) = 3.921, p = .06, η_p ² = .146, with a tendency for participants to bet money slightly more frequently with happy (M = 0.51, SE = 0.02) than with angry partners (M = 0.48, SE = 0.01).

The overall analysis³ of the participants’ mean reaction times (RTs) revealed a main effect of the Cue, F(1, 23) = 22.200, p < .001, η_p ² = .67, as participants were slower responding in the emotion (M = 739.87 ms, SE=20.05) than in the identity block (M = 701.11 ms, SE = 17.63) or in the color one (M = 638.17 ms, SE = 15.81), F(1, 23) = 9.665, p < .01, η_p ² = .30, and F(1, 23) = 45.548, p < .001, η_p ² = .66, respectively. The difference between the identity and the color blocks also reached significance, F(1, 23) = 23.553, p < .001, η_p ² = .51. In addition, responses were slower for expectations of losses (M = 705.33, SE = 15.73) than of winnings (M = 680.76, SE = 17.28), F(1, 23) = 11.21, p < .01, η_p ² = .33. No other main or interaction effect was significant (all Fs < 1).

The ANOVA for the emotional conflict on the mean RTs of the identity block showed the expected interaction between Cued-Outcome and Facial Expression, F(1, 23) = 13.42, p = .001, η_p ² = .37. Planned comparisons showed that participants were slower to respond to a partner whose identity cued likely win but that displayed anger (M = 699.34, SE = 19.52) than when the same identity showed an expression of happiness (M = 675.05, SE = 17.86). No other main or interaction effect was significant (all ps > .290; see Fig. 3).

Fig. 3

Mean reaction times: emotional conflict when participants predicted win or loss based on facial identity. Error bars represent standard error of the mean

Electrophysiological

P1.

The P1 peaked at around 100 ms and was analyzed from 80 to 120 ms over bilateral posterior electrodes (see Figs. 2 and 4). The overall ANOVA yielded a main effect of Hemisphere, F(1, 23) = 5.018, p < .05, η_p ² = .18, as this potential was more positive at right (2.17 μV; SE = 0.36) than at left electrodes (1.50 μV; SE = 0.30). The interaction between Cue and Hemisphere was close to significance, F(1, 23) = 4.212, p = .052, η_p ² = .15. The Hemisphere was significant for the emotion block, F(1, 23) = 7.553, p < .05, η_p ² = .25, with larger positive voltages over right (2.28 μV; SE = 0.36) than over left electrodes (1.43 μV; SE = 0.31). This effect was not significant for the identity block (F = 2.365, p = .138).

Fig. 4

Face-locked ERPs showing the modulation of the N170 potential by the effect of the cue at the left hemisphere (a: top panel) and by the facial expression on the identity block for faces that predicted win (b: bottom panel). Results for the P1 potential are not displayed in this figure, because the electrodes included in the analyses were different for the two potentials. The spatial window employed for the analyses is represented in the upper left diagram. Positivity is plotted upwards in all figures

The interaction between Cued-Outcome and Facial Expression in the emotional conflict ANOVA was not significant (F < 1).

N1.

The N1 over mediocentral electrodes peaked at 100 ms and was analyzed from 80 to 120 ms (see Figs. 2 and 5). The overall ANOVA showed a significant interaction between the Cued-Outcome and the Cue factors, F(1, 23) = 9.083, p < .01, η_p ² = .28. Planned contrasts revealed an effect of Cued-Outcome close to significance for the emotion block, F(1, 23) = 3.794, p = .06, η_p ² = .14. The N1 showed a tendency to be more negative for happy faces, which cued win (-1.11 μV; SE = 0.20), than for angry ones, associated with likely loss (-0.82 μV; SE = 0.20). The same contrasts were not significant in the identity block (F = 2.447, p = .131).

The interaction between Cued-Outcome and Facial Expression in the emotional conflict ANOVA was not significant (F < 1).

N170.

This potential peaked approximately at 150 ms after face onset and was analyzed over posterior bilateral electrodes (see Figs. 2 and 4a) from 125 to 175 ms. The overall ANOVA revealed a significant interaction between Cue and Hemisphere, F(1, 23) = 5.931, p < .05, η_p ² = .20. Differences in amplitude between the identity and the emotion blocks were significant for the left, (identity: -2.28 μV; SE = .40; emotion: -2.59 μV; SE = 0.85), F(1, 23) = 6.031, p < .05, η_p ² = .21, but not for the right hemisphere (F < 1).

The emotional conflict analysis revealed a significant interaction between Cued-Outcome and Facial Expression, F(1, 23) = 7.217, p < .05, η_p ² = .24. Subsequent analysis showed that when identities predicted win, the N170 was more negative for expressions of anger than for those of happiness (angry: -2.84 μV; SE = 0.40; happy: -2.19 μV; SE = 0.42), F(1, 23) = 11.804, p < .01, η_p ² = .34. There were no significant differences for identities that predicted loss, F = 1.436, p = .243 (see Fig. 4b).

VPP.

With identical temporal window than the N170 but over central electrodes (see Figs. 2 and 5), the overall ANOVA showed an interaction between Cue and Cued-Outcome, F(1, 23) = 7.217, p < .05, η_p ² = .24. Whereas in the emotion block the VPP was more positive for angry expressions (1.58 μV; SE = 0.30) associated with loss than for happy ones (2.06 μV; SE = 0.34), which cued win, F(1, 23) = 7.902, p < .05, η_p ² = .26, this was not the case in the identity block (F = 1.034, p = .320).

Fig. 5

Face-locked ERPs showing the modulation of the central N1 and VPP by emotion as relevant cue and by the central N2 by the identity as predictive cue

The interaction between Cued-Outcome and Facial Expression in the emotional conflict analysis was not significant (F < 1).

N2.

The N2 peaked at 230 ms over central electrodes and was analyzed from 200 to 260 ms (see Figs. 2 and 5). The overall analysis revealed a significant interaction between Cue and Cued-Outcome, F(1, 23) = 9.820, p < .01, η_p ² = .30. Posterior analyses showed a main effect of Cued-Outcome for the identity block, F(1, 23) = 5.102, p < .05, η_p ² = .18, due to a larger negative amplitude for faces associated to future loss (-0.28 μV; SE = 0.25) than for those that cued win (0.21 μV; SE = 0.26). This was not the case for the emotion block (F = 2.320, p = .141).

The interaction between Cued-Outcome and Facial Expression in the emotional conflict ANOVA was not significant (F < 1).

P3b.

The P3b potential peaked at 480 ms and was analyzed from 445 to 515 ms over bilateral central electrodes (see Figs. 2 and 6). The overall ANOVA showed a main effect of Hemisphere, F(1, 23) = 4.457, p < .05, η_p ² = .16, as this potential was more positive at right (2.63 μV; SE = 0.26) than at left electrodes (2.21 μV; SE = 0.27). There was also a main effect of Cued-Outcome, F(1, 23) = 4.392, p < .05, η_p ² = .16, as the P3b amplitude was more positive for cues signaling future wins (2.56 μV; SE = 0.25) than for those predicting losses (2.28 μV; SE = 0.25). This effect was better explained by its interaction with the Cue factor, F(1, 23) = 15.014, p = .001, η_p ² = .39 (see Fig. 6). Subsequent comparisons revealed a main effect of Cued-Outcome for the identity block, F(1, 23) = 13.802, p = .001, η_p ² = .37, as the P3b was more positive for faces linked to win (2.74 μV; SE = 0.31) than to loss (2.04 μV; SE = 0.27). This effect was not significant for the emotion block (F < 1).

Fig. 6

Face-locked ERPs showing the effect of Cued-Outcome on the identity block in the overall analysis on the P3b

The emotional conflict analysis yielded an interaction between Hemisphere, Cued-Outcome, and Facial Expression, F(1, 23) = 6.538, p < .05, η_p ² = .22. There was a main effect of Cued-Outcome for right hemisphere electrodes, F(1, 23) = 4.797, p < .05, η_p ² = .17, identical to that found in the overall ANOVA. At left locations, Cued-Outcome and Facial Expression interacted, F(1, 23) = 8.172, p < .01, η_p ² = .26. For identities predicting win, the P3b was larger for happy (2.22 μV; SE = 0.37) than for angry (1.60 μV; SE = 0.32) facial expressions, F(1, 23) = 9.481, p < .01, η_p ² = .29. The facial emotional expression of identities predicting loss, on the other hand, did not modulate the P3b (F < 1).

Discussion

The aim of this study was to evaluate how the strategic use of personal identity and facial emotional expression influences different processing levels when they are the relevant cues for the anticipation of cued economic outcomes in a social setting. We also evaluated how and at which processing levels ignored and nonpredictive facial emotions interact with identity decoding.

Overall, our behavioral results showed that participants used the relevant information to predict outcomes and make their decisions, which took longer for emotion than for identity judgments. The strategy participants followed is coherent with the role of motivational factors in the consecution of rewards. Previous studies have shown that reward expectations influence the way in which predictive signals are processed in decision-making scenarios (e.g., Baines, Ruz, Rao, Denison, & Nobre, 2011; Rushworth & Behrens, 2008). In the social realm, both personal identity and emotional expressions are crucial sources of information to predict the behavior of others and act accordingly (Ekman & Friesen, 2015; Frijda, 1988; Frijda & Mesquita, 1994; Frith & Frith, 2007). In this vein, angry expressions act as a message of likely negative intentions, which would reduce the number of bets during the game, while expressed happiness lead to infer future positive behaviors, which would increase bets during the game. Irrespective of the type of cue, participants needed more time to make their decision when they predicted losses as likely outcome than when they were expecting future winnings. This slowing down could be explained in terms of the mental cost derived from the specific trade-off of each behavioral choice (i.e., “easier” choices are made faster; see Chabris, Morris, Taubinsky, Laibson, & Schuldt, 2009; Krajbich, Bartling, Hare, & Fehr, 2015). Note that the expected payoff of betting over a likely loss was 0.42 EUR (0.17 * 2.5 + 0.83 * 0 = 0.42; for a discussion about the time allocation model, see Chabris et al., 2009), which was closer to 1 EUR (that is, the expected payoff associated with not betting) compared to that of betting over a likely win (0.83 * 2.5 + 0.17 * 0 = 2.07; Chabris et al., 2009). As a result, participants could have a stronger preference for betting when they predicted winnings than for not betting when they predicted losses. Thus, in our paradigm, choices might be “easier” in case of a likely win compared to a likely loss. It would be interesting for future research to explore this possibility in greater detail.

When participants used identity as cue, they needed more time to respond when a win was cued by an angry person. Such slowing down of responses reflects conflict between identity expectations and emotions, processed in a nonvolitional or automatic manner (Alguacil et al., 2015; Oosterhof & Todorov, 2008; Ruz & Tudela, 2011). Thus, contradictory predictions generated by the facial expression trigger an unavoidable conflict during facial identity cue processing, in line with previous results (Alguacil et al., 2015). Last, it is worth mentioning that although emotional content was gathered early, as discussed below, and did not suffer from any other source of conflict, participants needed more time to reach their decision during the emotion compared to the identity block. This slowing down has been previously observed in emotional conflict paradigms (e.g., Egner et al., 2008) and in emotional categorization studies (e.g., Atkinson et al., 2005). One of the plausible explanations is that emotions possess rich meaningfulness, so they require a kind of processing more extended in time, which would lead to increased response latencies (see Ben-David, Chajut, & Algom, 2012; Egner et al., 2008; Mogg, Holmes, Garner, & Bradley, 2008).

Turning to electrophysiological results, our data suggest early hemispheric differences according to the cue employed to draw predictions, and also rapid perceptual interactive adjustments depending on the nature of this cue and the valence of the prediction. Whereas outcomes cued by emotions enhanced the frontocentral N1 and VPP potentials, identity cues modulated the amplitude for the latter N2 and P3b. In addition, the conflict stemming from facial expressions conveyed by the identity cues influenced initial levels of face processing, as reflected on the N170, as well as decision-related processes indexed by the P3b.

Early processing stages

Early ERP responses, around 100 milliseconds after face onset have been preferably associated to perceptual processing and attentional orienting to low-level facial properties (e.g., Eimer & Holmes, 2002; Pitcher et al., 2007; Pizzagalli et al., 2002; Pourtois et al., 2005; Yang et al., 2011). In our experiment, the P1 and N1 potentials reflected only emotional, not identity, sensitivity. The amplitude of the P1 revealed a hemispheric asymmetry for the perception of salient emotional information, in coherence with the hypothesis that the right hemisphere is dominant for emotional content (see, e.g., Hagemann, Hewig, Naumann, Seifert, & Bartussek, 2003, 2005; Killgore & Yurgelun-Todd, 2007). At the same time, the central N1 showed a tendency to be sensitive to the strategic use of emotional cues. The amplitude of this potential increased with happy facial expressions anticipating win compared to angry ones, signaling loss. This enhancement only took place when emotions were the relevant predictors of trial outcome. Crucially, when personal identity was used as cue, the exact same facial expressions being ignored did not modulate the N1, which links the observed effects to the strategic use of emotions to generate predictions and not to the perceptual differences between emotions. Early effects reflected on the P1 and N1 potentials as well as their interpretation, however, must be taken with caution since results were only marginally significant. Note that, as explained in the Method section, when identity was the relevant cue, only trials in which happiness was paired with an identity predicting win and those in which anger appeared with an identity predicting loss were included in the ERPs, and thus the two data sets were fully equated in terms of stimuli distribution. Hence, predictive emotional information may receive an attentional benefit early in time to facilitate its preferential decoding (Vuilleumier, 2005), at least when it is explicitly employed as a valid signal to cue future outcomes and to guide decisions (Eimer & Holmes, 2002; Holmes, Vuilleumier, & Eimer, 2003; Pizzagalli, Koenig, Regard, & Lehmann, 1999). Some studies have found a “negative bias” given that such fast processing tends to occur with negative information (Carretié et al., 2004; Luo et al., 2010). However, the N1 reflects an increased response to positive valence. Our results are consistent with previous evidence suggesting an initial rapid global analysis of salient emotional information prior to a complex and fine visual categorization (Pizzagalli et al., 1999; Vuilleumier & Pourtois, 2007). Crude emotional information extracted from faces this early in time, probably by occipital extrastriate areas, could be a “header” to prepare areas for perceiving more detailed information in subsequent steps (e.g., Haxby et al., 2000; Pourtois et al., 2005; Sugase, Yamane, Ueno, & Kawano, 1999; Yang et al., 2011). The absence of similar effects for the identity cue may be partially due to the lack of differences at low-level physical characteristics among facial identities (in contrast to Ito & Urland, 2003; see also Tortosa, Lupiáñez, et al., 2013) as they were equally distributed (counterbalanced) across all experimental conditions.

Afterwards, a more detailed decoding of both invariant and changeable properties is reflected on the VPP and N170 (Eimer, 2000; Jeffreys, 1989; Joyce & Rossion, 2005; Rossion et al., 2000). Whereas the strategic use of emotion as predictive cue exerted influence in the amplitude of the VPP, identity did not. Its amplitude was heightened when angry faces signaled a likely loss. On the other hand, the effect of emotional conflict during the identity block was reflected on the N170, but not on the VPP. Identities displaying an expression of anger but linked to a likely win significantly enlarged the N170. Although the VPP has been defined as the equivalent dipole counterpart of the N170 (Joyce & Rossion, 2005), our results do not completely support such idea (see also Bötzel, Schulze, & Stodieck, 1995; Wong, Fung, McAlonan, & Chua, 2009). While some studies have proposed the inferior occipital and posterior fusiform gyri as neural generators of the VPP and, jointly with the STS, preferentially of the N170 potential (Bötzel et al., 1995; Caharel et al., 2002; Iidaka, Matsumoto, Haneda, Okada, & Sadato, 2006; Nguyen & Cunnington, 2014; Rossion et al., 2003), others have added the amygdala and the orbitofrontal cortex, both involved in emotion recognition (Adolphs, 2002; Wong et al., 2009), as the principal neural sources involved in the generation of the VPP. In addition, some fMRI studies have found a functional coupling between the fusiform gyrus and frontal areas during emotional conflict (Egner et al., 2008; Egner & Hirsch, 2005). All this evidence supports the VPP bias for emotional information and the influence of both identity and emotion on the N170 potential.

With respect to the emotional conflict observed at this stage, previous studies have found coincident effects to the extent that the valence congruency between words and emotional expression interacts at the N170 (Zhu et al., 2010; see also Hinojosa et al., 2015). This study adds a social context by analyzing conflict that stems from the incongruence between expectations arising from two different facial properties crucial for social adaptive communication, personal identity and emotion. Fisher and colleagues (2016) found interactive effects between identity and emotion-related processing at the N250r potential, but they did not find similar effects at the P1 or N170. Thus, our study is the first to show an effect of emotional conflict at the level of the N170 potential generated by the incongruence between expectations extracted from identity and those related to emotion. Interestingly, this effect displays the same direction as behavioral data, where we only observed differences for identities predicting winnings (replicating previous results; see Alguacil et al., 2015, Experiments 1 and 2). In this vein, our results suggest that the construction of the perceptual representation of faces when their identity is being used as a relevant cue to anticipate win as the most likely future outcome is influenced by the ignored emotional valence. This supports the interactive nature of identity and facial emotional information from an early stage of processing (Eimer, 2011; Fisher et al., 2016; Hinojosa et al., 2015; Jeffreys, 1989; Joyce & Rossion, 2005; Pitcher et al., 2007; Rossion et al., 2000; Vuilleumier & Pourtois, 2007).

Intermediate processing stages

The N2 appeared shortly after. Although previous studies have reported N2 modulations by both identity and emotional expressions (e.g., Ruz et al., 2013; Tortosa, Lupiáñez, et al., 2013), in our case this potential reflects the first stage of processing affected by expectations arising from facial identity. Faces linked to likely future loss heightened N2 amplitudes. Along these lines, the N2 deflection has been related to the presentation of task-relevant stimuli and the processing of likely outcomes (Baker & Holroyd, 2011), being especially sensitive to the processing of loss aversion (Potts, 2011). In the field of decision making, the anterior cingulate cortex, one of the neural sources of this potential (van Veen & Carter, 2002), is associated to the encoding of the value (i.e., win or loss) of an action before a decision is made (e.g., Rushworth & Behrens, 2008). Given the nature of our task demands, this N2 effect could be driven by a detection of the incongruence between the expectations associated to identity, future loss, and the motivation to achieve wins during the game. The lack of an effect of emotional conflict on this potential, related to cognitive control and conflict processing in classical studies (Folstein & Van Petten, 2008; Kopp et al., 1996), may be partially due to the nature of the stimuli employed. The N2 is heightened by conflict in classic stimulus–response paradigms (Egner et al., 2008; Folstein & Van Petten, 2008; Kopp et al., 1996) and also when there is emotional, nonfacial, interfering information (e.g., Kanske & Kotz, 2010b). In the field of social cognition studies, the N2 potential is highly sensitive to verbal descriptions in social categorization paradigms (e.g., words; see Derks et al., 2015) and also to emotional displays when they are strategically employed in interpersonal interactions (Ruz et al., 2013). However, the N2 potential seems less suitable to index emotional conflict in settings where emotional information is not provided by a face (e.g., Alguacil et al., 2013) or has no explicit association with relevant personal descriptions, such as in the current paradigm (see also Tortosa, Lupiáñez, et al., 2013).

Late processing stages

The last potential in which we observed modulations in terms of value predictions and conflict was the P3b, which has been associated to outcome evaluation, reward processing and inhibition during incongruent situations (Moser et al., 2014; Neuhaus et al., 2010; Polich, 2007; Valle-Inclán, 1996; Wu & Zhou, 2009). This potential showed a hemispheric asymmetry because expectations linked to facial identity influenced the P3b amplitude on the right hemisphere. This asymmetry has been reported previously and has been attributed to the association of the P3b to the frontoparietal right hemisphere attentional network (Polich, 2007). At left locations, conflict was reflected as an enhancement by the emotion expressed by identities linked to likely win. This parallels a similar result from a previous study where the authors found heightened amplitude for positive personal descriptions linked to partners in a decision-making paradigm (Moser et al., 2014). In our experiment, the identities linked to likely win are conceptually similar to trustworthy partners as far as both the positive personal description and the likely win are a signal of the most beneficial economic outcome. Under the rationale of the inhibition hypothesis, when the identity and the facial expression signaled a likely win (positive outcome) the implementation of an inhibition mechanism was not necessary. But when both properties cued contradictory outcomes (i.e., the identity predicted win and the angry facial expression led to an expectation of loss) the implementation of inhibition was more necessary, which reduced the P3b amplitude (see Neuhaus et al., 2010; Valle-Inclán, 1996). Our results provide further evidence to support the influence of personal identities and facial expressions on this potential (Campanella et al., 2013; Campanella et al., 2000; Campanella et al., 2002). It is worth noting that to employ the identity as a cue in our experiment, participants first had to retrieve from memory the expectations linked to each face. The capture of attention by the nonrelevant emotional content of the face could have boosted specific memory operations, which in turn could trigger inhibition mechanisms to avoid unnecessary information (Polich, 2007). The lack of effects at the level of the P3b potential for the emotion as predictive cue may also be partially caused by the slowing down of responses during the emotion block, which may have delayed the emotional effect on the P3b deflection.

Overall, our ERP results offer indices of emotional processing taking place earlier than personal identity decoding, which suggests a segregation between these dimensions at initial stages of face perception. Our data fit with previous theoretical statements of segregation of different facial properties (Bruce & Young, 1986; Calder, 2011; Calder & Young, 2005; Haxby & Gobbini, 2011; Haxby et al., 2000). On the other hand, our analyses also support an early emotional interference during the use of the identity as a cue. Such emotional information is automatically integrated in the construction of facial identity and also later on when a response must be selected. This interference provides strong evidence in favor of the interdependence between changeable and invariant facial properties from early perceptual processing to response selection stages (Baudouin et al., 2000; Fisher et al., 2016; Ganel & Goshen-Gottstein, 2004; Schweinberger & Soukup, 1998; see also Todorov et al., 2008). As far as emotional expression is a useful tool for predicting outcomes and also a key source of information for social judgments based on faces, the relevance of emotion may be strengthened in social scenarios where personal identity is used as a cue to guide decisions (see, e.g., Ganel & Goshen-Gottstein, 2004; Oosterhof & Todorov, 2008; Todorov et al., 2005; Todorov et al., 2008). As a result, identity and emotion cannot be separated, because the latter, which is processed faster, acts as a header to prepare subsequent areas in the system specialized in the decoding of invariant personal identity (see Sugase et al., 1999; Yang et al., 2011).

There are, however, some limitations in this study that warrant further investigations. Future studies should incorporate personality assessments to obtain a detailed profile of prosocial attitudes, impulsivity, and personal motivation to seek rewards, given the relevance of these factors (e.g., Appelt, Milch, Handgraaf, & Weber, 2011; Espín, Exadaktylos, & Neyse, 2016; Jia, Zhang, Li, Feng, & Li, 2013; Mohammed & Schwall, 2009). Future studies should also investigate the interference that identity information may exert on emotion-guided judgments in social contexts, although preliminary unpublished data from our lab suggest that emotional judgments are not affected by identity-interfering information (but see also Fisher et al., 2016). Also, in this and previous (Alguacil et al., 2015) studies we have only observed an emotional conflict effect for identities predicting wins, although the reasons for this are still unknown. Finally, a complementary fMRI study could help in confirming the role of occipital visual areas in the decoding of emotional information in the current paradigm and whether this may contribute to the unavoidable influence of emotional expressions on identity-based judgments. An fMRI study would also be useful to clarify how interference from facial expressions is resolved in social contexts and whether the supporting neural mechanisms are similar to those involved in the resolution of conflict in more classical interference situations (e.g., Botvinick, Braver, Barch, Carter & Cohen, 2001; Botvinick, Cohen & Carter, 2004; Egner et al., 2008).

Conclusions

In conclusion, our findings provide new evidence on the neural mechanisms that support the use of invariant and changeable facial properties to predict economic outcomes in decision-making scenarios. Facial emotional expressions exert an unavoidable influence in the construction of personal identity when the latter is employed as a cue to anticipate outcomes of choices, and we have provided evidence for this at both the behavioral and electrophysiological levels. Such influence is presumably guided by the early capture of attention by emotional information, which is rapidly passed as input to subsequent stages of facial analysis. Emotional expressions are integrated in the perceptual construction of personal identity and also at latter decision stages, where conflictive items may be inhibited to favor the selection of identities with highest motivational value.