Skip to main content
SpringerLink
Log in

What makes a smiling face look happy? Visual saliency, distinctiveness, and affect

  • Original Article
  • Published:
Psychological Research Aims and scope Submit manuscript

Abstract

We investigated the relative contribution of (a) perceptual (eyes and mouth visual saliency), (b) conceptual or categorical (eye expression distinctiveness), and (c) affective (rated valence and arousal) factors, and (d) specific morphological facial features (Action Units; AUs), to the recognition of facial happiness. The face stimuli conveyed truly happy expressions with a smiling mouth and happy eyes, or blended expressions with a smile but non-happy eyes (neutral, sad, fearful, disgusted, surprised, or angry). Saliency, distinctiveness, affect, and AUs served as predictors; the probability of judging a face as happy was the criterion. Both for truly happy and for blended expressions, the probability of perceiving happiness increased mainly as a function of positive valence of the facial configuration. In addition, for blended expressions, the probability of being (wrongly) perceived as happy increased as a function of (a) delayed saliency and (b) reduced distinctiveness of the non-happy eyes, and (c) enhanced AU 6 (cheek raiser) or (d) reduced AUs 4, 5, and 9 (brow lowerer, upper lid raiser, and nose wrinkler, respectively). Importantly, the later the eyes become visually salient relative to the smiling mouth, the more likely it is that faces will look happy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The age (range 18–25 years, all the studies) and sex proportion were practically the same in the current study and in the prior studies, and all the participants were drawn from the same undergraduate psychology population, albeit in different years (from 2011 to 2015): (a) current experiment (M age = 21.1 years; female/male proportion = 75/25%), (b) current norming study (M age = 21.3; f/m proportion = 67/33%); (c) Calvo et al. (2012) (M age = 21.5; f/m proportion = 72/28%); (d) Calvo, Gutiérrez-García et al. (2013) (M age = 21.7; f/m proportion = 76/24%).

References

  • Adolphs, R. (2002). Recognizing emotion from facial expressions: Psychological and neurological mechanisms. Behavioral and Cognitive Neuroscience Reviews, 1, 21–62.

    Article  PubMed  Google Scholar 

  • Ambadar, Z., Cohn, J. F., & Reed, L. I. (2009). All smiles are not created equal: Morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous. Journal of Nonverbal Behavior, 33, 17–34.

    Article  PubMed  PubMed Central  Google Scholar 

  • Barrett, L. F. (2006). Are emotions natural kinds? Perspectives on Psychological Science, 1, 28–58.

    Article  PubMed  Google Scholar 

  • Bartlett, M., & Whitehill, J. (2011). Automated facial expression measurement: Recent applications to basic research in human behavior, learning, and education. In A. Calder, G. Rhodes, M. Johnson, & J. Haxby (Eds.), Handbook of face perception (pp. 489–513). Oxford: Oxford University Press.

    Google Scholar 

  • Beaudry, O., Roy-Charland, A., Perron, M., Cormier, I., & Tapp, R. (2014). Featural processing in recognition of emotional facial expressions. Cognition and Emotion, 28, 416–432.

    Article  PubMed  Google Scholar 

  • Becker, D. V., & Srinivasan, N. (2014). The vividness of the happy face. Current Directions in Psychological Science, 23, 189–194.

    Article  Google Scholar 

  • Bombari, D., Schmid, P. C., Schmid-Mast, M., Birri, S., Mast, F. W., & Lobmaier, J. S. (2013). Emotion recognition: The role of featural and configural face information. Quarterly Journal of Experimental Psychology, 66, 2426–2442.

    Article  Google Scholar 

  • Borji, A., & Itti, L. (2013). State-of-the-art in visual attention modeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35, 185–207.

    Article  PubMed  Google Scholar 

  • Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001). Emotion and motivation I: Defensive and appetitive reactions in picture processing. Emotion, 1, 276–298.

    Article  PubMed  Google Scholar 

  • Calder, A. J., Young, A. W., Keane, J., & Dean, M. (2000). Configural information in facial expression perception. Journal of Experimental Psychology: Human Perception and Performance, 26, 527–551.

    PubMed  Google Scholar 

  • Calvo, M. G., & Beltrán, D. (2013). Recognition advantage of happy faces: Tracing the neurocognitive processes. Neuropsychologia, 51, 2051–2060.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., & Beltrán, D. (2014). Brain lateralization of holistic versus analytic processing of emotional facial expressions. NeuroImage, 92, 237–247.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., Beltrán, D., & Fernández-Martín, A. (2014). Early processing of happy facial expressions in peripheral vision: Neurophysiological evidence. Biological Psychology, 100, 60–70.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., Fernández-Martín, A., & Nummenmaa, L. (2012). Perceptual, categorical, and affective processing of ambiguous smiling facial expressions. Cognition, 125, 373–393.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., Fernández-Martín, A., & Nummenmaa, L. (2013). A smile biases the recognition of eye expressions: Configural projection from a salient mouth. The Quarterly Journal of Experimental Psychology, 66, 1159–1181.

    Article  Google Scholar 

  • Calvo, M. G., Fernández-Martín, A., & Nummenmaa, L. (2014). Facial expression recognition in peripheral versus central vision: Role of the eyes and the mouth. Psychological Research, 78, 180–195.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., Gutiérrez-García, A., Avero, P., & Lundqvist, D. (2013). Attentional mechanisms in judging genuine and fake smiles: Eye-movement patterns. Emotion, 13, 792–802.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., Marrero, H., & Beltrán, D. (2013). When does the brain distinguish between genuine and ambiguous smiles? An ERP study. Brain and Cognition, 81, 237–246.

    Article  PubMed  Google Scholar 

  • Calvo, M. G., & Nummenmaa, L. (2008). Detection of emotional faces: Salient physical features guide effective visual search. Journal of Experimental Psychology: General, 137, 471–494.

    Article  Google Scholar 

  • Calvo, M. G., & Nummenmaa, L. (2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition and Emotion, 30, 1081–1106.

    Article  PubMed  Google Scholar 

  • Carroll, N. C., & Young, A. W. (2005). Priming of emotion recognition. The Quarterly Journal of Experimental Psychology, 58A, 1173–1197.

    Article  Google Scholar 

  • Cohn, J. F., & De la Torre, F. (2015). Automated face analysis for affective computing. In R. A. Calvo, S. DiMello, J. Gratch, & A. Kappas (Eds.), The Oxford handbook of affective computing (pp. 131–150). New York: Oxford University Press.

    Google Scholar 

  • Cottrell, G. W., & Hsiao, J. H. (2011). Neurocomputational models of face processing. In A. Calder, G. Rhodes, M. Johnson, & J. Haxby (Eds.), The Oxford handbook of face perception (pp. 401–425). Oxford: Oxford University Press.

    Google Scholar 

  • Crivelli, C., Carrera, P., & Fernández-Dols, J. M. (2015). Are smiles a sign of happiness? Spontaneous expressions of judo winners. Evolution and Human Behavior, 36, 52–58.

    Article  Google Scholar 

  • Dailey, M. N., Cottrell, G. W., Padgett, C., & Adolphs, R. (2002). EMPATH: A neural network that categorizes facial expressions. Journal of Cognitive Neuroscience, 14, 1158–1173.

    Article  PubMed  Google Scholar 

  • Eisenbarth, H., Alpers, G. W., Segré, D., Calogero, A., & Angrilli, A. (2008). Categorization and evaluation of emotional faces in psychopathic women. Psychiatry Research, 159, 189–195.

    Article  PubMed  Google Scholar 

  • Ekman, P. (1992). Are there basic emotions? Psychological Review, 99, 550–553.

    Article  PubMed  Google Scholar 

  • Ekman, P. (2001). Telling lies: Clues to deceit in the marketplace, politics, and marriage. New York: W.W. Norton & Co.

    Google Scholar 

  • Ekman, P., & Cordaro, D. (2011). What is meant by calling emotions basic. Emotion Review, 3, 364–370.

    Article  Google Scholar 

  • Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system. Investigator’s guide. Salt Lake City: Human Face.

    Google Scholar 

  • Frank, M. G., Ekman, P., & Friesen, W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment. Journal of Personality and Social Psychology, 64, 83–93.

    Article  PubMed  Google Scholar 

  • Gardiner, J. M., Ramponi, C., & Richardson-Klavehn, A. (2002). Recognition memory and decision processes: A meta-analysis of remember, know, and guess responses. Memory, 10, 83–98.

    Article  PubMed  Google Scholar 

  • Girard, J. M., Cohn, J. F., Jeni, L. A., Sayette, M. A., & De la Torre, F. (2015). Spontaneous facial expression in unscripted social interactions can be measured automatically. Behavior Research Methods, 47, 1136–1147.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gunnery, S. D., & Hall, J. A. (2014). The Duchenne smile and persuasion. Journal of Nonverbal Behavior, 38, 181–194.

    Article  Google Scholar 

  • Gunnery, S. D., & Ruben, M. A. (2016). Perceptions of Duchenne and non-Duchenne smiles: A meta-analysis. Cognition and Emotion, 30, 501–515.

    Article  PubMed  Google Scholar 

  • Gutiérrez-García, A., & Calvo, M. G. (2015). Discrimination thresholds for smiles in genuine versus blended facial expressions. Cogent Psychology, 2, 1064586.

    Google Scholar 

  • Itti, L., & Koch, C. (2000). A saliency-based search mechanism for overt and covert shifts of visual attention. Vision Research, 40, 1489–1506.

    Article  PubMed  Google Scholar 

  • Johnston, L., Miles, L. K., & Macrae, C. (2010). Why are you smiling at me? Social functions of enjoyment and non-enjoyment smiles. British Journal of Social Psychology, 49, 107–127.

    Article  PubMed  Google Scholar 

  • Kohler, C. G., Turner, T., Stolar, N. M., Bilker, W. B., Brensinger, C. M., Gur, R. E., & Gur, R. C. (2004). Differences in facial expressions of four universal emotions. Psychiatry Research, 128, 235–244.

    Article  PubMed  Google Scholar 

  • Krumhuber, E. G., Likowski, K. U., & Weyers, P. (2014). Facial mimicry of spontaneous and deliberate Duchenne and non-Duchenne smiles. Journal of Nonverbal Behavior, 38, 1–11.

    Article  Google Scholar 

  • Krumhuber, E. G., & Manstead, A. S. R. (2009). Can Duchenne smiles be feigned? New evidence on felt and false smiles. Emotion, 9, 807–820.

    Article  PubMed  Google Scholar 

  • Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (2008). International affective picture system (IAPS): Affective ratings of pictures and instruction manual. Technical Report A-8. University of Florida, Gainesville, FL.

  • Lipp, O., Price, S. M., & Tellegen, C. L. (2009). No effect of inversion on attentional and affective processing of facial expressions. Emotion, 9, 248–259.

    Article  PubMed  Google Scholar 

  • Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska directed emotional faces—KDEF. CD-ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, Stockholm, Sweden. ISBN 91-630-7164-9.

  • Luo, W., Feng, W., He, W., Wang, N.-Y., & Luo, Y.-J. (2010). Three stages of facial expression processing: ERP study with rapid serial visual presentation. NeuroImage, 47, 1856–1867.

    Google Scholar 

  • McLellan, T. L., Johnston, L., Dalrymple-Alford, J., & Porter, R. (2010). Sensitivity to genuine vs. posed emotion specified in facial displays. Cognition and Emotion, 24, 1277–1292.

    Article  Google Scholar 

  • McLellan, T. L., Wilcke, J. C., Johnston, L., Watts, R., & Miles, L. K. (2012). Sensitivity to posed and genuine displays of happiness and sadness: A fMRI study. Neuroscience Letters, 531, 149–154.

    Article  PubMed  Google Scholar 

  • Miles, L. K., & Johnston, L. (2007). Detecting happiness: Perceiver sensitivity to enjoyment and non-enjoyment smiles. Journal of Nonverbal Behavior, 31, 259–275.

    Article  Google Scholar 

  • Nelson, N. L., & Russell, J. A. (2013). Universality revisited. Emotion Review, 5, 8–15.

    Article  Google Scholar 

  • Niedenthal, P. M., Mermillod, M., Maringer, M., & Hess, U. (2010). The simulation of smiles (SIMS) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33, 417–433.

    Article  PubMed  Google Scholar 

  • Nusseck, M., Cunningham, D. W., Wallraven, C., & Bülthoff, H. H. (2008). The contribution of different facial regions to the recognition of conversational expressions. Journal of Vision, 8(8), 1–23.

    Article  PubMed  Google Scholar 

  • Okubo, M., Kobayashi, A., & Ishikawa, K. (2012). A fake smile thwarts cheater detection. Journal of Nonverbal Behavior, 36, 217–225.

    Article  Google Scholar 

  • Olderbak, S., Hildebrandt, A., Pinkpank, T., Sommer, W., & Wilhelm, O. (2014). Psychometric challenges and proposed solutions when scoring facial emotion expression codes. Behavior Research Methods, 46, 992–1006.

    Article  PubMed  Google Scholar 

  • Quadflieg, S., Vermeulen, N., & Rossion, B. (2013). Differential reliance on the Duchenne marker during smile evaluations and person judgments. Journal of Nonverbal Behavior, 37, 69–77.

    Article  Google Scholar 

  • Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115, 102–141.

    Article  PubMed  Google Scholar 

  • Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145–172.

    Article  PubMed  Google Scholar 

  • Slessor, G., Bailey, P. E., Rendell, P. G., Ruffman, T., Henry, J. D., & Miles, L. K. (2014). Examining the time course of young and older adults’ mimicry of enjoyment and non-enjoyment smiles. Emotion, 14, 532–544.

    Article  PubMed  Google Scholar 

  • Smith, M. L., Cottrell, G., Gosselin, F., & Schyns, P. G. (2005). Transmitting and decoding facial expressions of emotions. Psychological Science, 16, 184–189.

    Article  PubMed  Google Scholar 

  • Snodgrass, J. G., & Corwin, J. (1988). Pragmatics of measuring recognition memory: Applications to dementia and amnesia. Journal of Experimental Psychology: General, 117, 34–50.

    Article  Google Scholar 

  • Susskind, J. M., Littlewort, G., Bartlett, M. S., Movellan, J., & Anderson, A. K. (2007). Human and computer recognition of facial expressions of emotion. Neuropsychologia, 45, 152–162.

    Article  PubMed  Google Scholar 

  • Underwood, G., & Foulsham, T. (2006). Visual saliency and semantic incongruency influence eye movements when inspecting pictures. The Quarterly Journal of Experimental Psychology, 59, 1931–1949.

    Article  PubMed  Google Scholar 

  • Walther, D., & Koch, C. (2006). Modelling attention to salient proto-objects. Neural Networks, 19, 1395–1407.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This research was supported by Grants PSI2009-07245 from the Spanish Ministerio de Ciencia e Innovación, and PSI2014-54720-P from the Spanish Ministerio de Economía y Competitividad.

Author information

Authors and Affiliations

  1. Department of Cognitive Psychology, Universidad de La Laguna, 38205, Tenerife, Spain

    Manuel G. Calvo

  2. Universidad de Burgos, Burgos, Spain

    Aida Gutiérrez-García & Mario Del Líbano

Authors
  1. Manuel G. Calvo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aida Gutiérrez-García
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mario Del Líbano
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel G. Calvo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and the national research committee (according to the research projects approved under Grants PSI2009-07245 and PSI2014-54720-P) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (XLSX 34 kb)

Supplementary material 2 (DOCX 11 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Calvo, M.G., Gutiérrez-García, A. & Del Líbano, M. What makes a smiling face look happy? Visual saliency, distinctiveness, and affect. Psychological Research 82, 296–309 (2018). https://doi.org/10.1007/s00426-016-0829-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00426-016-0829-3

Search

Navigation