Advertisement

Journal of Nonverbal Behavior

, Volume 41, Issue 3, pp 221–238 | Cite as

Emotion Recognition in Nonverbal Face-to-Face Communication

  • Janina KüneckeEmail author
  • Oliver Wilhelm
  • Werner Sommer
Original Paper

Abstract

The perception of emotional facial expressions may activate corresponding facial muscles in the receiver, also referred to as facial mimicry. Facial mimicry is highly dependent on the context and type of facial expressions. While previous research almost exclusively investigated mimicry in response to pictures or videos of emotional expressions, studies with a real, face-to-face partner are still rare. Here we compared facial mimicry of angry, happy, and sad expressions and emotion recognition in a dyadic face-to-face setting. In sender-receiver dyads, we recorded facial electromyograms in parallel. Senders communicated to the receivers—with facial expressions only—the emotions felt during specific personal situations in the past, eliciting anger, happiness, or sadness. Receivers mostly mimicked happiness, to a lesser degree, sadness, and anger as the least mimicked emotion. In actor-partner interdependence models we showed that the receivers’ own facial activity influenced their ratings, which increased the agreement between the senders’ and receivers’ ratings for happiness, but not for angry and sad expressions. These results are in line with the Emotion Mimicry in Context View, holding that humans mimic happy expressions according to affiliative intentions. The mimicry of sad expressions is less intense, presumably because it signals empathy and might imply personal costs. Direct anger expressions are mimicked the least, possibly because anger communicates threat and aggression. Taken together, we show that incidental facial mimicry in a face-to-face setting is positively related to the recognition accuracy for non-stereotype happy expressions, supporting the functionality of facial mimicry.

Keywords

Facial mimicry Facial EMG Emotion recognition Face-to-face-interaction Actor-partner interdependence model (APIM) 

Supplementary material

10919_2017_255_MOESM1_ESM.docx (21 kb)
Supplementary material 1 (DOCX 21 kb)

References

  1. Barton, K., & Barton, M. K. (2015). Package ‘mumin’. Version, 1, 18. https://cran.r-project.org/web/packages/MuMIn/MuMIn.pdf.
  2. Bates, D., Mächler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using LME4. Journal of Statistical Software. doi: 10.18637/jss.v067.i01.Google Scholar
  3. Blairy, S., Herrera, P., & Hess, U. (1999). Mimicry and the judgment of emotional facial expressions. Journal of Nonverbal Behavior, 23(1), 5–41. doi: 10.1023/a:1021370825283.CrossRefGoogle Scholar
  4. Bogart, K. R., & Matsumoto, D. (2010). Facial mimicry is not necessary to recognize emotion: Facial expression recognition by people with moebius syndrome. Social Neuroscience, 5(2), 241–251. doi: 10.1080/17470910903395692.CrossRefGoogle Scholar
  5. Boker, S. M., Rotondo, J. L., Xu, M., & King, K. (2002). Windowed cross-correlation and peak picking for the analysis of variability in the association between behavioral time series. Psychological Methods, 7(3), 338. doi: 10.1037/1082-989X.7.3.338.PubMedCrossRefGoogle Scholar
  6. Boucsein, W. (2012). Electrodermal activity (2nd ed.). New York: Springer Science & Business Media. doi: 10.1007/978-1-4614-1126-0.CrossRefGoogle Scholar
  7. Bourgeois, P., & Hess, U. (2008). The impact of social context on mimicry. Biological Psychology, 77(3), 343–352. doi: 10.1016/j.biopsycho.2007.11.008.PubMedCrossRefGoogle Scholar
  8. Chartrand, T. L., & Bargh, J. A. (1999). The chameleon effect: The perception-behavior link and social interaction. Journal of Personality and Social Psychology, 76(6), 893–910. doi: 10.1037/0022-3514.76.6.893.PubMedCrossRefGoogle Scholar
  9. Chartrand, T. L., & Lakin, J. L. (2012). The antecedents and consequences of human behavioral mimicry. Annual Review of Psychology. doi: 10.1146/annurev-psych-113011-143754.PubMedGoogle Scholar
  10. Chartrand, T. L., & van Baaren, R. (2009) Chapter 5 human mimicry. 41, 219–274. doi: 10.1016/s0065-2601(08)00405-x.
  11. de Groot, J. H., Smeets, M. A., Rowson, M. J., Bulsing, P. J., Blonk, C. G., Wilkinson, J. E., et al. (2015). A sniff of happiness. Psychological Science, 26(6), 684–700. doi: 10.1177/0956797614566318.PubMedCrossRefGoogle Scholar
  12. Dimberg, U., Thunberg, M., & Grunedal, S. (2002). Facial reactions to emotional stimuli: Automatically controlled emotional responses. Cognition and Emotion, 6(4), 449–471. doi: 10.1080/02699930143000356.CrossRefGoogle Scholar
  13. Draine, S. (1998). Inquisit [computer software]. Seattle, WA: Millisecond Software.Google Scholar
  14. Ekman, P., Friesen, W. V., & Ellsworth, P. C. (1972). Emotion in the humanface: Guidelinesfor research and an integration offindings. New York: Pergamon.Google Scholar
  15. Fischer, A. H., Becker, D., & Veenstra, L. (2012). Emotional mimicry in social context: The case of disgust and pride. Frontiers in Psychology, 3, 475. doi: 10.3389/fpsyg.2012.00475.PubMedPubMedCentralGoogle Scholar
  16. Fischer, A. H., & van Kleef, G. A. (2010). Where have all the people gone? A plea for including social interaction in emotion research. Emotion Review, 2(3), 208–211. doi: 10.1177/1754073910361980.CrossRefGoogle Scholar
  17. Fridlund, A. J. (1994). Human facial expression: An evolutionary view. San Diego, CA: Academic Press.Google Scholar
  18. Fridlund, A. J., & Cacioppo, J. T. (1986). Guidelines for human electromyographic research. Psychophysiology, 23, 567–589. doi: 10.1111/j.1469-8986.1986.tb00676.x.PubMedCrossRefGoogle Scholar
  19. Goldman, A. I., & Sripada, C. S. (2005). Simulationist models of face-based emotion recognition. Cognition, 94(3), 193–213. doi: 10.1016/j.cognition.2004.01.005.PubMedCrossRefGoogle Scholar
  20. Gunnery, S. D., Hall, J. A., & Ruben, M. A. (2012). The deliberate Duchenne smile: Individual differences in expressive control. Journal of Nonverbal Behavior, 37(1), 29–41. doi: 10.1007/s10919-012-0139-4.CrossRefGoogle Scholar
  21. Hess, U., & Blairy, S. (2001). Facial mimicry and emotional contagion to dynamic emotional facial expressions and their influence on decoding accuracy. International Journal of Psychophysiology, 40, 129–141. doi: 10.1016/S0167-8760(00)00161-6.PubMedCrossRefGoogle Scholar
  22. Hess, U., & Bourgeois, P. (2010). You smile–I smile: Emotion expression in social interaction. Biological Psychology, 84(3), 514–520. doi: 10.1016/j.biopsycho.2009.11.001.PubMedCrossRefGoogle Scholar
  23. Hess, U., & Fischer, A. (2013). Emotional mimicry as social regulation. Personality and Social Psychology Review, 17(2), 142–157. doi: 10.1177/1088868312472607.PubMedCrossRefGoogle Scholar
  24. Hess, U., & Fischer, A. (2014). Emotional mimicry: Why and when we mimic emotions. Social and Personality Psychology Compass, 8(2), 45–57. doi: 10.1111/spc3.12083.CrossRefGoogle Scholar
  25. Hildebrandt, A., Sommer, W., Schacht, A., & Wilhelm, O. (2015). Perceiving and remembering emotional facial expressions: A basic facet of emotional intelligence. Intelligence, 50, 52–67. doi: 10.1016/j.intell.2015.02.003.CrossRefGoogle Scholar
  26. Hyniewska, S., & Sato, W. (2015). Facial feedback affects valence judgments of dynamic and static emotional expressions. Frontiers in Psychology, 6, 291. doi: 10.3389/fpsyg.2015.00291.PubMedPubMedCentralCrossRefGoogle Scholar
  27. Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). Dyadic data analysis. New York: Guilford Press.Google Scholar
  28. Kosonogov, V., Titova, A., & Vorobyeva, E. (2015). Empathy, but not mimicry restriction, influences the recognition of change in emotional facial expressions. Quarterly Journal of Experimental Psychology, 68(10), 2106. doi: 10.1080/17470218.2015.1009476.CrossRefGoogle Scholar
  29. Kret, M. E. (2015). Emotional expressions beyond facial muscle actions. A call for studying autonomic signals and their impact on social perception. Frontiers in Psychology, 6, 711. doi: 10.3389/fpsyg.2015.00711.PubMedPubMedCentralCrossRefGoogle Scholar
  30. Kulesza, W. M., Cisłak, A., Vallacher, R. R., Nowak, A., Czekiel, M., & Bedynska, S. (2015). The face of the chameleon: The experience of facial mimicry for the mimicker and the mimickee. The Journal of Social Psychology, 155(6), 590–604. doi: 10.1080/00224545.PubMedPubMedCentralCrossRefGoogle Scholar
  31. Künecke, J., Hildebrandt, A., Recio, G., Sommer, W., & Wilhelm, O. (2014). Facial EMG responses to emotional expressions are related to emotion perception ability. PLoS ONE, 9(1), e84503. doi: 10.1371/journal.pone.0084053.CrossRefGoogle Scholar
  32. Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2013). Lmertest: Tests for random and fixed effects for linear mixed effect models (LMER objects of LME4 package). R package version, 2(6). http://www2.uaem.mx/r-mirror/web/packages/lmerTest/lmerTest.pdf.
  33. Lakin, J. L., Jefferis, V. E., Cheng, C. M., & Chartrand, T. L. (2003). The chameleon effect as social cue: Evidence for the evolutionary significance of nonconscious mimicry. Journal of Nonverbal Behavior, 27(3), 145–162. doi: 10.1023/A:1025389814290.CrossRefGoogle Scholar
  34. Lench, H. C., Flores, S. A., & Bench, S. W. (2011). Discrete emotions predict changes in cognition, judgment, experience, behavior, and physiology: A meta-analysis of experimental emotion elicitations. Psychological Bulletin, 37(5), 834–855. doi: 10.1037/a0024244.supp.CrossRefGoogle Scholar
  35. Mauersberger, H., Blaison, C., Kafetsios, K., Kessler, C. L., & Hess, U. (2015). Individual differences in emotional mimicry: Underlying traits and social consequences. European Journal of Personality, 29(5), 512–529. doi: 10.1002/per.2008.CrossRefGoogle Scholar
  36. McIntosh, D. N. (1996). Facial feedback hypotheses: Evidence, implications, and directions. Motivation and Emotion, 20(2), 121–147. doi: 10.1007/BF02253868.CrossRefGoogle Scholar
  37. Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining R2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. doi: 10.1111/j.2041-210x.2012.00261.x.CrossRefGoogle Scholar
  38. Neal, D. T., & Chartrand, T. L. (2011). Embodied emotion perception: Amplifying and dampening facial feedback modulates emotion perception accuracy. Social Psychological and Personality Science, 2(6), 673–678. doi: 10.1177/1948550611406138.CrossRefGoogle Scholar
  39. Niedenthal, P. M., Augustinova, M., & Rychlowska, M. (2010a). Body and mind: Zajonc’s (re)introduction of the motor system to emotion and cognition. Emotion Review, 2(4), 340–347. doi: 10.1177/1754073910376423.CrossRefGoogle Scholar
  40. Niedenthal, P. M., Brauer, M., Halberstadt, J. B., & Innes-Ker, Å. H. (2001). When did her smile drop? Facial mimicry and the influences of emotional state on the detection of change in emotional expression. Cognition and Emotion, 15(6), 853–864. doi: 10.1080/02699930143000194.CrossRefGoogle Scholar
  41. Niedenthal, P. M., Mermillod, M., Maringer, M., & Hess, U. (2010b). The simulation of smiles (sims) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences, 33(6), 417–433. doi: 10.1017/S0140525X10000865.PubMedCrossRefGoogle Scholar
  42. Oberman, L. M., Winkielman, P., & Ramachandran, V. S. (2007). Face to face: Blocking facial mimicry can selectively impair recognition of emotional expressions. Social Neuroscience, 2(3–4), 167–178. doi: 10.1080/17470910701391943.PubMedCrossRefGoogle Scholar
  43. R Core Team. (2014). R: A language and environment for statistical computing. R foundation for statistical computing (Version R version 3.1.2). Vienna, Austria. Retrieved from http://www.R-project.org/.
  44. Ramseyer, F., & Tschacher, W. (2011). Nonverbal synchrony in psychotherapy: coordinated body movement reflects relationship quality and outcome. Journal of Consulting and Clinical Psychology, 79(3), 284. doi: 10.1037/a0023419.PubMedCrossRefGoogle Scholar
  45. Rychlowska, M., Canadas, E., Wood, A., Krumhuber, E. G., Fischer, A., & Niedenthal, P. M. (2014). Blocking mimicry makes true and false smiles look the same. PLoS ONE, 9(3), e90876. doi: 10.1371/journal.pone.0090876.PubMedPubMedCentralCrossRefGoogle Scholar
  46. Safdar, S., Friedlmeier, W., Matsumoto, D., Yoo, S. H., Kwantes, C. T., Kakai, H., et al. (2009). Variations of emotional display rules within and across cultures: A comparison between canada, USA, and japan. Canadian Journal of Behavioural Science, 41(1), 1–10. doi: 10.1037/a0014387.CrossRefGoogle Scholar
  47. Sato, W., Fujimura, T., Kochiyama, T., & Suzuki, N. (2013). Relationships among facial mimicry, emotional experience, and emotion recognition. PLoS ONE, 8(3), e57889. doi: 10.1371/journal.pone.0057889.g001.PubMedPubMedCentralCrossRefGoogle Scholar
  48. Sonnby-Borgström, M. (2002). The facial expression says more than words: Emotional “contagion” related to empathy? Lakartidningen, 99(13), 1438–1442. Retrieved from <Go to ISI>://WOS:000174751300002.Google Scholar
  49. Stel, M., & Harinck, F. (2011). Being mimicked makes you a prosocial voter. Experimetal Psychology, 58(1), 79–84. doi: 10.1027/1618-3169/a000070.Google Scholar
  50. Stel, M., Van Baaren, R. B., & Vonk, R. (2008). Effects of mimicking: Acting prosocially by being emotionally moved. European Journal of Social Psychology, 38(6), 965–976.CrossRefGoogle Scholar
  51. Stel, M., & van Knippenberg, A. (2008). The role of facial mimicry in the recognition of affect. Psychological Science, 19(10), 984–985. doi: 10.1111/j.1467-9280.2008.02188.x.PubMedCrossRefGoogle Scholar
  52. van Baaren, R., Janssen, L., Chartrand, T. L., & Dijksterhuis, A. (2009). Where is the love? The social aspects of mimicry. Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences, 364(1528), 2381–2389. doi: 10.1098/rstb.2009.0057.PubMedPubMedCentralCrossRefGoogle Scholar
  53. Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070. doi: 10.1037/0022-3514.54.6.1063.PubMedCrossRefGoogle Scholar
  54. Weyers, P., Muhlberger, A., Kund, A., Hess, U., & Pauli, P. (2009). Modulation of facial reactions to avatar emotional faces by nonconscious competition priming. Psychophysiology, 46(2), 328–335. doi: 10.1111/j.1469-8986.2008.00771.x.PubMedCrossRefGoogle Scholar
  55. Wilhelm, O., Hildebrandt, A., Manske, K., Schacht, A., & Sommer, W. (2014). Test battery for measuring the perception and recognition of facial expressions of emotion. Frontiers in Psychology, 5, 404. doi: 10.3389/fpsyg.2014.00404.PubMedPubMedCentralCrossRefGoogle Scholar
  56. Winkielman, P., Niedenthal, P., Wielgosz, J., Eelen, J., & Kavanagh, L. C. (2015). Embodiment of cognition and emotion. APA Handbook of Personality and Social Psychology, 1, 151–175. doi: 10.1037/14341-004.Google Scholar
  57. Zaki, J., Bolger, N., & Ochsner, K. (2008). It takes two: The interpersonal nature of empathic accuracy. Psychological Science, 19(4), 399–404. doi: 10.1111/j.1467-9280.2008.02099.x.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of PsychologyHumboldt-Universität zu BerlinBerlinGermany
  2. 2.Department of PsychologyUlm UniversityUlmGermany
  3. 3.Psychologische Hochschule BerlinBerlinGermany

Personalised recommendations