Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior

Abstract

Self-report studies have found evidence that cultures differ in the display rules they have for facial expressions (i.e., for what is appropriate for different people at different times). However, observational studies of actual patterns of facial behavior have been rare and typically limited to the analysis of dozens of participants from two or three regions. We present the first large-scale evidence of cultural differences in observed facial behavior, including 740,984 participants from 12 countries around the world. We used an Internet-based framework to collect video data of participants in two different settings: in their homes and in market research facilities. Using computer vision algorithms designed for this dataset, we measured smiling and brow furrowing expressions as participants watched television ads. Our results reveal novel findings and provide empirical evidence to support theories about cultural and gender differences in display rules. Participants from more individualist cultures displayed more brow furrowing overall, whereas smiling depended on both culture and setting. Specifically, participants from more individualist countries were more expressive in the facility setting, while participants from more collectivist countries were more expressive in the home setting. Female participants displayed more smiling and less brow furrowing than male participants overall, with the latter difference being more pronounced in more individualist countries. This is the first study to leverage advances in computer science to enable large-scale observational research that would not have been possible using traditional methods.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

Notes

  1. 1.

    The base rate is the proportion of time that a participant displays a facial action.

References

  1. Ansfield, M. E. (2007). Smiling when distressed: When a smile is a frown turned upside down. Personality and Social Psychology Bulletin, 33(6), 763–775.

    Article  PubMed  Google Scholar 

  2. Argyle, M., Henderson, M., Bond, M., Iizuka, Y., & Contarello, A. (1986). Cross-cultural variations in relationship rules. International Journal of Psychology, 21(1–4), 287–315.

    Article  Google Scholar 

  3. Brennan, R. L., & Prediger, D. J. (1981). Coefficient Kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687–699.

    Article  Google Scholar 

  4. Brewer, M. B., & Kramer, R. M. (1985). The psychology of intergroup attitudes and behavior. Annual Review of Psychology, 36(1), 219–243.

    Article  Google Scholar 

  5. Brody, L. R., & Hall, J. A. (2008). Gender and emotion in context. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 395–408). New York, NY: The Guilford Press.

    Google Scholar 

  6. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., & Robinson, E. S. J. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.

    Article  PubMed  Google Scholar 

  7. Cacioppo, J. T., Petty, R. E., Losch, M. E., & Kim, H. S. (1986). Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. Journal of Personality and Social Psychology, 50(2), 260.

    Article  PubMed  Google Scholar 

  8. Cacioppo, J. T., & Tassinary, L. G. (1990). Inferring psychological significance from physiological signals. The American Psychologist, 45(1), 16–28.

    Article  PubMed  Google Scholar 

  9. Camras, L. A., Bakeman, R., Chen, Y., Norris, K., & Cain, T. R. (2006). Culture, ethnicity, and children’s facial expressions: A study of European American, Mainland Chinese, Chinese American, and adopted Chinese girls. Emotion, 6(1), 103–114.

    Article  PubMed  Google Scholar 

  10. Camras, L. A., Oster, H., Campos, J., Campos, R., Ujiie, T., Miyake, K., et al. (1998). Production of emotional facial expressions in European American, Japanese, and Chinese infants. Developmental Psychology, 34(4), 616–628.

    Article  PubMed  Google Scholar 

  11. Casella, G., & George, E. I. (1992). Explaining the gibbs sampler. The American Statistician, 46(3), 167–174.

    Google Scholar 

  12. Cohn, J. F., & Ekman, P. (2005). Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of nonverbal behavior research (pp. 9–64). New York, NY: Oxford University Press.

    Google Scholar 

  13. Darwin, C. (1872). The expression of emotions in man and animals (3rd ed.). New York, NY: Oxford University.

    Google Scholar 

  14. Ekman, P. (1977). Biological and cultural contributions to body and facial movement. In J. Blacking (Ed.), The anthropology of the body. London: Academic Press.

    Google Scholar 

  15. Ekman, P. (1979). About brows: Emotional and conversational signals. In M. V. Cranach, K. Foppa, W. Lepenies, & D. Ploog (Eds.), Human ethology (pp. 169–202). Cambridge: Cambridge University Press.

    Google Scholar 

  16. Ekman, P. (1982). Methods for measuring facial action. In K. R. Scherer & P. Ekman (Eds.), Handbook of methods in nonverbal behavior research (pp. 45–90). Cambridge: Cambridge University Press.

    Google Scholar 

  17. Ekman, P., Davidson, R. J., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain physiology: II. Journal of Personality and Social Psychology, 58(2), 342–353.

    Article  PubMed  Google Scholar 

  18. Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, 124–129.

    Article  PubMed  Google Scholar 

  19. Ekman, P., Friesen, W. V., & Ellsworth, P. (1972). Emotion in the human face: Guidelines for research and an integration of findings. New York, NY: Pergamon Press.

    Google Scholar 

  20. Ekman, P., Friesen, W. V., & Hager, J. (2002). Facial action coding system: A technique for the measurement of facial movement. Salt Lake City, UT: Research Nexus.

    Google Scholar 

  21. Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., et al. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712–717.

    Article  PubMed  Google Scholar 

  22. Ekman, P., & Rosenberg, E. L. (2005). What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.). New York, NY: Oxford University Press.

    Google Scholar 

  23. Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164(3875), 86–88.

    Article  PubMed  Google Scholar 

  24. Evans, E. C. (1969). Physiognomies in the ancient world. Transactions of the American Philosophical Society, 59, 1–101.

    Article  Google Scholar 

  25. Fischer, A. H., & Manstead, A. S. R. (2000). The relation between gender and emotions in different cultures. In A. H. Fischer (Ed.), Gender and emotion: Social psychological perspectives (pp. 71–94). New York, NY: Cambridge University Press.

    Google Scholar 

  26. Fridlund, A. J. (1991). Sociality of solitary smiling: Potentiation by an implicit audience. Journal of Personality and Social Psychology, 60(2), 229–240.

    Article  Google Scholar 

  27. Friesen, W. V. (1973). Cultural differences in facial expressions in a social situation: An experimental test on the concept of display rules (Doctoral dissertation). University of California San Francisco.

  28. Frith, H., & Kitzinger, C. (1998). “Emotion work” as a participant re-source: A feminist analysis of young women’s talk-in-interaction. Sociology, 32, 299–320.

    Article  Google Scholar 

  29. Girard, J. M., & Cohn, J. F. (2016). A primer on observational measurement. Assessment, 23(4), 404–413.

    Article  PubMed  Google Scholar 

  30. Girard, J. M., Cohn, J. F., Jeni, L. A., Sayette, M. A., & De la Torre, F. (2015). Spontaneous facial expression in unscripted social interactions can be measured automatically. Behavior Research Methods, 47(4), 1136–1147.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32(10), 641–647.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Gosling, S. D., John, O. E., Craik, K. H., & Robins, R. W. (1998). Do people know how they behave? Self-reported act frequencies compared with on-line codings by observers. Journal of Personality and Social Psychology, 74(5), 1337–1349.

    Article  PubMed  Google Scholar 

  33. Hall, J. A., & Halberstadt, A. G. (1986). Smiling and gazing. In J. S. Hyde & M. C. Inn (Eds.), The psychology of gender: Advances through meta-analysis (pp. 136–185). Baltimore, MD: Johns Hopkins University Press.

    Google Scholar 

  34. Hecht, M. A., & LaFrance, M. (1998). License or obligation to smile: The effect of power and sex on amount and type of smiling. Personality and Social Psychology Bulletin, 24(12), 1332–1342.

    Article  Google Scholar 

  35. Heck, R. H., & Thomas, S. L. (2015). An introduction to multilevel modeling techniques: MLM and SEM approaches using Mplus (3rd ed.). New York, NY: Routledge.

    Google Scholar 

  36. Hess, U., Adams, R. B., Jr., & Kleck, R. E. (2005). Who may frown and who should smile? Dominance, affiliation, and the display of happiness and anger. Cognition and Emotion, 19(4), 515–536.

    Article  Google Scholar 

  37. Hofstede, G. H. (1980). Culture’s consequences: International differences in work-related values. Beverly Hills, CA: Sage Publications.

    Google Scholar 

  38. Hofstede, G. H. (2001). Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations (2nd ed.). Thousand Oaks: Sage Publications.

    Google Scholar 

  39. Hofstede, G. H., & McCrae, R. R. (2004). Personality and culture revisited: Linking traits and dimensions of culture. Cross-Cultural Research, 38(1), 52–88.

    Article  Google Scholar 

  40. Kagan, J., Arcus, D., Snidman, N., Feng, W. Y., Hendler, J., & Greene, S. (1994). Reactivity in infants: A cross-national comparison. Developmental Psychology, 30(3), 342–345.

    Article  Google Scholar 

  41. Kisilevsky, B. S., Hains, S. M. J., Lee, K., Muir, D. W., Xu, F., Fu, G., et al. (1998). The still-face effect in Chinese and Canadian 3- to 6-month-old infants. Developmental Psychology, 34(4), 629–639.

    Article  PubMed  Google Scholar 

  42. Klineberg, O. (1940). Social psychology. New York, NY: Holt.

    Google Scholar 

  43. LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003). The contingent smile: A meta-analysis of sex differences in smiling. Psychological Bulletin, 129(2), 305–334.

    Article  PubMed  Google Scholar 

  44. Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 224–253.

    Article  Google Scholar 

  45. Matsumoto, D. (1990). Cultural similarities and differences in display rules. Motivation and Emotion, 14(3), 195–214.

    Article  Google Scholar 

  46. Matsumoto, D. (2006). Culture and nonverbal behavior. In Handbook of nonverbal communication (pp. 219–236).

  47. Matsumoto, D., & Kupperbusch, C. (2001). Idiocentric and allocentric differences in emotional expression, experience, and the coherence between expression and experience. Asian Journal of Social Psychology, 4(2), 113–131.

    Article  Google Scholar 

  48. Matsumoto, D., & Willingham, B. (2006). The thrill of victory and the agony of defeat: Spontaneous expressions of medal winners of the 2004 Athens Olympic games. Journal of Personality and Social Psychology, 91(3), 568–581.

    Article  PubMed  Google Scholar 

  49. Matsumoto, D., Willingham, B., & Olide, A. (2009). Sequential dynamics and culturally-moderated facial expressions of emotion. Psychological Science, 20(10), 1269–1274.

    Article  PubMed  Google Scholar 

  50. Matsumoto, D., Yoo, S. H., & Fontaine, J. (2008). Mapping expressive differences around the world: The relationship between emotional display rules and individualism versus collectivism. Journal of Cross-Cultural Psychology, 39(1), 55–74.

    Article  Google Scholar 

  51. McCrae, R. R., & Costa, P. T. (1999). A five-factor theory of personality. In L. A. Pervin & O. John (Eds.), Handbook of personality: Theory and research (2nd ed., pp. 139–153). New York, NY: Guilford Press.

    Google Scholar 

  52. McDuff, D. (2014). Crowdsourcing affective responses for predicting media effectiveness (Doctoral dissertation). Massachusetts Institute of Technology.

  53. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.

    Article  Google Scholar 

  54. Messick, D. M., & Mackie, D. M. (1989). Intergroup relations. Annual Review of Psychology, 40, 45–81.

    Article  PubMed  Google Scholar 

  55. Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

    Google Scholar 

  56. Oster, H. (1978). Facial expression and affect development. In M. Lewis & L. A. Rosenblum (Eds.), The development of affect. New York, NY: Plenum Press.

    Google Scholar 

  57. Oster, H., Daily, L., & Goldenthal, P. (1989). Processing facial affect. In A. W. Young & H. D. Ellis (Eds.), Handbook of research on face processing (pp. 101–161). Amsterdam: Elsevier.

    Google Scholar 

  58. Oyserman, D., Coon, H. M., & Kemmelmeier, M. (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin, 128(1), 3.

    Article  PubMed  Google Scholar 

  59. Russell, J. A. (1994). Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102.

    PubMed  Google Scholar 

  60. Rychlowska, M., Miyamoto, Y., Matsumoto, D., Hess, U., Gilboa-Schechtman, E., Kamble, S., et al. (2015). Heterogeneity of long-history migration explains cultural differences in reports of emotional expressivity and the functions of smiles. Proceedings of the National Academy of Sciences, 112(19), E2429–E2436.

    Article  Google Scholar 

  61. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177.

    Article  PubMed  Google Scholar 

  62. Scherer, K. R., Wallbott, H. F., & Summerfield, A. B. (1986). Experiencing emotion: A cross-cultural study. New York, NY: Cambridge University Press.

    Google Scholar 

  63. Schwartz, G. E., Ahern, G. L., & Brown, S. L. (1979). Lateralized facial muscle response to positive and negative emotional stimuli. Psychophysiology, 16(6), 561–571.

    Article  PubMed  Google Scholar 

  64. Senechal, T., McDuff, D., & Kaliouby, R. (2015). Facial action unit detection using active learning and an efficient non-linear kernel approximation. In Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 10–18).

  65. Singelis, T. M. (1994). The measurement of independent and interdependent self-construals. Personality and Social Psychology Bulletin, 20(5), 580–591.

    Article  Google Scholar 

  66. Suh, E., Diener, E., Oishi, S., & Triandis, H. C. (1998). The shifting basis of life satisfaction judgments across cultures: Emotions versus norms. Journal of Personality and Social Psychology, 74(2), 482–493.

    Article  Google Scholar 

  67. Tajfel, H. (1982). Social psychology of intergroup relations. Annual Review of Psychology, 33(1), 1–39.

    Article  Google Scholar 

  68. Waxer, P. H. (1985). Video ethology: Television as a data base for cross-cultural studies in nonverbal displays. Journal of Nonverbal Behavior, 9(2), 111–120.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding authors

Correspondence to Daniel McDuff or Jeffrey M. Girard.

Additional information

Daniel McDuff and Jeffrey M. Girard have contributed equally to this study.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

McDuff, D., Girard, J.M. & Kaliouby, R.e. Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior. J Nonverbal Behav 41, 1–19 (2017). https://doi.org/10.1007/s10919-016-0244-x

Download citation

Keywords

  • Facial expression
  • Culture
  • Gender
  • Crowdsourcing
  • Computer vision