Journal of Nonverbal Behavior

, Volume 41, Issue 1, pp 1–19 | Cite as

Large-Scale Observational Evidence of Cross-Cultural Differences in Facial Behavior

  • Daniel McDuffEmail author
  • Jeffrey M. GirardEmail author
  • Rana el Kaliouby
Original Paper


Self-report studies have found evidence that cultures differ in the display rules they have for facial expressions (i.e., for what is appropriate for different people at different times). However, observational studies of actual patterns of facial behavior have been rare and typically limited to the analysis of dozens of participants from two or three regions. We present the first large-scale evidence of cultural differences in observed facial behavior, including 740,984 participants from 12 countries around the world. We used an Internet-based framework to collect video data of participants in two different settings: in their homes and in market research facilities. Using computer vision algorithms designed for this dataset, we measured smiling and brow furrowing expressions as participants watched television ads. Our results reveal novel findings and provide empirical evidence to support theories about cultural and gender differences in display rules. Participants from more individualist cultures displayed more brow furrowing overall, whereas smiling depended on both culture and setting. Specifically, participants from more individualist countries were more expressive in the facility setting, while participants from more collectivist countries were more expressive in the home setting. Female participants displayed more smiling and less brow furrowing than male participants overall, with the latter difference being more pronounced in more individualist countries. This is the first study to leverage advances in computer science to enable large-scale observational research that would not have been possible using traditional methods.


Facial expression Culture Gender Crowdsourcing Computer vision 


  1. Ansfield, M. E. (2007). Smiling when distressed: When a smile is a frown turned upside down. Personality and Social Psychology Bulletin, 33(6), 763–775.CrossRefPubMedGoogle Scholar
  2. Argyle, M., Henderson, M., Bond, M., Iizuka, Y., & Contarello, A. (1986). Cross-cultural variations in relationship rules. International Journal of Psychology, 21(1–4), 287–315.CrossRefGoogle Scholar
  3. Brennan, R. L., & Prediger, D. J. (1981). Coefficient Kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 41(3), 687–699.CrossRefGoogle Scholar
  4. Brewer, M. B., & Kramer, R. M. (1985). The psychology of intergroup attitudes and behavior. Annual Review of Psychology, 36(1), 219–243.CrossRefGoogle Scholar
  5. Brody, L. R., & Hall, J. A. (2008). Gender and emotion in context. In M. Lewis, J. M. Haviland-Jones, & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 395–408). New York, NY: The Guilford Press.Google Scholar
  6. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., & Robinson, E. S. J. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376.CrossRefPubMedGoogle Scholar
  7. Cacioppo, J. T., Petty, R. E., Losch, M. E., & Kim, H. S. (1986). Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. Journal of Personality and Social Psychology, 50(2), 260.CrossRefPubMedGoogle Scholar
  8. Cacioppo, J. T., & Tassinary, L. G. (1990). Inferring psychological significance from physiological signals. The American Psychologist, 45(1), 16–28.CrossRefPubMedGoogle Scholar
  9. Camras, L. A., Bakeman, R., Chen, Y., Norris, K., & Cain, T. R. (2006). Culture, ethnicity, and children’s facial expressions: A study of European American, Mainland Chinese, Chinese American, and adopted Chinese girls. Emotion, 6(1), 103–114.CrossRefPubMedGoogle Scholar
  10. Camras, L. A., Oster, H., Campos, J., Campos, R., Ujiie, T., Miyake, K., et al. (1998). Production of emotional facial expressions in European American, Japanese, and Chinese infants. Developmental Psychology, 34(4), 616–628.CrossRefPubMedGoogle Scholar
  11. Casella, G., & George, E. I. (1992). Explaining the gibbs sampler. The American Statistician, 46(3), 167–174.Google Scholar
  12. Cohn, J. F., & Ekman, P. (2005). Measuring facial action by manual coding, facial EMG, and automatic facial image analysis. In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of nonverbal behavior research (pp. 9–64). New York, NY: Oxford University Press.Google Scholar
  13. Darwin, C. (1872). The expression of emotions in man and animals (3rd ed.). New York, NY: Oxford University.CrossRefGoogle Scholar
  14. Ekman, P. (1977). Biological and cultural contributions to body and facial movement. In J. Blacking (Ed.), The anthropology of the body. London: Academic Press.Google Scholar
  15. Ekman, P. (1979). About brows: Emotional and conversational signals. In M. V. Cranach, K. Foppa, W. Lepenies, & D. Ploog (Eds.), Human ethology (pp. 169–202). Cambridge: Cambridge University Press.Google Scholar
  16. Ekman, P. (1982). Methods for measuring facial action. In K. R. Scherer & P. Ekman (Eds.), Handbook of methods in nonverbal behavior research (pp. 45–90). Cambridge: Cambridge University Press.Google Scholar
  17. Ekman, P., Davidson, R. J., & Friesen, W. V. (1990). The Duchenne smile: Emotional expression and brain physiology: II. Journal of Personality and Social Psychology, 58(2), 342–353.CrossRefPubMedGoogle Scholar
  18. Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17, 124–129.CrossRefPubMedGoogle Scholar
  19. Ekman, P., Friesen, W. V., & Ellsworth, P. (1972). Emotion in the human face: Guidelines for research and an integration of findings. New York, NY: Pergamon Press.Google Scholar
  20. Ekman, P., Friesen, W. V., & Hager, J. (2002). Facial action coding system: A technique for the measurement of facial movement. Salt Lake City, UT: Research Nexus.Google Scholar
  21. Ekman, P., Friesen, W. V., O’Sullivan, M., Chan, A., Diacoyanni-Tarlatzis, I., Heider, K., et al. (1987). Universals and cultural differences in the judgments of facial expressions of emotion. Journal of Personality and Social Psychology, 53(4), 712–717.CrossRefPubMedGoogle Scholar
  22. Ekman, P., & Rosenberg, E. L. (2005). What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.). New York, NY: Oxford University Press.CrossRefGoogle Scholar
  23. Ekman, P., Sorenson, E. R., & Friesen, W. V. (1969). Pan-cultural elements in facial displays of emotion. Science, 164(3875), 86–88.CrossRefPubMedGoogle Scholar
  24. Evans, E. C. (1969). Physiognomies in the ancient world. Transactions of the American Philosophical Society, 59, 1–101.CrossRefGoogle Scholar
  25. Fischer, A. H., & Manstead, A. S. R. (2000). The relation between gender and emotions in different cultures. In A. H. Fischer (Ed.), Gender and emotion: Social psychological perspectives (pp. 71–94). New York, NY: Cambridge University Press.CrossRefGoogle Scholar
  26. Fridlund, A. J. (1991). Sociality of solitary smiling: Potentiation by an implicit audience. Journal of Personality and Social Psychology, 60(2), 229–240.CrossRefGoogle Scholar
  27. Friesen, W. V. (1973). Cultural differences in facial expressions in a social situation: An experimental test on the concept of display rules (Doctoral dissertation). University of California San Francisco.Google Scholar
  28. Frith, H., & Kitzinger, C. (1998). “Emotion work” as a participant re-source: A feminist analysis of young women’s talk-in-interaction. Sociology, 32, 299–320.CrossRefGoogle Scholar
  29. Girard, J. M., & Cohn, J. F. (2016). A primer on observational measurement. Assessment, 23(4), 404–413.CrossRefPubMedGoogle Scholar
  30. Girard, J. M., Cohn, J. F., Jeni, L. A., Sayette, M. A., & De la Torre, F. (2015). Spontaneous facial expression in unscripted social interactions can be measured automatically. Behavior Research Methods, 47(4), 1136–1147.CrossRefPubMedPubMedCentralGoogle Scholar
  31. Girard, J. M., Cohn, J. F., Mahoor, M. H., Mavadati, S. M., Hammal, Z., & Rosenwald, D. P. (2014). Nonverbal social withdrawal in depression: Evidence from manual and automatic analyses. Image and Vision Computing, 32(10), 641–647.CrossRefPubMedPubMedCentralGoogle Scholar
  32. Gosling, S. D., John, O. E., Craik, K. H., & Robins, R. W. (1998). Do people know how they behave? Self-reported act frequencies compared with on-line codings by observers. Journal of Personality and Social Psychology, 74(5), 1337–1349.CrossRefPubMedGoogle Scholar
  33. Hall, J. A., & Halberstadt, A. G. (1986). Smiling and gazing. In J. S. Hyde & M. C. Inn (Eds.), The psychology of gender: Advances through meta-analysis (pp. 136–185). Baltimore, MD: Johns Hopkins University Press.Google Scholar
  34. Hecht, M. A., & LaFrance, M. (1998). License or obligation to smile: The effect of power and sex on amount and type of smiling. Personality and Social Psychology Bulletin, 24(12), 1332–1342.CrossRefGoogle Scholar
  35. Heck, R. H., & Thomas, S. L. (2015). An introduction to multilevel modeling techniques: MLM and SEM approaches using Mplus (3rd ed.). New York, NY: Routledge.Google Scholar
  36. Hess, U., Adams, R. B., Jr., & Kleck, R. E. (2005). Who may frown and who should smile? Dominance, affiliation, and the display of happiness and anger. Cognition and Emotion, 19(4), 515–536.CrossRefGoogle Scholar
  37. Hofstede, G. H. (1980). Culture’s consequences: International differences in work-related values. Beverly Hills, CA: Sage Publications.Google Scholar
  38. Hofstede, G. H. (2001). Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations (2nd ed.). Thousand Oaks: Sage Publications.Google Scholar
  39. Hofstede, G. H., & McCrae, R. R. (2004). Personality and culture revisited: Linking traits and dimensions of culture. Cross-Cultural Research, 38(1), 52–88.CrossRefGoogle Scholar
  40. Kagan, J., Arcus, D., Snidman, N., Feng, W. Y., Hendler, J., & Greene, S. (1994). Reactivity in infants: A cross-national comparison. Developmental Psychology, 30(3), 342–345.CrossRefGoogle Scholar
  41. Kisilevsky, B. S., Hains, S. M. J., Lee, K., Muir, D. W., Xu, F., Fu, G., et al. (1998). The still-face effect in Chinese and Canadian 3- to 6-month-old infants. Developmental Psychology, 34(4), 629–639.CrossRefPubMedGoogle Scholar
  42. Klineberg, O. (1940). Social psychology. New York, NY: Holt.CrossRefGoogle Scholar
  43. LaFrance, M., Hecht, M. A., & Paluck, E. L. (2003). The contingent smile: A meta-analysis of sex differences in smiling. Psychological Bulletin, 129(2), 305–334.CrossRefPubMedGoogle Scholar
  44. Markus, H. R., & Kitayama, S. (1991). Culture and the self: Implications for cognition, emotion, and motivation. Psychological Review, 98(2), 224–253.CrossRefGoogle Scholar
  45. Matsumoto, D. (1990). Cultural similarities and differences in display rules. Motivation and Emotion, 14(3), 195–214.CrossRefGoogle Scholar
  46. Matsumoto, D. (2006). Culture and nonverbal behavior. In Handbook of nonverbal communication (pp. 219–236).Google Scholar
  47. Matsumoto, D., & Kupperbusch, C. (2001). Idiocentric and allocentric differences in emotional expression, experience, and the coherence between expression and experience. Asian Journal of Social Psychology, 4(2), 113–131.CrossRefGoogle Scholar
  48. Matsumoto, D., & Willingham, B. (2006). The thrill of victory and the agony of defeat: Spontaneous expressions of medal winners of the 2004 Athens Olympic games. Journal of Personality and Social Psychology, 91(3), 568–581.CrossRefPubMedGoogle Scholar
  49. Matsumoto, D., Willingham, B., & Olide, A. (2009). Sequential dynamics and culturally-moderated facial expressions of emotion. Psychological Science, 20(10), 1269–1274.CrossRefPubMedGoogle Scholar
  50. Matsumoto, D., Yoo, S. H., & Fontaine, J. (2008). Mapping expressive differences around the world: The relationship between emotional display rules and individualism versus collectivism. Journal of Cross-Cultural Psychology, 39(1), 55–74.CrossRefGoogle Scholar
  51. McCrae, R. R., & Costa, P. T. (1999). A five-factor theory of personality. In L. A. Pervin & O. John (Eds.), Handbook of personality: Theory and research (2nd ed., pp. 139–153). New York, NY: Guilford Press.Google Scholar
  52. McDuff, D. (2014). Crowdsourcing affective responses for predicting media effectiveness (Doctoral dissertation). Massachusetts Institute of Technology.Google Scholar
  53. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.CrossRefGoogle Scholar
  54. Messick, D. M., & Mackie, D. M. (1989). Intergroup relations. Annual Review of Psychology, 40, 45–81.CrossRefPubMedGoogle Scholar
  55. Muthén, L. K., & Muthén, B. O. (2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
  56. Oster, H. (1978). Facial expression and affect development. In M. Lewis & L. A. Rosenblum (Eds.), The development of affect. New York, NY: Plenum Press.Google Scholar
  57. Oster, H., Daily, L., & Goldenthal, P. (1989). Processing facial affect. In A. W. Young & H. D. Ellis (Eds.), Handbook of research on face processing (pp. 101–161). Amsterdam: Elsevier.Google Scholar
  58. Oyserman, D., Coon, H. M., & Kemmelmeier, M. (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin, 128(1), 3.CrossRefPubMedGoogle Scholar
  59. Russell, J. A. (1994). Is there universal recognition of emotion from facial expressions? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102.PubMedGoogle Scholar
  60. Rychlowska, M., Miyamoto, Y., Matsumoto, D., Hess, U., Gilboa-Schechtman, E., Kamble, S., et al. (2015). Heterogeneity of long-history migration explains cultural differences in reports of emotional expressivity and the functions of smiles. Proceedings of the National Academy of Sciences, 112(19), E2429–E2436.CrossRefGoogle Scholar
  61. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177.CrossRefPubMedGoogle Scholar
  62. Scherer, K. R., Wallbott, H. F., & Summerfield, A. B. (1986). Experiencing emotion: A cross-cultural study. New York, NY: Cambridge University Press.Google Scholar
  63. Schwartz, G. E., Ahern, G. L., & Brown, S. L. (1979). Lateralized facial muscle response to positive and negative emotional stimuli. Psychophysiology, 16(6), 561–571.CrossRefPubMedGoogle Scholar
  64. Senechal, T., McDuff, D., & Kaliouby, R. (2015). Facial action unit detection using active learning and an efficient non-linear kernel approximation. In Proceedings of the IEEE International Conference on Computer Vision Workshops (pp. 10–18).Google Scholar
  65. Singelis, T. M. (1994). The measurement of independent and interdependent self-construals. Personality and Social Psychology Bulletin, 20(5), 580–591.CrossRefGoogle Scholar
  66. Suh, E., Diener, E., Oishi, S., & Triandis, H. C. (1998). The shifting basis of life satisfaction judgments across cultures: Emotions versus norms. Journal of Personality and Social Psychology, 74(2), 482–493.CrossRefGoogle Scholar
  67. Tajfel, H. (1982). Social psychology of intergroup relations. Annual Review of Psychology, 33(1), 1–39.CrossRefGoogle Scholar
  68. Waxer, P. H. (1985). Video ethology: Television as a data base for cross-cultural studies in nonverbal displays. Journal of Nonverbal Behavior, 9(2), 111–120.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Affectiva, Inc.WalthamUSA
  2. 2.Department of PsychologyUniversity of PittsburghPittsburghUSA

Personalised recommendations