Exploring the Facial Expression Perception-Production Link Using Real-Time Automated Facial Expression Recognition

  • David M. Deriso
  • Josh Susskind
  • Jim Tanaka
  • Piotr Winkielman
  • John Herrington
  • Robert Schultz
  • Marian Bartlett
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7584)


Motor production may play an important role in learning to recognize facial expressions. The present study explores the influence of facial production training on the perception of facial expressions by employing a novel production training intervention built on feedback from automated facial expression recognition. We hypothesized that production training using the automated feedback system would improve an individual’s ability to identify dynamic emotional faces. Thirty-four participants were administered a dynamic expression recognition task before and after either interacting with a production training video game called the Emotion Mirror or playing a control video game. Consistent with the prediction that perceptual benefits are tied to expression production, individuals with high engagement in production training improved more than individuals with low engagement or individuals who did not receive production training. These results suggest that the visual-motor associations involved in expression production training are related to perceptual abilities. Additionally, this study demonstrates a novel application of computer vision for real-time facial expression intervention training.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Eisenberger, N., Cole, S.: Social neuroscience and health: neurophysiological mechanisms linking social ties with physical health. Nature Neuroscience (2012)Google Scholar
  2. 2.
    House, J., Landis, K., Umberson, D.: Social relationships and health. Science 241, 540–545 (1988)CrossRefGoogle Scholar
  3. 3.
    McIntosh, D., Reichmann-Decker, A., Winkielman, P., Wilbarger, J.: When the social mirror breaks: deficits in automatic, but not voluntary, mimicry of emotional facial expressions in autism. Developmental Science 9, 295–302 (2006)CrossRefGoogle Scholar
  4. 4.
    Adolphs, R., Sears, L., Piven, J.: Abnormal processing of social information from faces in autism. Journal of Cognitive Neuroscience 13, 232–240 (2001)CrossRefGoogle Scholar
  5. 5.
    Rump, K., Giovannelli, J., Minshew, N., Strauss, M.: The development of emotion recognition in individuals with autism. Child Development 80, 1434–1447 (2009)CrossRefGoogle Scholar
  6. 6.
    Madsen, M., El Kaliouby, R., Goodwin, M., Picard, R.: Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 19–26. ACM (2008)Google Scholar
  7. 7.
    Tanaka, J., Wolf, J., Klaiman, C., Koenig, K., Cockburn, J., Herlihy, L., Brown, C., Stahl, S., Kaiser, M., Schultz, R.: Using computerized games to teach face recognition skills to children with autism spectrum disorder: the lets face it! program. Journal of Child Psychology and Psychiatry 51, 944–952 (2010)CrossRefGoogle Scholar
  8. 8.
    Liberman, A., Mattingly, I.: The motor theory of speech perception revised. Cognition 21, 1–36 (1985)CrossRefGoogle Scholar
  9. 9.
    Rizzolatti, G., Craighero, L.: The mirror-neuron system. Annu. Rev. Neurosci. 27, 169–192 (2004)CrossRefGoogle Scholar
  10. 10.
    Heberlein, A., Adolphs, R.: Neurobiology of emotion recognition. Social Neuroscience: Integrating Biological and Psychological Explanations of Social Behavior 31 (2007)Google Scholar
  11. 11.
    Niedenthal, P., Mermillod, M., Maringer, M., Hess, U.: The simulation of smiles (sims) model: Embodied simulation and the meaning of facial expression. Behavioral and Brain Sciences 33, 417–433 (2010)CrossRefGoogle Scholar
  12. 12.
    Oberman, L., Winkielman, P., Ramachandran, V.: Face to face: Blocking facial mimicry can selectively impair recognition of emotional expressions. Social Neuroscience 2, 167–178 (2007)CrossRefGoogle Scholar
  13. 13.
    Pitcher, D., Garrido, L., Walsh, V., Duchaine, B.: Transcranial magnetic stimulation disrupts the perception and embodiment of facial expressions. The Journal of Neuroscience 28, 8929–8933 (2008)CrossRefGoogle Scholar
  14. 14.
    Montagne, B., Kessels, R., De Haan, E., Perrett, D.: The emotion recognition task: A paradigm to measure the perception of facial emotional expressions at different intensities. Perceptual and Motor Skills 104, 589–598 (2007)CrossRefGoogle Scholar
  15. 15.
    Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D., Hawk, S., Van Knippenberg, A.: Presentation and validation of the radboud faces database. Cognition and Emotion 24, 1377–1388 (2010)CrossRefGoogle Scholar
  16. 16.
    Ekman, P., Friesen, W.: Facial action coding system: A technique for the measurement of facial movement (1978)Google Scholar
  17. 17.
    Galantucci, B., Fowler, C., Turvey, M.: The motor theory of speech perception reviewed. Psychonomic Bulletin & Review 13, 361–377 (2006)CrossRefGoogle Scholar
  18. 18.
    Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)Google Scholar
  19. 19.
    Pelc, K., Kornreich, C., Foisy, M., Dan, B.: Recognition of emotional facial expressions in attention-deficit hyperactivity disorder. Pediatric Neurology 35, 93–97 (2006)CrossRefGoogle Scholar
  20. 20.
    Schyns, P., Petro, L., Smith, M.: Transmission of facial expressions of emotion co-evolved with their efficient decoding in the brain: behavioral and brain evidence. PLoS One 4, e5625 (2009)Google Scholar
  21. 21.
    Cribbie, R., Jamieson, J.: Decreases in posttest variance and the measurement of change. Methods of Psychological Research Online 9, 37–55 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • David M. Deriso
    • 1
  • Josh Susskind
    • 1
  • Jim Tanaka
    • 2
  • Piotr Winkielman
    • 3
  • John Herrington
    • 4
  • Robert Schultz
    • 4
  • Marian Bartlett
    • 1
  1. 1.Machine Perception LaboratoryUniversity of CaliforniaSan DiegoUSA
  2. 2.Department of PsychologyUniversity of VictoriaCanada
  3. 3.Department of PsychologyUniversity of CaliforniaSan DiegoUSA
  4. 4.Center for Autism ResearchChildren’s Hospital of PhiladelphiaUSA

Personalised recommendations