Data Mining Spontaneous Facial Behavior with Automatic Expression Coding

  • Marian Bartlett
  • Gwen Littlewort
  • Esra Vural
  • Kang Lee
  • Mujdat Cetin
  • Aytul Ercil
  • Javier Movellan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5042)


The computer vision field has advanced to the point that we are now able to begin to apply automatic facial expression recognition systems to important research questions in behavioral science. The machine perception lab at UC San Diego has developed a system based on machine learning for fully automated detection of 30 actions from the facial action coding system (FACS). The system, called Computer Expression Recognition Toolbox (CERT), operates in real-time and is robust to the video conditions in real applications. This paper describes two experiments which are the first applications of this system to analyzing spontaneous human behavior: Automated discrimination of posed from genuine expressions of pain, and automated detection of driver drowsiness. The analysis revealed information about facial behavior during these conditions that were previously unknown, including the coupling of movements. Automated classifiers were able to differentiate real from fake pain significantly better than naïve human subjects, and to detect critical drowsiness above 98% accuracy.  Issues for application of machine learning systems to facial expression analysis are discussed.


Facial expression recognition machine learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek, C., Fasel, I., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1(6), 22–35 (2006)CrossRefGoogle Scholar
  2. 2.
    Cobb, W.: Recommendations for the practice of clinical neurophysiology. Elsevier, Amsterdam (1983)Google Scholar
  3. 3.
    Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing 2(2), 121–132 (2004)CrossRefGoogle Scholar
  4. 4.
    Craig, K.D., Hyde, S., Patrick, C.J.: Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46, 161–172 (1991)CrossRefGoogle Scholar
  5. 5.
    Craig, K.D., Patrick, C.J.: Facial expression during induced pain. J Pers Soc Psychol. 48(4), 1080–1091 (1985)CrossRefGoogle Scholar
  6. 6.
    Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)CrossRefGoogle Scholar
  7. 7.
    DOT, Saving lives through advanced vehicle safety technology. USA Department of Transportation. (2001),
  8. 8.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  9. 9.
    Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York (2001)Google Scholar
  10. 10.
    Ekman, P., Rosenberg, E.L. (eds.): What the face reveals: Basic and applied studies of spontaneous expression using the FACS. Oxford University Press, Oxford (2005)Google Scholar
  11. 11.
    Fasel, I., Fortenberry, B., Movellan, J.R.: A generative framework for real-time object detection and classification. Computer Vision and Image Understanding 98 (2005)Google Scholar
  12. 12.
    Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain 15(4), 244–274 (1999)CrossRefGoogle Scholar
  13. 13.
    Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3), 214–234 (2006)CrossRefGoogle Scholar
  14. 14.
    Frank, M.G., Ekman, P., Friesen, W.V.: Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1), 83–93 (1993)CrossRefGoogle Scholar
  15. 15.
    Grossman, S., Shielder, V., Swedeen, K., Mucenski, J.: Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), 53–57 (1991)CrossRefGoogle Scholar
  16. 16.
    Gu, H., Ji, Q.: An automated face reader for fatigue detection. In: FGR, pp. 111–116 (2004)Google Scholar
  17. 17.
    Gu, H., Zhang, Y., Ji, Q.: Task oriented facial behavior recognition with selective sensing. Comput. Vis. Image Underst. 100(3), 385–415 (2005)CrossRefGoogle Scholar
  18. 18.
    Hadjistavropoulos, H.D., Craig, K.D., Hadjistavropoulos, T., Poole, G.D.: Subjective judgments of deception in pain expression: accuracy and errors. Pain 65(2-3), 251–258 (1996)CrossRefGoogle Scholar
  19. 19.
    Hill, M.L., Craig, K.D.: Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain 98(1-2), 135–144 (2002)CrossRefGoogle Scholar
  20. 20.
    Hong, C.K.: Electroencephalographic study of drowsiness in simulated driving with sleep deprivation. International Journal of Industrial Ergonomics 35(4), 307–320 (2005)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Igarashi, K., Takeda, K., Itakura, F., Abut, H.: DSP for In-Vehicle and Mobile Systems. Springer, US (2005)Google Scholar
  22. 22.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE International conference on automatic face and gesture recognition (FG 2000), Grenoble, France, pp. 46–53 (2000)Google Scholar
  23. 23.
    Larochette, A.C., Chambers, C.T., Craig, K.D.: Genuine, suppressed and faked facial expressions of pain in children. Pain 126(1-3), 64–71 (2006)CrossRefGoogle Scholar
  24. 24.
    Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing 24(6), 615–625 (2006)CrossRefGoogle Scholar
  25. 25.
    Morecraft, R.J., Louie, J.L., Herrick, J.L., Stilwell-Morecraft, K.S.: Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1), 176–208 (2001)CrossRefGoogle Scholar
  26. 26.
    Orden, K.F.V., Jung, T.P., Makeig, S.: Combined eye activity measures accurately estimate changes in sustained visual task performance. Biological Psychology 52(3), 221–240 (2000)CrossRefGoogle Scholar
  27. 27.
    Pantic, M., Pentland, A., Nijholt, A., Huang, T.: Human Computing and machine understanding of human behaviour: A Survey. In: Proc. ACM Int’l Conf. Multimodal Interfaces, pp. 239–248 (2006)Google Scholar
  28. 28.
    Pantic, M.F.V., Rademaker, R., Maat, L.: Web- based Database for Facial Expression Analysis. In: Proc. IEEE Int’l Conf. Multmedia and Expo (ICME 2005), Amsterdam, The Netherlands (July 2005)Google Scholar
  29. 29.
    Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. Pain 51(3), 297–306 (1992)CrossRefGoogle Scholar
  30. 30.
    Prkachin, K.M., Schultz, I., Berkowitz, J., Hughes, E., Hunt, D.: Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5), 595–607Google Scholar
  31. 31.
    Rinn, W.E.: The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95, 52–77Google Scholar
  32. 32.
    Schmand, B., Lindeboom, J., Schagen, S., Heijt, R., Koene, T., Hamburger, H.L.: Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry 64(3), 339–343Google Scholar
  33. 33.
    Schmidt, K.L., Cohn, J.F., Tian, Y.: Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1), 49–66 (2003)CrossRefGoogle Scholar
  34. 34.
    Schneiderman, H., Kanade, T.: Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 45–51 (1998)Google Scholar
  35. 35.
    Takei, Y., Furukawa, Y.: Estimate of driver’s fatigue through steering motion. In: Man and Cybernetics, 2005 IEEE International Conference, vol. 2, pp. 1765–1770 (2005)Google Scholar
  36. 36.
    Viola, P., Jones, M.: Robust real-time face detection. J. Computer Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  37. 37.
    Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., Movellan, J.R.: Machine learning systems for detecting driver drowsiness. In: Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems (2007)Google Scholar
  38. 38.
    Zhang, Z., Shu Zhang, J.: Driver fatigue detection based intelligent vehicle control. In: Proceedings of the 18th International Conference on Pattern Recognition, Washington, DC, USA, pp. 1262–1265. IEEE Computer Society, Los Alamitos (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Marian Bartlett
    • 1
  • Gwen Littlewort
    • 1
  • Esra Vural
    • 1
    • 3
  • Kang Lee
    • 2
  • Mujdat Cetin
    • 3
  • Aytul Ercil
    • 3
  • Javier Movellan
    • 1
  1. 1.Institute for Neural ComputationUniversity of CaliforniaSan Diego, La JollaUSA
  2. 2.Human Development and Applied PsychologyUniversity of TorontoOntarioCanada
  3. 3.Engineering and Natural ScienceSabanci UniversityIstanbulTurkey

Personalised recommendations