Abstract
The computer vision field has advanced to the point that we are now able to begin to apply automatic facial expression recognition systems to important research questions in behavioral science. The machine perception lab at UC San Diego has developed a system based on machine learning for fully automated detection of 30 actions from the facial action coding system (FACS). The system, called Computer Expression Recognition Toolbox (CERT), operates in real-time and is robust to the video conditions in real applications. This paper describes two experiments which are the first applications of this system to analyzing spontaneous human behavior: Automated discrimination of posed from genuine expressions of pain, and automated detection of driver drowsiness. The analysis revealed information about facial behavior during these conditions that were previously unknown, including the coupling of movements. Automated classifiers were able to differentiate real from fake pain significantly better than naïve human subjects, and to detect critical drowsiness above 98% accuracy. Issues for application of machine learning systems to facial expression analysis are discussed.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bartlett, M.S., Littlewort, G.C., Frank, M.G., Lainscsek, C., Fasel, I., Movellan, J.R.: Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia 1(6), 22–35 (2006)
Cobb, W.: Recommendations for the practice of clinical neurophysiology. Elsevier, Amsterdam (1983)
Cohn, J.F., Schmidt, K.L.: The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing 2(2), 121–132 (2004)
Craig, K.D., Hyde, S., Patrick, C.J.: Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46, 161–172 (1991)
Craig, K.D., Patrick, C.J.: Facial expression during induced pain. J Pers Soc Psychol. 48(4), 1080–1091 (1985)
Donato, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Analysis and Machine Intelligence 21(10), 974–989 (1999)
DOT, Saving lives through advanced vehicle safety technology. USA Department of Transportation. (2001), http://www.its.dot.gov/ivi/docs/AR2001.pdf
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)
Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York (2001)
Ekman, P., Rosenberg, E.L. (eds.): What the face reveals: Basic and applied studies of spontaneous expression using the FACS. Oxford University Press, Oxford (2005)
Fasel, I., Fortenberry, B., Movellan, J.R.: A generative framework for real-time object detection and classification. Computer Vision and Image Understanding 98 (2005)
Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain 15(4), 244–274 (1999)
Fishbain, D.A., Cutler, R., Rosomoff, H.L., Rosomoff, R.S.: Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3), 214–234 (2006)
Frank, M.G., Ekman, P., Friesen, W.V.: Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1), 83–93 (1993)
Grossman, S., Shielder, V., Swedeen, K., Mucenski, J.: Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), 53–57 (1991)
Gu, H., Ji, Q.: An automated face reader for fatigue detection. In: FGR, pp. 111–116 (2004)
Gu, H., Zhang, Y., Ji, Q.: Task oriented facial behavior recognition with selective sensing. Comput. Vis. Image Underst. 100(3), 385–415 (2005)
Hadjistavropoulos, H.D., Craig, K.D., Hadjistavropoulos, T., Poole, G.D.: Subjective judgments of deception in pain expression: accuracy and errors. Pain 65(2-3), 251–258 (1996)
Hill, M.L., Craig, K.D.: Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain 98(1-2), 135–144 (2002)
Hong, C.K.: Electroencephalographic study of drowsiness in simulated driving with sleep deprivation. International Journal of Industrial Ergonomics 35(4), 307–320 (2005)
Igarashi, K., Takeda, K., Itakura, F., Abut, H.: DSP for In-Vehicle and Mobile Systems. Springer, US (2005)
Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the fourth IEEE International conference on automatic face and gesture recognition (FG 2000), Grenoble, France, pp. 46–53 (2000)
Larochette, A.C., Chambers, C.T., Craig, K.D.: Genuine, suppressed and faked facial expressions of pain in children. Pain 126(1-3), 64–71 (2006)
Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J.: Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing 24(6), 615–625 (2006)
Morecraft, R.J., Louie, J.L., Herrick, J.L., Stilwell-Morecraft, K.S.: Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1), 176–208 (2001)
Orden, K.F.V., Jung, T.P., Makeig, S.: Combined eye activity measures accurately estimate changes in sustained visual task performance. Biological Psychology 52(3), 221–240 (2000)
Pantic, M., Pentland, A., Nijholt, A., Huang, T.: Human Computing and machine understanding of human behaviour: A Survey. In: Proc. ACM Int’l Conf. Multimodal Interfaces, pp. 239–248 (2006)
Pantic, M.F.V., Rademaker, R., Maat, L.: Web- based Database for Facial Expression Analysis. In: Proc. IEEE Int’l Conf. Multmedia and Expo (ICME 2005), Amsterdam, The Netherlands (July 2005)
Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. Pain 51(3), 297–306 (1992)
Prkachin, K.M., Schultz, I., Berkowitz, J., Hughes, E., Hunt, D.: Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5), 595–607
Rinn, W.E.: The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95, 52–77
Schmand, B., Lindeboom, J., Schagen, S., Heijt, R., Koene, T., Hamburger, H.L.: Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry 64(3), 339–343
Schmidt, K.L., Cohn, J.F., Tian, Y.: Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1), 49–66 (2003)
Schneiderman, H., Kanade, T.: Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 45–51 (1998)
Takei, Y., Furukawa, Y.: Estimate of driver’s fatigue through steering motion. In: Man and Cybernetics, 2005 IEEE International Conference, vol. 2, pp. 1765–1770 (2005)
Viola, P., Jones, M.: Robust real-time face detection. J. Computer Vision 57(2), 137–154 (2004)
Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., Movellan, J.R.: Machine learning systems for detecting driver drowsiness. In: Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems (2007)
Zhang, Z., Shu Zhang, J.: Driver fatigue detection based intelligent vehicle control. In: Proceedings of the 18th International Conference on Pattern Recognition, Washington, DC, USA, pp. 1262–1265. IEEE Computer Society, Los Alamitos (2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bartlett, M. et al. (2008). Data Mining Spontaneous Facial Behavior with Automatic Expression Coding. In: Esposito, A., Bourbakis, N.G., Avouris, N., Hatzilygeroudis, I. (eds) Verbal and Nonverbal Features of Human-Human and Human-Machine Interaction. Lecture Notes in Computer Science(), vol 5042. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70872-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-540-70872-8_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-70871-1
Online ISBN: 978-3-540-70872-8
eBook Packages: Computer ScienceComputer Science (R0)