Facial Expression Recognition in Nonvisual Imagery

  • Gustavo Olague
  • Riad Hammoud
  • Leonardo Trujillo
  • Benjamín Hernández
  • Eva Romero
Part of the Advances in Pattern Recognition book series (ACVPR)


This chapter presents two novel approaches that allow computer vision applications to perform human facial expression recognition (FER). From a prob lem standpoint, we focus on FER beyond the human visual spectrum, in long-wave infrared imagery, thus allowing us to offer illumination-independent solutions to this important human-computer interaction problem. From a methodological stand point, we introduce two different feature extraction techniques: a principal com ponent analysis-based approach with automatic feature selection and one based on texture information selected by an evolutionary algorithm. In the former, facial fea tures are selected based on interest point clusters, and classification is carried out us ing eigenfeature information; in the latter, an evolutionary-based learning algorithm searches for optimal regions of interest and texture features based on classification accuracy. Both of these approaches use a support vector machine-committee for classification. Results show effective performance for both techniques, from which we can conclude that thermal imagery contains worthwhile information for the FER problem beyond the human visual spectrum.


Facial Feature Principle Component Analysis Thermal Image Interest Point Facial Expression Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Chapter's References

  1. 1.
    Maja Pantic and Leon J. M. Rothkrantz. Automatic analysis of facial expressions: The state of the art.IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1424–1445, 2000CrossRefGoogle Scholar
  2. 2.
    B. Fasel and J. Luettin. Automatic facial expression analysis: A survey.Pattern Recognition, 36(1):259–275, 2003zbMATHCrossRefGoogle Scholar
  3. 3.
    Fabrice Bourel, Claude C. Chibelushi, and Adrian A. Low. Robust facial expression recog nition using a state-based model of spatially-localised facial dynamics. InProceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition (FGR.02), pages 113–118, 2002Google Scholar
  4. 4.
    Maja Pantic and Leon J. M. Rothkrantz. An expert system for recognition of facial actions and their intensity.Image and Vision Computing, 18:881–905, 2000CrossRefGoogle Scholar
  5. 5.
    Ying-Li Tian, Takeo Kanade, and Jeffrey Cohn. Recognizing action units for facial expression analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2):97–115, February 2001CrossRefGoogle Scholar
  6. 6.
    Severine Dubuisson, Franck Davoine, and Jean Pierre Cocquerez. Automatic facial feature extraction and facial expression recognition. InAVBPA '01: Proceedings of the Third Interna tional Conference on Audio- and Video-Based Biometric Person Authentication, pages 121– 126, Springer-Verlag, London, 2001CrossRefGoogle Scholar
  7. 7.
    C. Padgett and G. Cottrell. Representing face images for emotion classification.Advances in Neural Information Processing Systems, 9, 1997Google Scholar
  8. 8.
    Gianluca Donato, Marian Stewart Bartlett, Joseph C. Hager, Paul Ekman, and Terrence J. Sejnowski. Classifying facial actions.IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10):974–989, 1999CrossRefGoogle Scholar
  9. 9.
    Michael J. Lyons, Julien Budynek, and Shigeru Akamatsu. Automatic classification of single facial images.IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(12):1357– 1362, 1999CrossRefGoogle Scholar
  10. 10.
    Matthew Turk and Alex Paul Pentland. Eigenfaces for recognition.Journal of Cognitive Neu-roscience, 3(1):71–86, 1991CrossRefGoogle Scholar
  11. 11.
    Joseph Wilder, P. Jonathon Phillips, Cunhong Jiang, and Stephen Wiener. Comparison of visi ble and infra-red imagery for face recognition. In2nd International Conference on Automatic Face and Gesture Recognition (FG '96), October 14–16, 1996, Killington, VT, pages 182– 191, 1996Google Scholar
  12. 12.
    F. Prokoski. History, current status, and future of infrared identification. InCVBVS '00: Pro ceedings of the IEEE Workshop on Computer Vision Beyond the Visible Spectrum: Methods and Applications (CVBVS 2000), page 5, IEEE Computer Society, Washington, DC, 2000CrossRefGoogle Scholar
  13. 13.
    Leonardo Trujillo, Gustavo Olague, Riad Hammoud, and Benjamín Hernández. Automatic feature localization in thermal images for facial expression recognition. InCVPR '05: Pro ceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)– Workshops, page 14, IEEE Computer Society, Washington, DC, 2005CrossRefGoogle Scholar
  14. 14.
    I. Pavlidis, J. Levine, and P. Baukol. Thermal image analysis for anxiety detection. InInterna tional Conference on Image Processing, volume 2, pages 315–318, 2001Google Scholar
  15. 15.
    Y. Sugimoto, Y. Yoshitomi, and S. Tomita. A method for detecting transitions of emotional states using a thermal facial image based on a synthesis of facial expressions.Journal of Robotics and Autonomous Systems, 31:147–160, 2000CrossRefGoogle Scholar
  16. 16.
    Y. Yoshitomi, S. Kim, T. Kawano, and T. Kitazoe. Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. InProceeding of the 2000 IEEE International Workshop on Robot and Human Interactive Communication, Osaka. Japan – September 27–29 2000, pages 178–183Google Scholar
  17. 17.
    Seong G. Kong, Jingu Heo, Besma R. Abidi, Joonki Paik, and Mongi A. Abidi. Recent advances in visual and infrared face recognition: a review.Computer Vision and Image Under standing, 97(1):103–135, 2005CrossRefGoogle Scholar
  18. 18.
    Séverine Dubuisson, Franck Davoine, and Jean Pierre Cocquerez. Automatic facial feature extraction and facial expression recognition. In Josef Bigün and Fabrizio Smeraldi, editors,AVBPA, volume 2091,Lecture Notes in Computer Science, pages 121–126, Springer, New York, 2001Google Scholar
  19. 19.
    Leonardo Trujillo and Gustavo Olague. Using evolution to learn how to perform interest point detection. InProceedings from the 18th International Conference on Pattern Recognition, volume 1, pages 211–214, IEEE Computer Society, Washington, DC, 2006CrossRefGoogle Scholar
  20. 20.
    Xin Chen, Patrick J. Flynn, and Kevin W. Bowyer. PCA-based face recognition in infrared imagery: Baseline and comparative studies. InAMFG '03: Proceedings of the IEEE Interna tional Workshop on Analysis and Modeling of Faces and Gestures, page 127, IEEE Computer Society, Washington, DC, 2003Google Scholar
  21. 21.
    Christopher J.C. Burges. A tutorial on support vector machines for pattern recognition.Data Mining and Knowledge Discovery, 2(2):121–167, 1998CrossRefGoogle Scholar
  22. 22.
    Zhong-Qiu Zhao, De-Shuang Huang, and Bing-Yu Sun. Human face recognition based on multi-features using neural networks committee.Pattern Recognition Letters, 25(12):1351– 1358, 2004CrossRefGoogle Scholar
  23. 23.
    J.H. Holland.Adaptation in Natural and Artificial Systems.University of Michigan Press, Ann Arbor, MI, 1975Google Scholar
  24. 24.
    David E. Goldberg.Genetic Algorithms in Search, Optimization, and Machine Learning.Addison-Wesley Professional, Reading, MA, January 1989zbMATHGoogle Scholar
  25. 25.
    J. Malik and P. Perona. Preattentive texture discrimination with early vision mechanisms.Op tical Society of America, 7:923–932, May 1990CrossRefGoogle Scholar
  26. 26.
    Jianchang Mao and Anil K. Jain. Texture classification and segmentation using multiresolution simultaneous autoregressive models.Pattern Recognition, 25(2):173–188, 1992CrossRefGoogle Scholar
  27. 27.
    B. Julesz and J.R. Bergen.Textons, the Fundamental Elements in Preattentive Vision and Perception of Textures. Kaufmann, Los Altos, CA, 1987Google Scholar
  28. 28.
    Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. A sparse texture representation us ing local affine regions.IEEE Transactions on Pattern Analysis ' Machine Intelligence, 27(8):1265–1278, 2005CrossRefGoogle Scholar
  29. 29.
    R.M. Haralick. Statistical and structural approaches to texture.Proceedings of the IEEE, 67:786–804, 1979CrossRefGoogle Scholar
  30. 30.
    J. Kjell. Comparative study of noise-tolerant texture classification.1994 IEEE International Conference on Systems, Man, and Cybernetics. “Humans, Information and Technology”, 3:2431–2436, October 1994Google Scholar
  31. 31.
    Peter Howarth and Stefan M. Rüger. Evaluation of texture features for content-based image retrieval. InACM International Conference on Image and Video Retrieval (CIVR), pages 326– 334, 2004Google Scholar
  32. 32.
    Philippe P. Ohanian and Richard C. Dubes. Performance evaluation for four classes of textural features.Pattern Recognition, 25(8):819–833, 1992CrossRefGoogle Scholar
  33. 33.
    Zehang Sun, George Bebis, and Ronald Miller. Object detection using feature subset selection.Pattern Recognition, 37(11):2165–2176, 2004CrossRefGoogle Scholar
  34. 34.
    Chih-Chung Chang and Chih-Jen Lin.LIBSVM: A Library for Support Vector Machines, MIT Press Journals-Neural Computation, 2001Google Scholar
  35. 35.
    Cyril Goutte. Note on free lunches and cross-validation.Neural Computation, 9(6):1245– 1249, 1997CrossRefGoogle Scholar
  36. 36.
    IEEE OTCBVS WS Series Bench; DOE University Research Program in Robotics under grant DOE-DE-FG02-86NE37968; DOD/TACOM/NAC/ARC Program under grant R01-1344-18; FAA/NSSA grant R01-1344-48/49; Office of Naval Research under grant N000143010022.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Gustavo Olague
    • 1
  • Riad Hammoud
    • 2
  • Leonardo Trujillo
    • 2
  • Benjamín Hernández
    • 2
  • Eva Romero
    • 2
  1. 1.Delphi CorporationDelphi Electronics & SafetyKokomoUSA
  2. 2.Centro de Investigación Científica y de Educación Superior de EnsenadaEnsenadaMéxico

Personalised recommendations