A Comparative Analysis of Thermal and Visual Modalities for Automated Facial Expression Recognition

  • Avinash Wesley
  • Pradeep Buddharaju
  • Robert Pienta
  • Ioannis Pavlidis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7432)


Facial expressions are formed through complicated muscular actions and can be taxonomized using the Facial Action Coding System (FACS). FACS breaks down human facial expressions into discreet action units (AUs) and often combines them together to form more elaborate expressions. In this paper, we present a comparative analysis of performance of automated facial expression recognition from thermal facial videos, visual facial videos, and their fusion. The feature extraction process consists of first placing regions of interest (ROIs) at 13 fiducial regions on the face that are critical for evaluating all action units, then extracting mean value in each of the ROIs, and finally applying principal component analysis (PCA) to extract the deviation from neutral expression at each of the corresponding ROIs. To classify facial expressions, we train a feed-forward multilayer perceptron with the standard deviation expression profiles obtained from the feature extraction stage. Our experimental results depicts that the thermal imaging modality outperforms visual modality, and hence overcomes some of the shortcomings usually noticed in the visual domain due to illumination and skin complexion variations. We have also shown that the decision level fusion of thermal and visual expression classification algorithms gives better results than either of the individual modalities.


Facial Expression Action Unit Visual Imagery Visual Modality Facial Expression Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Fasel, B., Luettin, J.: Automatic facial expression analysis: A survey. Pattern Recognition 36, 259–275 (1999)CrossRefGoogle Scholar
  2. 2.
    Pantic, M., Member, S., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1424–1445 (2000)CrossRefGoogle Scholar
  3. 3.
    Bartlett, M.S., Littlewort, G., Lainscsek, C., Fasel, I., Movellan, J.: Machine learning methods for fully automatic recognition of facial expressions and facial actions. In: Proc. IEEE Int’l Conf. Systems, Man and Cybernetics, pp. 592–597 (2004)Google Scholar
  4. 4.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C., Fasel, I., Movellan, J.: Recognizing facial expression: Machine learning and application to spontaneous behavior. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, San Diego, California, pp. 568–573. IEEE Computer Society (2005)Google Scholar
  5. 5.
    Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16, 172–187 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Whitehill, J., Littlewort, G., Pasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2106–2111 (2009)CrossRefGoogle Scholar
  7. 7.
    Socolinsky, D.A., Wolff, L.B., Neuheisel, J.D., Eveland, C.K.: Illumination invariant face recognition using thermal infrared imagery. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, p. 527 (2001)Google Scholar
  8. 8.
    Kong, S.G., Heo, J., Abidi, B.R., Paik, J., Abidi, M.A.: Recent advances in visual and infrared face recognition - a review. Computer Vision and Image Understanding 97, 103–135 (2005)CrossRefGoogle Scholar
  9. 9.
    Khan, M.M., Ingleby, M., Ward, R.D.: Automated facial expression classification and affect interpretation using infrared measurement of facial skin temperature variations. ACM Trans. Auton. Adapt. Syst. 1, 91–113 (2006)CrossRefGoogle Scholar
  10. 10.
    Yoshitomi, Y., Miyawaki, N., Tomita, S., Kimura, S.: Facial expression recognition using thermal image processing and neural network. Robot and Human Communication, 380–385 (1997)Google Scholar
  11. 11.
    Trujillo, L., Olague, G., Hammoud, R., Hernandez, B.: Automatic feature localization in thermal images for facial expression recognition. In: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2005) - Workshops, vol. 03, pp. 14–21. IEEE Computer Society, Washington, DC (2005)CrossRefGoogle Scholar
  12. 12.
    Socolinsky, D.A., Selinger, A.: Thermal face recognition over time. In: Proceedings of 17th International Conference on of the Pattern Recognition, ICPR 2004, vol. 4, pp. 187–190. IEEE Computer Society, Washington, DC (2004)CrossRefGoogle Scholar
  13. 13.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press (1978)Google Scholar
  14. 14.
    Ekman, P., Friesen, W., Hager, J.C.: Facial Action Coding System Investigator’s Guide (2002)Google Scholar
  15. 15.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explorations 11 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Avinash Wesley
    • 1
  • Pradeep Buddharaju
    • 1
  • Robert Pienta
    • 1
  • Ioannis Pavlidis
    • 1
  1. 1.University of Houston and Georgia Institute of TechnologyUSA

Personalised recommendations