Spontaneous Facial Expression Recognition by Fusing Thermal Infrared and Visible Images

Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 194)


In this paper, we propose a method for spontaneous facial expression recognition by fusing features extracted from visible and thermal infrared images. First, the active appearance model parameters and head motion features are extracted from the visible images, and several thermal statistical parameters are extracted from the infrared images. Second, a multiple genetic algorithms-based fusion method is proposed for fusing these two spectrums. We use this proposed fusion method to search for the optimal combination of a similarity measurement and a feature subset. Then, a k-nearest neighbors classifier with the optimal combination is used to classify spontaneous facial expressions. Comparative experiments implemented on the Natural Visible and Infrared Facial Expression database show the effectiveness of the proposed similarity measurement and the feature selection method, and demonstrate the fusion method’s advantage over only using visible features.


facial expression recognition thermal infrared visible fusion genetic algorithm 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE TPAMI 31(1), 39–58 (2009)CrossRefGoogle Scholar
  2. 2.
    Hernández, B., Olague, G., Hammoud, R., Trujillo, L., Romero, E.: Visual learning of texture descriptors for facial expression recognition in thermal imagery. CVIU 106(2-3), 258–269 (2007)Google Scholar
  3. 3.
    Khan, M.M.: Cluster-analytic classification of facial expressions using infrared measurements of facial thermal features. Ph.D. Thesis, Department of Computing and Engineering, University of Huddersfield (2008)Google Scholar
  4. 4.
    Jarlier, S., Grandjean, D., Delplanque, S., N’Diaye, K., Cayeux, I., Velazco, M., Sander, D., Vuilleumier, P., Scherer, K.: Thermal analysis of facial muscles contractions. IEEE TAC 2(1), 2–9 (2011)Google Scholar
  5. 5.
    Yoshitomi, Y., Kim, S.I., Kawano, T., Kilazoe, T.: Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proceedings of the 9th IEEE International Workshop on Robot and Human Interactive Communication, pp. 178–183 (2000)Google Scholar
  6. 6.
    Tong, Y., Chen, J., Ji, Q.: A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE TPAMI 32(2), 258–273 (2010)CrossRefGoogle Scholar
  7. 7.
    Valstar, M.F., Gunes, H., Pantic, M.: How to distinguish posed from spontaneous smiles using geometric features. In: ICMI 2007, pp. 38–45 (November 2007)Google Scholar
  8. 8.
    Gunes, H., Pantic, M.: Dimensional Emotion Prediction from Spontaneous Head Gestures for Interaction with Sensitive Artificial Listeners. In: Allbeck, J., Badler, N., Bickmore, T., Pelachaud, C., Safonova, A. (eds.) IVA 2010. LNCS, vol. 6356, pp. 371–377. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE TMM 12(7), 682–691 (2010)Google Scholar
  10. 10.
    Yang, L.: An overview of distance metric learning. Technical report, School of Computer Science, Carnegie Mellon University (2007)Google Scholar
  11. 11.
    Dash, M., Liu, H.: Feature selection for classification. Intelligent Data Analysis 1(3), 131–156 (1997)CrossRefGoogle Scholar
  12. 12.
    Wang, S., Zhu, H.: Musical perceptual similarity estimation using interactive genetic algorithm. In: IEEE CEC 2010, pp. 1–7 (July 2010)Google Scholar
  13. 13.
    Tong, Y., Wang, Y., Zhu, Z., Ji, Q.: Robust facial feature tracking under varying face pose and facial expression. Pattern Recognition 40(11), 3195–3208 (2007)MATHCrossRefGoogle Scholar
  14. 14.
    Lv, Y., Wang, S.: A spontaneous facial expression recognition method using head motion and aam features. In: 2010 Second World Congress on NaBIC, pp. 334–339 (December 2010)Google Scholar
  15. 15.
    Lucey, S., Ashraf, A.B., Cohn, J.: Investigating spontaneous facial action recognition through aam representations of the face. Face Recognition, 275–286 (2007)Google Scholar
  16. 16.
  17. 17.
    Sebe, N., Lew, M.S., Sun, Y., Cohen, I., Gevers, T., Huang, T.S.: Authentic facial expression analysis. Image and Vision Computing 25(12), 1856–1863 (2007)CrossRefGoogle Scholar
  18. 18.
    Li, L., Darden, T.A., Weingberg, C.R., Levine, A.J., Pedersen, L.G.: Gene assessment and sample classification for gene expression data using a genetic algorithm/k-nearest neighbor method. Combinatorial Chemistry and High Throughput Screening 4(8), 727–739 (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Key Lab of Computing and Communicating Software of Anhui Province School of Computer Science and TechnologyUniversity of Science and Technology of ChinaHefeiP.R. China

Personalised recommendations