Using Kinect for Facial Expression Recognition under Varying Poses and Illumination

  • Filip Malawski
  • Bogdan Kwolek
  • Shinji Sako
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8610)


Emotions analysis and recognition by the smartphones with front cameras is a relatively new concept. In this paper we present an algorithm that uses a low resolution 3D sensor for facial expression recognition. The 3D head pose as well as 3D location of the fiducial points are determined using Face Tracking SDK. Tens of the features are automatically selected from a pool determined by all possible line segments between such facial landmarks. We compared correctly classified ratios using features selected by AdaBoost, Lasso and histogram-based algorithms. We compared the classification accuracies obtained both on 3D maps and RGB images. Our results justify the feasibility of low accuracy 3D sensing devices for facial emotion recognition.


Facial Image Analysis Depth Maps Analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions: The state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1424–1445 (2000)CrossRefGoogle Scholar
  2. 2.
    Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3d facial expression recognition: A comprehensive survey. Image Vision Comput. 30(10), 683–697 (2012)CrossRefGoogle Scholar
  3. 3.
    Fang, T., Zhao, X., Ocegueda, O., Shah, S., Kakadiaris, I.A.: 3D facial expression recognition: A perspective on promises and challenges. In: FG, pp. 603–610 (2011)Google Scholar
  4. 4.
    Tsalakanidou, F., Malassiotis, S.: Real-time 2d+3d facial action and expression recognition. Pattern Recogn. 43(5), 1763–1775 (2010)CrossRefGoogle Scholar
  5. 5.
    Li, B., Mian, A., Liu, W., Krishna, A.: Using Kinect for face recognition under varying poses, expressions, illumination and disguise. In: WACV, pp. 186–192 (2013)Google Scholar
  6. 6.
    Brooks, R.A., Breazeal, C., Marjanovic, M., Scassellati, B., Williamson, M.M.: The cog project: Building a humanoid robot. In: Nehaniv, C.L. (ed.) CMAA 1998. LNCS (LNAI), vol. 1562, pp. 52–87. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  7. 7.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models - their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)CrossRefGoogle Scholar
  8. 8.
    Ahlberg, J.: Candide-3 - an updated parameterised face. Technical report, Dept. of Electrical Engineering, Linkping University, Sweden (2001)Google Scholar
  9. 9.
    Kwolek, B.: Model based facial pose tracking using a particle filter. In: Int. Conf. on Geometric Modeling and Imaging: New Trends, pp. 203–208. IEEE Comp. Soc. (2006)Google Scholar
  10. 10.
    Soyel, H., Demirel, H.: Optimal feature selection for 3D facial expression recognition with geometrically localized facial features. In: Int. Conf. on Soft Computing, Comp. with Words and Perceptions in Syst. Anal., Dec. and Control, pp. 1–4 (2009)Google Scholar
  11. 11.
    Sha, T., Song, M., Bu, J., Chen, C., Tao, D.: Feature level analysis for 3D facial expression recognition. Neurocomputing 74(12-13), 2135–2141 (2011)CrossRefGoogle Scholar
  12. 12.
    Soyel, H., Demirel, H.: Facial expression recognition using 3d facial feature distances. In: Kamel, M.S., Campilho, A. (eds.) ICIAR 2007. LNCS, vol. 4633, pp. 831–838. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  13. 13.
    Tang, H., Huang, T.: 3D facial expression recognition based on automatically selected features. In: Conf. CVPR, pp. 1–8 (2008)Google Scholar
  14. 14.
    Tang, H., Huang, T.: 3D facial expression recognition based on properties of line segments connecting facial feature points. In: Conf. FG, pp. 1–6 (2008)Google Scholar
  15. 15.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. In: Vitányi, P.M.B. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995)CrossRefGoogle Scholar
  16. 16.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  17. 17.
    Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B 58(1), 267–288 (1996)zbMATHMathSciNetGoogle Scholar
  18. 18.
    Hsu, C.W., Chang, C.C., Lin, C.J.: A practical guide to support vector classification. Technical report, Department of Computer Science, National Taiwan University, Taipei 106, Taiwan (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Filip Malawski
    • 1
  • Bogdan Kwolek
    • 1
  • Shinji Sako
    • 2
  1. 1.AGH University of Science and TechnologyKrakowPoland
  2. 2.Nagoya Institute of TechnologyJapan

Personalised recommendations