Facial Expression Recognition Using Local Region Specific Dense Optical Flow and LBP Features

  • Deepak GhimireEmail author
  • Sang Hyun Park
  • Mi Jin Kim
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10617)


Recognition of facial expression has many applications including human-computer interaction, human emotion analysis, personality development, cognitive science, health-care, virtual reality, image retrieval, etc. In this paper we propose a new method for recognition of facial expression using local region specific mean optical flow and local binary pattern feature descriptor with support vector machine classification. In general, facial expression recognition techniques divide the face into regular grid (holistic representation) and the facial features are extracted. However, in this paper we divide the face into domain specific local regions. At first a robust optical flow is utilized to get mean optical flow in different directions for each local region which considers both local statistic motion information and its spatial location. The features are used only from the key frames; which are detected based on maximal mean optical flow magnitude within a sequence w.r.t. neutral frame. Now, the region specific local binary pattern is extracted from key frame and concatenated with mean optical flow features. The performance of the proposed facial expression recognition system has been validated on CK+ facial expression dataset.


Facial expression Local representation Optical flow Local binary pattern Support vector machines 



This work was supported by the Technology Innovation Program (10052289, Development of HD high-reliability stereo ADAS vision system) funded By the Ministry of Trade, Industry & Energy (MI, Korea).


  1. 1.
    Mehrabian, A.: Communication without words. Psychol. Today 2, 53–56 (1968)Google Scholar
  2. 2.
    Ekman, P.: Strong evidence of universal in facial expressions: a reply to Russell’s mistaken critique. Psychol. Bull. 115, 268–287 (1994)CrossRefGoogle Scholar
  3. 3.
    Ghimire, D., Jeong, S., Lee, J., Park, S.-H.: Facial expression recognition based on local region specific features and support vector machines. Multimed. Tools Appl. 76(6), 7803–7821 (2017)CrossRefGoogle Scholar
  4. 4.
    Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13, 7714–7734 (2013)CrossRefGoogle Scholar
  5. 5.
    Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimed. Tools Appl. 74(15), 5577–5615 (2014)CrossRefGoogle Scholar
  6. 6.
    Ghimire, D., Lee, J., Li, Z.-N., Jeong, S.: Recognition of facial expressions based on salient geometric features and support vector machines. Multimed. Tools Appl. 76(6), 7921–7946 (2017)CrossRefGoogle Scholar
  7. 7.
    Ghimire, D., Lee, J.: Histogram of orientation gradient feature-based facial expression classification using bagging with extreme learning machine. Adv. Sci. Lett. 16, 156–161 (2012)CrossRefGoogle Scholar
  8. 8.
    Cruz, A.C., Bhanu, B., Thakoor, N.S.: Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans. Affect. Comput. 5(4), 418–431 (2014)CrossRefGoogle Scholar
  9. 9.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009)CrossRefGoogle Scholar
  10. 10.
    Zhang, S., Zhao, X., Lei, B.: Robust facial expression recognition via compressive sensing. Sensors 12, 3747–3761 (2012)CrossRefGoogle Scholar
  11. 11.
    Ghimire, D., Lee, J.: Extreme learning machine ensemble using bagging for facial expression recognition. J. Inf. Process. Syst. 10, 443–458 (2014)CrossRefGoogle Scholar
  12. 12.
    Siddiqi, M.H., Lee, S., Lee, Y.K., Khan, A.M., Truc, P.: Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13, 16682–166713 (2013)CrossRefGoogle Scholar
  13. 13.
    Rudovic, O., Pantic, M., Patras, I.: Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1357–1369 (2012)Google Scholar
  14. 14.
    Essa, I.A., Pentland, A.P.: Coding analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997)CrossRefGoogle Scholar
  15. 15.
    Sun, D., Roth, S., Black, M.J.: A qualitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2014)CrossRefGoogle Scholar
  16. 16.
    Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(2), 299–310 (2016)CrossRefGoogle Scholar
  17. 17.
    Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans. Pattern Anal. Mach. Intell. 27(8), 1226–1238 (2005)CrossRefGoogle Scholar
  18. 18.
    Jiang, B., Martinez, B., Valster, M.F., Pantic, M.: Decision level fusion of domain specific regions for facial action recognition. In: 22nd International Conference on Pattern Recognition, Stockholm, Sweden, pp. 1776–1781, 24–28 August 2014Google Scholar
  19. 19.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychological Press, Palo Alto (1978)Google Scholar
  20. 20.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression tree. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, pp. 1867–1874, 23-28 June 2014Google Scholar
  21. 21.
    King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)Google Scholar
  22. 22.
    Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools (2000)Google Scholar
  23. 23.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1Optical Flow. In: Hamprecht, Fred A., Schnörr, Christoph, Jähne, Bernd (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007). CrossRefGoogle Scholar
  24. 24.
    Javier, S.P., Enric, M.-L., Gabriele, F.: TV-L1 optical flow estimation. Image Process. Line 3, 137–150 (2013). CrossRefGoogle Scholar
  25. 25.
    Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histogram of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: Proceeding of IEEE Conference Computer Vision and Pattern Recognition (CVPR), Miami Beach, Floridia, USA, pp. 1932–1939. 20–25 June 2009Google Scholar
  26. 26.
    Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray scale and rotation invariant texture analysis with local binary patterns. IEEE Tnans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefzbMATHGoogle Scholar
  27. 27.
    Pantic, M., Valster, M., Rademaker, R., Maat, L.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specific expressions. In: Proceeding of 3rd IEEE Workshop on CVPR for Human Communication Behavior Analysis, pp. 94–101, June 2010Google Scholar
  28. 28.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. Accessed on 15 Apr 2017
  29. 29.
    Hsu, C.W., Chang, C.-C., Lin, C.-J.: A practical guide to support vector classification. Technical report, Department of Computer Science, National Taiwan University, Taiwan (2010)Google Scholar
  30. 30.
    Poursaberi, A., Noubari, H.A., Gavrolova, M., Yanushkevich, S.N.: Gauss-Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J. Image Video Process. 17, 1–13 (2012)Google Scholar
  31. 31.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2011)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Korea Electronics Technology InstituteBundang-gu, Seongnam-siRepublic of Korea

Personalised recommendations