Multimedia Systems

, Volume 21, Issue 6, pp 541–555 | Cite as

Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection

  • Muhammad Hameed Siddiqi
  • Rahman Ali
  • Adil Mehmood Khan
  • Eun Soo Kim
  • Gerard Junghyun Kim
  • Sungyoung LeeEmail author
Regular Paper


Knowledge about people’s emotions can serve as an important context for automatic service delivery in context-aware systems. Hence, human facial expression recognition (FER) has emerged as an important research area over the last two decades. To accurately recognize expressions, FER systems require automatic face detection followed by the extraction of robust features from important facial parts. Furthermore, the process should be less susceptible to the presence of noise, such as different lighting conditions and variations in facial characteristics of subjects. Accordingly, this work implements a robust FER system, capable of providing high recognition accuracy even in the presence of aforementioned variations. The system uses an unsupervised technique based on active contour model for automatic face detection and extraction. In this model, a combination of two energy functions: Chan–Vese energy and Bhattacharyya distance functions are employed to minimize the dissimilarities within a face and maximize the distance between the face and the background. Next, noise reduction is achieved by means of wavelet decomposition, followed by the extraction of facial movement features using optical flow. These features reflect facial muscle movements which signify static, dynamic, geometric, and appearance characteristics of facial expressions. Post-feature extraction, feature selection, is performed using Stepwise Linear Discriminant Analysis, which is more robust in contrast to previously employed feature selection methods for FER. Finally, expressions are recognized using trained HMM(s). To show the robustness of the proposed system, unlike most of the previous works, which were evaluated using a single dataset, performance of the proposed system is assessed in a large-scale experimentation using five publicly available different datasets. The weighted average recognition rate across these datasets indicates the success of employing the proposed system for FER.


Facial expressions Face detection Active contour  Level set Wavelet transform Optical flow Stepwise linear discriminant analysis Hidden Markov model 



This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2013-067321). This research was also supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2014-(H0301-14-1003)).


  1. 1.
    Ahsan, T., Jabid, T., Chong, U.P., et al.: Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Tech. Rev. 30(1), 47–52 (2013)CrossRefGoogle Scholar
  2. 2.
    Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)CrossRefGoogle Scholar
  3. 3.
    Bartlett, M.S., Littlewort, G., Fasel, I., Movellan, J.R.: Real time face detection and facial expression recognition: development and applications to human computer interaction. In: Conference on Computer Vision and Pattern Recognition Workshop, 2003. CVPRW’03, vol. 5, pp. 53–53. IEEE (2003)Google Scholar
  4. 4.
    Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Comput. 12(10), 2385–2404 (2000)CrossRefGoogle Scholar
  5. 5.
    Boutsidis, C., Mahoney, M.W., Drineas, P.: Unsupervised feature selection for principal components analysis. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 61–69. ACM (2008)Google Scholar
  6. 6.
    Buciu, I., Pitas, I.: Application of non-negative and local non negative matrix factorization to facial expression recognition. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol. 1, pp. 288–291. IEEE (2004)Google Scholar
  7. 7.
    Cendrillon, R., Lovell, B.C.: Real-time face recognition using eigenfaces. In: Proceedings-SPIE the International Society for Optical Engineering, number 1, pp. 269–276. International Society for Optical Engineering (1999, 2000)Google Scholar
  8. 8.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)zbMATHCrossRefGoogle Scholar
  9. 9.
    Chandra, D.V.S.: Image enhancement and noise reduction using wavelet transform. In: Proceedings of the 40th Midwest Symposium on Circuits and Systems, 1997, vol. 2, pp. 989–992. IEEE (1997)Google Scholar
  10. 10.
    de Carrera, P.F.: Face recognition algorithms. PhD thesis, Universidad del País Vasco (2010)Google Scholar
  11. 11.
    Duc, N.M., Minh, B.Q.: Your face is not your password face authentication bypassing Lenovo-Asus-Toshiba. Black Hat Briefings (2009)Google Scholar
  12. 12.
    Feng, G.-C., Yuen, P.C., Dai, D.-Q.: Human face recognition using PCA on wavelet subband. J. Electron. Imaging 9(2), 226–233 (2000)CrossRefGoogle Scholar
  13. 13.
    Georghiades, A.S., Belhumeur, P.N., Kriegman, D.J.: From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 643–660 (2001)CrossRefGoogle Scholar
  14. 14.
    Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)CrossRefGoogle Scholar
  15. 15.
    Gu, W., Xiang, C., Venkatesh, Y.V., Huang, D., Lin, H.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)CrossRefGoogle Scholar
  16. 16.
    He, L., Wee, W.G., Zheng, S., Wang, L.: A level set model without initial contour. In: 2009 Workshop on Applications of Computer Vision (WACV), pp. 1–6. IEEE (2009)Google Scholar
  17. 17.
    Hjelmås, E., Low, B.K.: Face detection: a survey. Comput. Vis. Image Underst. 83(3), 236–274 (2001)zbMATHCrossRefGoogle Scholar
  18. 18.
    Jabid, T., Kabir, Md.H., Chae, O.: Robust facial expression recognition based on local directional pattern. ETRI J. 32(5), 784–794 (2010)Google Scholar
  19. 19.
    Jafri, R., Arabnia, H.R.: A survey of face recognition techniques. J. Inf. Process. Syst. 5(2), 41–68 (2009)CrossRefGoogle Scholar
  20. 20.
    Jin, Z., Lou, Z., Yang, J., Sun, Q.: Face detection using template matching and skin-color information. Neurocomputing 70(4), 794–800 (2007)CrossRefGoogle Scholar
  21. 21.
    Kailath, T.: The divergence and Bhattacharyya distance measures in signal selection. IEEE Trans. Commun. Technol. 15(1), 52–60 (1967)CrossRefGoogle Scholar
  22. 22.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000, pp. 46–53. IEEE (2000)Google Scholar
  23. 23.
    Kotropoulos, C., Pitas, I.: Rule-based face detection in frontal views. In: 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1997. ICASSP-97, vol. 4, pp. 2537–2540. IEEE (1997)Google Scholar
  24. 24.
    Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans. Image Process. 16(1), 172–187 (2007)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Krusienski, D.J., Sellers, E.W., McFarland, D.J., Vaughan, T.M., Wolpaw, J.R.: Toward enhanced p300 speller performance. J. Neurosci. Methods 167(1), 15–21 (2008)CrossRefGoogle Scholar
  26. 26.
    Li, S.Z., Hou, X.W., Zhang, H.J., Cheng, Q.S.: Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I207–I212. IEEE (2001)Google Scholar
  27. 27.
    Lu, J., Plataniotis, K.N., Venetsanopoulos, A.N.: Face recognition using LDA-based algorithms. IEEE Trans. Neural Netw. 14(1), 195–200 (2003)CrossRefGoogle Scholar
  28. 28.
    Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 200–205. IEEE (1998)Google Scholar
  29. 29.
    Mahmood, M.T.: Face detection by image discriminating. M.Sc. thesis (unpublished), Blekinge Institute of Technology (2006)Google Scholar
  30. 30.
    Mika, S.: Kernel fisher discriminants. PhD thesis, Universitätsbibliothek (2002)Google Scholar
  31. 31.
    Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.R.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop, pp. 41–48. IEEE (1999)Google Scholar
  32. 32.
    Pang, Y., Yuan, Y., Li, X.: Iterative subspace analysis based on feature line distance. IEEE Trans. Image Process. 18(4), 903–907 (2009)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Park, S.W., Savvides, M.: A multifactor extension of linear discriminant analysis for face recognition under varying pose and illumination. EURASIP J. Adv. Signal Process. 2010, 6 (2010)CrossRefGoogle Scholar
  34. 34.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  35. 35.
    Rahulamathavan, Y., Phan, R.C.-W., Chambers, J.A., Parish, D.J.: Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 4(1), 83–92 (2013)CrossRefGoogle Scholar
  36. 36.
    Rivera, A.R., Castillo, J.R., Chae, O.: Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2013)MathSciNetCrossRefGoogle Scholar
  37. 37.
    Samaria, F.S.: Face recognition using hidden Markov models. PhD thesis, University of Cambridge (1994)Google Scholar
  38. 38.
    Shamir, L.: Evaluation of face datasets as tools for assessing the performance of face recognition methods. Int. J. Comput. Vis. 79(3), 225–230 (2008)CrossRefGoogle Scholar
  39. 39.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  40. 40.
    Singh, S.K., Chauhan, D.S., Vatsa, M., Singh, R.: A robust skin color based face detection algorithm. Tamkang J. Sci. Eng. 6(4), 227–234 (2003)Google Scholar
  41. 41.
    Thomaz, C.E., Giraldi, G.A.: A new ranking method for principal components analysis and its application to face image analysis. Image Vis. Comput. 28(6), 902–913 (2010)CrossRefGoogle Scholar
  42. 42.
    Turunen, J.: A wavelet-based method for estimating damping in power systems. PhD thesis, Aalto University (2011)Google Scholar
  43. 43.
    Uddin, Md.Z., Lee, J.J., Kim, T.-S.: An enhanced independent component-based human facial expression recognition from video. IEEE Trans. Consum. Electron. 55(4), 2216–2224 (2009)Google Scholar
  44. 44.
    Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimed. 12(7), 682–691 (2010)CrossRefGoogle Scholar
  45. 45.
    Yang, J., Zhang, D., Frangi, A.F., Yang, J.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004)CrossRefGoogle Scholar
  46. 46.
    Yang, M.-H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)CrossRefGoogle Scholar
  47. 47.
    Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)CrossRefGoogle Scholar
  48. 48.
    Zhang, S., Zhao, X., Lei, B.: Facial expression recognition based on local binary patterns and local fisher discriminant analysis. WSEAS Trans. Signal Process. 8(1), 21–31 (2012)Google Scholar
  49. 49.
    Zhang, X., Gao, Y.: Face recognition across pose: a review. Pattern Recognit. 42(11), 2876–2896 (2009)CrossRefGoogle Scholar
  50. 50.
    Zheng, W., Zhao, L., Zou, C.: A modified algorithm for generalized discriminant analysis. Neural Comput. 16(6), 1283–1297 (2004)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Muhammad Hameed Siddiqi
    • 1
  • Rahman Ali
    • 1
  • Adil Mehmood Khan
    • 2
  • Eun Soo Kim
    • 3
  • Gerard Junghyun Kim
    • 4
  • Sungyoung Lee
    • 1
    Email author
  1. 1.Department of Computer EngineeringKyung Hee UniversitySuwonRepublic of Korea
  2. 2.Division of Information and Computer EngineeringAjou UniversitySuwonRepublic of Korea
  3. 3.Department of Electronic EngineeringKwangwoon UniversitySeoulRepublic of Korea
  4. 4.Department of Software Technology and EnterprizeKorea UniversitySeoulRepublic of Korea

Personalised recommendations