Skip to main content

Advertisement

SpringerLink
Log in

Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Knowledge about people’s emotions can serve as an important context for automatic service delivery in context-aware systems. Hence, human facial expression recognition (FER) has emerged as an important research area over the last two decades. To accurately recognize expressions, FER systems require automatic face detection followed by the extraction of robust features from important facial parts. Furthermore, the process should be less susceptible to the presence of noise, such as different lighting conditions and variations in facial characteristics of subjects. Accordingly, this work implements a robust FER system, capable of providing high recognition accuracy even in the presence of aforementioned variations. The system uses an unsupervised technique based on active contour model for automatic face detection and extraction. In this model, a combination of two energy functions: Chan–Vese energy and Bhattacharyya distance functions are employed to minimize the dissimilarities within a face and maximize the distance between the face and the background. Next, noise reduction is achieved by means of wavelet decomposition, followed by the extraction of facial movement features using optical flow. These features reflect facial muscle movements which signify static, dynamic, geometric, and appearance characteristics of facial expressions. Post-feature extraction, feature selection, is performed using Stepwise Linear Discriminant Analysis, which is more robust in contrast to previously employed feature selection methods for FER. Finally, expressions are recognized using trained HMM(s). To show the robustness of the proposed system, unlike most of the previous works, which were evaluated using a single dataset, performance of the proposed system is assessed in a large-scale experimentation using five publicly available different datasets. The weighted average recognition rate across these datasets indicates the success of employing the proposed system for FER.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Ahsan, T., Jabid, T., Chong, U.P., et al.: Facial expression recognition using local transitional pattern on Gabor filtered facial images. IETE Tech. Rev. 30(1), 47–52 (2013)

    Article  Google Scholar 

  2. Barron, J.L., Fleet, D.J., Beauchemin, S.S.: Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)

    Article  Google Scholar 

  3. Bartlett, M.S., Littlewort, G., Fasel, I., Movellan, J.R.: Real time face detection and facial expression recognition: development and applications to human computer interaction. In: Conference on Computer Vision and Pattern Recognition Workshop, 2003. CVPRW’03, vol. 5, pp. 53–53. IEEE (2003)

  4. Baudat, G., Anouar, F.: Generalized discriminant analysis using a kernel approach. Neural Comput. 12(10), 2385–2404 (2000)

    Article  Google Scholar 

  5. Boutsidis, C., Mahoney, M.W., Drineas, P.: Unsupervised feature selection for principal components analysis. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 61–69. ACM (2008)

  6. Buciu, I., Pitas, I.: Application of non-negative and local non negative matrix factorization to facial expression recognition. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004, vol. 1, pp. 288–291. IEEE (2004)

  7. Cendrillon, R., Lovell, B.C.: Real-time face recognition using eigenfaces. In: Proceedings-SPIE the International Society for Optical Engineering, number 1, pp. 269–276. International Society for Optical Engineering (1999, 2000)

  8. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)

    Article  MATH  Google Scholar 

  9. Chandra, D.V.S.: Image enhancement and noise reduction using wavelet transform. In: Proceedings of the 40th Midwest Symposium on Circuits and Systems, 1997, vol. 2, pp. 989–992. IEEE (1997)

  10. de Carrera, P.F.: Face recognition algorithms. PhD thesis, Universidad del País Vasco (2010)

  11. Duc, N.M., Minh, B.Q.: Your face is not your password face authentication bypassing Lenovo-Asus-Toshiba. Black Hat Briefings (2009)

  12. Feng, G.-C., Yuen, P.C., Dai, D.-Q.: Human face recognition using PCA on wavelet subband. J. Electron. Imaging 9(2), 226–233 (2000)

    Article  Google Scholar 

  13. Georghiades, A.S., Belhumeur, P.N., Kriegman, D.J.: From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 643–660 (2001)

    Article  Google Scholar 

  14. Ghimire, D., Lee, J.: Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)

    Article  Google Scholar 

  15. Gu, W., Xiang, C., Venkatesh, Y.V., Huang, D., Lin, H.: Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognit. 45(1), 80–91 (2012)

    Article  Google Scholar 

  16. He, L., Wee, W.G., Zheng, S., Wang, L.: A level set model without initial contour. In: 2009 Workshop on Applications of Computer Vision (WACV), pp. 1–6. IEEE (2009)

  17. Hjelmås, E., Low, B.K.: Face detection: a survey. Comput. Vis. Image Underst. 83(3), 236–274 (2001)

    Article  MATH  Google Scholar 

  18. Jabid, T., Kabir, Md.H., Chae, O.: Robust facial expression recognition based on local directional pattern. ETRI J. 32(5), 784–794 (2010)

  19. Jafri, R., Arabnia, H.R.: A survey of face recognition techniques. J. Inf. Process. Syst. 5(2), 41–68 (2009)

    Article  Google Scholar 

  20. Jin, Z., Lou, Z., Yang, J., Sun, Q.: Face detection using template matching and skin-color information. Neurocomputing 70(4), 794–800 (2007)

    Article  Google Scholar 

  21. Kailath, T.: The divergence and Bhattacharyya distance measures in signal selection. IEEE Trans. Commun. Technol. 15(1), 52–60 (1967)

    Article  Google Scholar 

  22. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000, pp. 46–53. IEEE (2000)

  23. Kotropoulos, C., Pitas, I.: Rule-based face detection in frontal views. In: 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, 1997. ICASSP-97, vol. 4, pp. 2537–2540. IEEE (1997)

  24. Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans. Image Process. 16(1), 172–187 (2007)

    Article  MathSciNet  Google Scholar 

  25. Krusienski, D.J., Sellers, E.W., McFarland, D.J., Vaughan, T.M., Wolpaw, J.R.: Toward enhanced p300 speller performance. J. Neurosci. Methods 167(1), 15–21 (2008)

    Article  Google Scholar 

  26. Li, S.Z., Hou, X.W., Zhang, H.J., Cheng, Q.S.: Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I207–I212. IEEE (2001)

  27. Lu, J., Plataniotis, K.N., Venetsanopoulos, A.N.: Face recognition using LDA-based algorithms. IEEE Trans. Neural Netw. 14(1), 195–200 (2003)

    Article  Google Scholar 

  28. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Proceedings of the Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 200–205. IEEE (1998)

  29. Mahmood, M.T.: Face detection by image discriminating. M.Sc. thesis (unpublished), Blekinge Institute of Technology (2006)

  30. Mika, S.: Kernel fisher discriminants. PhD thesis, Universitätsbibliothek (2002)

  31. Mika, S., Ratsch, G., Weston, J., Scholkopf, B., Mullers, K.R.: Fisher discriminant analysis with kernels. In: Neural Networks for Signal Processing IX, 1999. Proceedings of the 1999 IEEE Signal Processing Society Workshop, pp. 41–48. IEEE (1999)

  32. Pang, Y., Yuan, Y., Li, X.: Iterative subspace analysis based on feature line distance. IEEE Trans. Image Process. 18(4), 903–907 (2009)

    Article  MathSciNet  Google Scholar 

  33. Park, S.W., Savvides, M.: A multifactor extension of linear discriminant analysis for face recognition under varying pose and illumination. EURASIP J. Adv. Signal Process. 2010, 6 (2010)

    Article  Google Scholar 

  34. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)

    Article  Google Scholar 

  35. Rahulamathavan, Y., Phan, R.C.-W., Chambers, J.A., Parish, D.J.: Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 4(1), 83–92 (2013)

    Article  Google Scholar 

  36. Rivera, A.R., Castillo, J.R., Chae, O.: Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2013)

    Article  MathSciNet  Google Scholar 

  37. Samaria, F.S.: Face recognition using hidden Markov models. PhD thesis, University of Cambridge (1994)

  38. Shamir, L.: Evaluation of face datasets as tools for assessing the performance of face recognition methods. Int. J. Comput. Vis. 79(3), 225–230 (2008)

    Article  Google Scholar 

  39. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  40. Singh, S.K., Chauhan, D.S., Vatsa, M., Singh, R.: A robust skin color based face detection algorithm. Tamkang J. Sci. Eng. 6(4), 227–234 (2003)

    Google Scholar 

  41. Thomaz, C.E., Giraldi, G.A.: A new ranking method for principal components analysis and its application to face image analysis. Image Vis. Comput. 28(6), 902–913 (2010)

    Article  Google Scholar 

  42. Turunen, J.: A wavelet-based method for estimating damping in power systems. PhD thesis, Aalto University (2011)

  43. Uddin, Md.Z., Lee, J.J., Kim, T.-S.: An enhanced independent component-based human facial expression recognition from video. IEEE Trans. Consum. Electron. 55(4), 2216–2224 (2009)

  44. Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimed. 12(7), 682–691 (2010)

    Article  Google Scholar 

  45. Yang, J., Zhang, D., Frangi, A.F., Yang, J.: Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 131–137 (2004)

    Article  Google Scholar 

  46. Yang, M.-H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 24(1), 34–58 (2002)

    Article  Google Scholar 

  47. Zhang, L., Tjondronegoro, D.: Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)

    Article  Google Scholar 

  48. Zhang, S., Zhao, X., Lei, B.: Facial expression recognition based on local binary patterns and local fisher discriminant analysis. WSEAS Trans. Signal Process. 8(1), 21–31 (2012)

    Google Scholar 

  49. Zhang, X., Gao, Y.: Face recognition across pose: a review. Pattern Recognit. 42(11), 2876–2896 (2009)

    Article  Google Scholar 

  50. Zheng, W., Zhao, L., Zou, C.: A modified algorithm for generalized discriminant analysis. Neural Comput. 16(6), 1283–1297 (2004)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIP) (No. 2013-067321). This research was also supported by the MSIP (Ministry of Science, ICT and Future Planning), Korea, under the ITRC (Information Technology Research Center) support program supervised by the NIPA (National IT Industry Promotion Agency) (NIPA-2014-(H0301-14-1003)).

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Kyung Hee University, Suwon, Republic of Korea

    Muhammad Hameed Siddiqi, Rahman Ali & Sungyoung Lee

  2. Division of Information and Computer Engineering, Ajou University, Suwon, Republic of Korea

    Adil Mehmood Khan

  3. Department of Electronic Engineering, Kwangwoon University, Seoul, Republic of Korea

    Eun Soo Kim

  4. Department of Software Technology and Enterprize, Korea University, Seoul, Republic of Korea

    Gerard Junghyun Kim

Authors
  1. Muhammad Hameed Siddiqi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rahman Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adil Mehmood Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eun Soo Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gerard Junghyun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sungyoung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sungyoung Lee.

Additional information

Communicated by P. Pala.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Siddiqi, M.H., Ali, R., Khan, A.M. et al. Facial expression recognition using active contour-based face detection, facial movement-based feature extraction, and non-linear feature selection. Multimedia Systems 21, 541–555 (2015). https://doi.org/10.1007/s00530-014-0400-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-014-0400-2

Keywords

Search

Navigation