Skip to main content
SpringerLink
Log in

Recognition of facial expressions based on salient geometric features and support vector machines

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expressions convey nonverbal cues which play an important role in interpersonal relations, and are widely used in behavior interpretation of emotions, cognitive science, and social interactions. In this paper we analyze different ways of representing geometric feature and present a fully automatic facial expression recognition (FER) system using salient geometric features. In geometric feature-based FER approach, the first important step is to initialize and track dense set of facial points as the expression evolves over time in consecutive frames. In the proposed system, facial points are initialized using elastic bunch graph matching (EBGM) algorithm and tracking is performed using Kanade-Lucas-Tomaci (KLT) tracker. We extract geometric features from point, line and triangle composed of tracking results of facial points. The most discriminative line and triangle features are extracted using feature selective multi-class AdaBoost with the help of extreme learning machine (ELM) classification. Finally the geometric features for FER are extracted from the boosted line, and triangles composed of facial points. The recognition accuracy using features from point, line and triangle are analyzed independently. The performance of the proposed FER system is evaluated on three different data sets: namely CK+, MMI and MUG facial expression data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Aifanti N, Delopoulos A (2014) Linear subspace for facial expression recognition. Signal Process Image Commun 29:177–188

    Article  Google Scholar 

  2. Aifanti N, Papachristou A, Delpoulos A (2010) The MUG facial expression database. In Proceeding of 11th international workshop on image analysis for multimedia interactive services, pp 1–4

  3. Asthana A, Saragih J, Wagner M, Goecke R (2009) Evaluating AAM fitting methods for facial expression recognition. In Proceeding of the international conference on affective computing and intelligent interaction, pp 1–8

  4. Blome DS (2003) Elastic bunch graph matching. M.Sc. Thesis, Colorado State University: Fort Collins, CO, USA

  5. Bouguet JY (1999) Pyramidal implementation of the Lucas-Kanade feature tracker. Technological Report, Intel Corporation, Microprocessor Research Lab

  6. Chang Y, Hu C, Feris R, Turk M (2006) Manifold based analysis of facial expression. Image Vis Comput 24:605–614

    Article  Google Scholar 

  7. Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines. Available online: http://www.csie.ntu.edu.tw/~cjlin/libsvm. Accessed on 16 Jan 2015

  8. Choi HC, Oh SY (2006) Realtime facial expression recognition using active appearance model and multilayer perceptron. In Proceeding of the international joint conference SICE-ICASE, pp 5924–5927

  9. Cid F, Moreno J, Bustos P, Nunez P (2014) Muecas: a multi-sensor robotic head for affective human robot interaction and imitation. Sensors 14:7711–7737

    Article  Google Scholar 

  10. Cruz AC, Bhanu B, Thakoor NS (2014) Vision and attention theory based sampling for continuous facial emotion recognition. IEEE Trans Affect Comput 5:418–431

    Article  Google Scholar 

  11. Ekman P (1994) Strong evidence of universal in facial expressions: a reply to Russell’s mistaken critique. Psychol Bull 115:268–287

    Article  Google Scholar 

  12. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275

    Article  MATH  Google Scholar 

  13. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139

    Article  MathSciNet  MATH  Google Scholar 

  14. Ghimire D, Lee J (2013) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13:7714–7734

    Article  Google Scholar 

  15. Ghimire D, Lee J (2014) Extreme learning machine ensemble using bagging for facial expression recognition. J Inf Process Syst 10:443–458

    Article  Google Scholar 

  16. Hsu CW, Chang CC, Lin CJ (2010) A practical guide to support vector classification. Technical Report. Department of Computer Science, National Taiwan University, Taiwan

  17. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501

    Article  Google Scholar 

  18. Kotisa I, Buciu I, Pitas I (2008) An analysis of facial expression recognition under partial facial image occlusion. Image Vis Comput 26:1033–1046

    Google Scholar 

  19. Kotisa I, Pitas I (2007) Facial expression recognition in image sequence using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187

    Article  MathSciNet  Google Scholar 

  20. Li Y, Wang S, Zhao Y, Ji Q (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22:2559–2573

    Article  Google Scholar 

  21. Liu W, Lu J, Wang Z, Song H (2008) An expression space model for facial expression analysis. In Proceeding of IEEE congress in image and signal processing, pp 680–684

  22. Mehrabian A (1968) Communication without words. Psychol Today 2:53–56

    Google Scholar 

  23. Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115:541–558

    Article  Google Scholar 

  24. Pantic M, Rothkrantz L (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22:1424–1445

    Article  Google Scholar 

  25. Pantic M, Valster R, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Proceeding of IEEE international conference multimedia and expo, pp. 317–321

  26. Pantic M, Valster M, Rademaker R, Maat L (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specific expression. In Proceeding of 3rd IEEE workshop on CVPR for human communication behavior analysis, pp. 94–101

  27. Rahulmathavan Y, Phan RCW, Chambers JA, Parish DJ (2013) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4:83–92

    Article  Google Scholar 

  28. Rudovic O, Pantic M, Patras I (2012) Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans Pattern Anal Mach Intell 25:1357–1369

    Google Scholar 

  29. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometric features. Adv Hum Comput Interact 2014:1–13

    Article  Google Scholar 

  30. Samal A, Iyenger PA (1994) Automatic recognition of human face and facial expressions: a survey. Pattern Recogn 25:65–77

    Article  Google Scholar 

  31. Schels M, Schwenker F (2010) A multiple classifier system approach for facial expressions in image sequence utilizing GMM Supervectors. In Proceeding of the 2010 20th international conference on pattern recognition, pp 4251–4254

  32. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816

    Article  Google Scholar 

  33. Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713

    Article  Google Scholar 

  34. Soyel H, Demirel H (2012) Localized discriminative scale invariant feature transform based facial expression recognition. Comput Electr Eng 38:1299–1309

    Article  Google Scholar 

  35. Sung J, Kin D (2009) Real-time facial expression recognition using STAAM and layered GDA classifiers. Image Vis Comput 27:1313–1325

    Article  Google Scholar 

  36. Tian Y-L, Kanade T, Cohn JF (2005) Handbook of face recognition. Springer, Berlin, pp 247–275

    Book  Google Scholar 

  37. Uddin M, Lee J, Kim T (2009) An enhanced independent component-based human facial expression recognition from video. IEEE Trans Consum Electron 55:2216–2224

    Article  Google Scholar 

  38. Uhls YT, Michikyan M, Morris J, Garcia D, Small GW, Zgourou E, Greenfield PM (2014) Five days at outdoor education camp without screen improves pattern skills with nonverbal emotion cues. Comput Hum Behav 39:387–392

    Article  Google Scholar 

  39. Valster MF, Mehu M, Jiang B, Pantic M, Scherer K (2012) Meta-analysis of the first facial expression recognition challenge. IEEE Trans Syst Man Cybern B Cybern 42:966–979

    Article  Google Scholar 

  40. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57:137–154

    Article  Google Scholar 

  41. Wiskott L, Fellous JM, Krüger N (1997) Face recognition by elastic bunch graph matching. IEEE Trans Pattern Anal Mach Intell 19:775–779

    Article  Google Scholar 

  42. Wright J, Yang AY, Ganesh A, Sastry SS, Ma Y (2009) Robust face recognition via sparse representation. IEEE Trans Pattern Anal Mach Intell 31:210–227

    Article  Google Scholar 

  43. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurements of levels of interest from video. IEEE Trans Multimed 8:500–508

    Article  Google Scholar 

  44. Zafeirius S, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimed 10:1528–1540

    Article  Google Scholar 

  45. Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 31:39–58

    Article  Google Scholar 

  46. Zhang S, Zhao X, Lei B (2012) Robust facial expression recognition via compressive sensing. Sensors 12:3747–3761

    Article  Google Scholar 

  47. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29:915–928

    Article  Google Scholar 

  48. Zhao X, Zhang S (2011) Facial expression recognition based on local binary patterns and kernel discriminant isomap. Sensors 11:9573–9588

    Article  MathSciNet  Google Scholar 

  49. Zhi R, Flierl M, Ruan Q, Kleijn WB (2011) Graph-preserving sparse nonnegative matrix factorization with application to facial expression recognition. IEEE Trans Syst Man Cybern B Cybern 41:38–52

    Article  Google Scholar 

  50. Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class AdaBoost. Stat Its Interface 2:349–360

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Korea Electronics Technology Institute, Jeonju-si, Jeollabuk-do, 561-844, Republic of Korea

    Deepak Ghimire & Sunghwan Jeong

  2. Division of Computer Engineering, Chonbuk National University, Jeonju-si, Jeollabuk-do, 561-756, Republic of Korea

    Joonwhoan Lee

  3. School of Computing Science, Simon Fraser University, Burnaby, B.C., Canada

    Ze-Nian Li

Authors
  1. Deepak Ghimire
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joonwhoan Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ze-Nian Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sunghwan Jeong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joonwhoan Lee.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghimire, D., Lee, J., Li, ZN. et al. Recognition of facial expressions based on salient geometric features and support vector machines. Multimed Tools Appl 76, 7921–7946 (2017). https://doi.org/10.1007/s11042-016-3428-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3428-9

Keywords

Search

Navigation