Multimedia Tools and Applications

, Volume 76, Issue 1, pp 1073–1099 | Cite as

Facial expression recognition through adaptive learning of local motion descriptor

  • Swapna AgarwalEmail author
  • Dipti Prasad Mukherjee


A novel bag-of-words based approach is proposed for recognizing facial expressions corresponding to each of the six basic prototypic emotions from a video sequence. Each video sequence is represented as a specific combination of local (in spatio-temporal scale) motion patterns. These local motion patterns are captured in motion descriptors (MDs) which are unique combinations of optical flow and image gradient. These MDs can be compared to the words in the bag-of-words setting. Generally, the key-words in the wordbook as reported in the literature, are rigid, i.e., are taken as it is from the training data and cannot generalize well. We propose a novel adaptive learning technique for the key-words. The adapted key-MDs better represent the local motion patterns of the videos and generalize well to the unseen data and thus give better expression recognition accuracy. To test the efficiency of the proposed approach, we have experimented extensively on three well known datasets. We have also compared the results with existing state-of-the-art expression descriptors. Our method gives better accuracy. The proposed approach have been able to reduce the training time including the time for feature-extraction more than nine times and test time more than twice as compared to current state-of-the-art descriptor.


Facial expression recognition Motion descriptor Expression descriptor Bag-of-words Adaptive learning 



This work was supported by DST, Govt. of India project no. SR/WOS-A/ET-53/2012(G).


  1. 1.
    Abboud B, Davoine F, Dang M (2003) Expressive face recognition and synthesis. In: CVPR Workshop. doi: 10.1109/CVPRW.2003.10056, vol 5
  2. 2.
    Agarwal S, Chatterjee M, Mukherjee DP (2012) Recognizing facial expressions using a novel shape motion descriptor. In: ICVGIP. doi: 10.1145/2425333.2425362
  3. 3.
    Agarwal S, Mukherjee DP, Decoding mixed emotions from expression map of face images (2013). In: AFGR. doi: 10.1109/FG.2013.6553731
  4. 4.
    Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: WIAMIS, pp. 76–84Google Scholar
  5. 5.
    Aleksic SP, Katsaggelos KA (2006) Automatic facial expression recognition using fcial animation parameters and multi-stream hmms. IEEE Transactions on Signal Preocessing, Supplement on Secure Media 1:3–11Google Scholar
  6. 6.
    Bartlett MS, Littlewort G, Fasel I, Movellan JR (2003) Real time face detection and facial expression recognition: Development and application to human computer interaction. In: CVPR Workship on CVPR for HCI, vol. 5, p. 53Google Scholar
  7. 7.
    Bejani M, Gharavian D, Charkari NM (2014) Audiovisual emotion recognition using anova feature selection method and multi-classifier neural networks. Neural Comput & Applic 24(2):399–412CrossRefGoogle Scholar
  8. 8.
    Boughrara H, Chtourou M, Amar CB, Chen L (2014) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimedia Tools and Applications:1–23Google Scholar
  9. 9.
    Buenaposada JM, Muñoz E, Baumela L (2008) Recognising facial expressions in video sequences. Pattern Anal Applic 11(1):101–116MathSciNetCrossRefGoogle Scholar
  10. 10.
    Chakraborty D, Pal NR (2008) Selecting useful groups of features in a connectionist framework. IEEE Trans Neural Netw 19(3):381–396CrossRefGoogle Scholar
  11. 11.
    Chew SW, Rana R, Lucey P, Lucey S, Sridharan S (2012) Sparse temporal representations for facial expression recognition. In: AIVT, pp. 311–322. SpringerGoogle Scholar
  12. 12.
    Dhall A, Asthana A, Goecke R, Gedeon T (2011) Emotion recognition using phog and lpq features. In: AFGR, pp. 878–883Google Scholar
  13. 13.
    Duchenne GB (1862) Mecanisme de la Physionomie Humaine, ou analyse electro-physiologique de I’expression des passions. Jules Renouard, ParisCrossRefGoogle Scholar
  14. 14.
    Ekman P (1999) Facial expressions. In: T. Dalgleish, M. Power (eds) Handbook of Cognition and Emotion. Wiley, New YorkGoogle Scholar
  15. 15.
    Ekman P, Friesen WV (1978) The facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press Inc.Google Scholar
  16. 16.
    Girard J, Cohn J, Mahoor M, Mavadati SM, Rosenwald D (2013) Social risk and depression: Evidence from manual and automatic facial expression analysis. In: AFGR. doi: 10.1109/FG.2013.6553748
  17. 17.
    Hoque ME, Courgeon M, Martin JC, Mutlu B, Picard RW (2013) Mach: My automated conversation coach. In: UbiComp, pp. 697–706. ACMGoogle Scholar
  18. 18.
    Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2015) Effective semantic features for facial expressions recognition using svm. Multimedia Tools and Applications:1–20Google Scholar
  19. 19.
    Hsu FS, Lin WY, Tsai TW (2014) Facial expression recognition using bag of distances. Multimedia Tools and Applications 73(1):309–326CrossRefGoogle Scholar
  20. 20.
    Jain S, Hu C, Aggarwal JK (2011) Facial expression recognition with temporal modeling of shapes. In: ICCV Workshops, pp. 1642–1649. IEEEGoogle Scholar
  21. 21.
    Jiang B, Valstar MF, Pantic M (2011) Action unit detection uing sparse appearance descriptors in space-time video volumes. In: AFGR, pp. 314–321Google Scholar
  22. 22.
    Kanade T, Tian Y, Cohn JF (2000) Comprehensive database for facial expression analysis. In: AFGR, pp. 46–53. doi: 10.1109/AFGR.2000.840611
  23. 23.
    Kaur M, Vashist R, Neeru N (2010) Recognition of facial expression with principal component analysis and singular value decomposition. Int J Comput Appl 9:36–40Google Scholar
  24. 24.
    Lajevardi SM, Hussain ZM (2012) Automatic facial expression recognition: feature extraction and selection. SIViP 6(1):159–169CrossRefGoogle Scholar
  25. 25.
    Li Y, Wang S, Zhao Y, Ji Q (2013) Simultaneous facial feature tracking and facial expression recognition. IEEE Trans Image Process 22(7):2559–2573CrossRefGoogle Scholar
  26. 26.
    Littlewort G, Bartlett MS, Fasel I, Sussking J, Movellan J (2004) Dynamics of facial expression extracted automatically from video. In: CVPR Workshop, p. 80Google Scholar
  27. 27.
    Lucey P, Cohn JF, Kanade T, Saragih J (2010) The extended cohn-kanade dataset (ck + ): A complete dataset for action unit and emotion-specified expression. In: CVPR, pp. 94–101Google Scholar
  28. 28.
    Matsumoto D, Hwang HS, Skinner L, Frank M (2011) Evaluating truthfulness and detecting deception. In: FBI law enforcement bulletinGoogle Scholar
  29. 29.
    McDuff D, Kalioubyb RE, Senechal T, Demirdjian D, Picard R (2014) Automatic measurement of ad preferences from facial responses gathered over the internet. Image and Vision Computing. doi: 10.1016/j.imavis.2014.01.004
  30. 30.
    Mukherjee S, Biswas S, Mukherjee DP (2011) Recognising human action at a distance in video by key poses. IEEE Trans Circuits Syst Video Technol 21:1228–1241CrossRefGoogle Scholar
  31. 31.
    Narayan BL, Murthy CA, Pal SK (2006) Maxdiff kd- trees for data condensation. Pattern Recognition Letters 27:187–200CrossRefGoogle Scholar
  32. 32.
    Ojala M, Garriga GC (2010) Permutation tests for studying classifier performance. J Mach Learn Res 11:1833–1863MathSciNetzbMATHGoogle Scholar
  33. 33.
    Pantic M, Valstar MF, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: ICMEGoogle Scholar
  34. 34.
    Rudovic O, Pavlovic V, Pantic M (2012) Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: CVPR, pp. 2634–2641. IEEEGoogle Scholar
  35. 35.
    Sanchez A, Ruiz JV, Moreno AB, Montemayor AS, Hernndez J, Pantrigo JJ (2011) Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74:1272–1282CrossRefGoogle Scholar
  36. 36.
    Shan C, Gong S, McOwan PW (2005) Robust facial expression recognition using local binary patterns. In: ICIP, vol. 2, pp. II 370 – II 373Google Scholar
  37. 37.
    Shan C, Gong S, McOwan PW (2006) Dynamic facial expression recognition using a bayesian temporal manifold model. In: BMVC, pp. 297–306Google Scholar
  38. 38.
    Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: A comprehensive study. Image Vis Comput 27(6):803–816CrossRefGoogle Scholar
  39. 39.
    Sikka K, Dhall A, Bartlett M (2013) Weakly supervised pain localization using multiple instance learning. In: AFGR’13. doi: 10.1109/FG.2013.6553762
  40. 40.
    Suk M, Prabhakaran B (2014) Real-time mobile facial expression recognition system - a case study. In: CVPR Workshops, pp. 132–137Google Scholar
  41. 41.
    Tariq U, Yang J, Huang TS (2013) Maximum margin gmm learning for facial expression recognition. In: AFGR’13, p. NA. doi: 10.1109/FG.2013.6553794
  42. 42.
    Tian Y, Kanade T, Cohn JF (2001) Recognising action units for facial expression analysis, vol 23, p 2Google Scholar
  43. 43.
    Tian YL (2004) Evaluation of face resolution for expression analysis. In: FPIV, p. 82Google Scholar
  44. 44.
    Valstar M, Patras I, Pantic M (2005) Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data. In: CVPR, vol. 3, pp. 76–84. doi: 10.1109/CVPR.2005.457
  45. 45.
    Valstar MF, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database. In: LRECGoogle Scholar
  46. 46.
    Van Gemert JC, Veenman CJ, Smeulders AW, Geusebroek JM (2010) Visual word ambiguity. IEEE Trans Pattern Anal Mach Intell 32(7):1271–1283CrossRefGoogle Scholar
  47. 47.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: CVPR, pp. 511–518Google Scholar
  48. 48.
    Wang J, Yin L (2007) Static topographic modeling for facial expression recognition and analysis. Comput. Vis Image Underst 108(1-2):19–34CrossRefGoogle Scholar
  49. 49.
    Wang Z, Wang S, Ji Q (2013) Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In: CVPR, pp. 3422–3429. IEEEGoogle Scholar
  50. 50.
    Xiao R, Zhao Q, Zhang D, Shi P (2011) Facial expression recognition on multiple manifolds. Pattern Recognit 44(1):107–116CrossRefzbMATHGoogle Scholar
  51. 51.
    Xu L, Mordohai P (2010) Automatic facial expression recognition using bags of motion words. In: BMVC. doi: 10.5244/C.24.13
  52. 52.
    Yeasin M, Bullot B, Sharma R (2004) From facial expression to level of interest: A spatio-temporal approach. In: CVPR, vol. 2, pp. II–922 –II–927Google Scholar
  53. 53.
    Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affective Comput 2(4):219–229CrossRefGoogle Scholar
  54. 54.
    Zhang Z, Lyons MJ, Schuster M, Akamatsu S (1998) Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: AFGR, pp. 454– 459Google Scholar
  55. 55.
    Zhao G, Pietikinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915928CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Electronics and Communication Sciences UnitIndian Statistical InstituteKolkataIndia

Personalised recommendations