Expression Recognition

  • S. M. Mahbubur RahmanEmail author
  • Tamanna Howlader
  • Dimitrios Hatzinakos
Part of the Cognitive Intelligence and Robotics book series (CIR)


Facial expression is regarded as one of the most powerful means for humans to convey their feelings, attitudes, or opinions to each other. It has been revealed from the psychological studies that during conversations between humans, over 50% of information is conveyed through facial expressions [39]. Automatic facial expression recognition (FER), which uses machines to recognize human facial expressions, has been an active area of research due to its several notable applications. Examples include lie detection, intelligent interaction in social media, emotional therapy for autistic patient, e-commerce, and multimodal human–computer interface [52].


  1. 1.
    GENKI-4K Database. Accessed 01 Jan 2015
  2. 2.
    N. Alugupally, A. Samal, D. Marx, S. Bhatia, Analysis of landmarks in recognition of face expressions. Pattern Recognit. Image Anal. 21(4), 681–693 (2011)CrossRefGoogle Scholar
  3. 3.
    M.S. Bartlett, G.C. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, J.R. Movellan, Automatic recognition of facial actions in spontaneous expressions. J. Multimed. 1(6), 22–35 (2006)CrossRefGoogle Scholar
  4. 4.
    J.N. Bassili, Facial motion in the perception of faces and of emotional expression. J. Exp. Psychol. - Hum. Percept. Perform. 4(3), 373–379 (1978)Google Scholar
  5. 5.
    V. Bettadapura, Facial expression recognition and analysis: the state of the art. Technical report 1203.6722, Cornell University, arXiv e-prints (2012)Google Scholar
  6. 6.
    A. Bordes, S. Ertekin, J. Weston, L. Bottou, Fast kernel classifiers with online and active learning. J. Mach. Learn. Res. 6, 1579–1619 (2005)MathSciNetzbMATHGoogle Scholar
  7. 7.
    I. Borg, P.J.F. Groenen, Modern Multidimensional Scaling, 2nd edn. (Springer, New York, 2005)zbMATHGoogle Scholar
  8. 8.
    C.C. Chang, C.J. Lin, LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3, Article 27), 1–27 (2011)Google Scholar
  9. 9.
    I. Cohen, N. Sebe, A. Garg, L.S. Chen, T.S. Huanga, Facial expression recognition from video sequences: temporal and static modeling. Comput. Vis. Image Underst. 91(1–2), 160–187 (2003)CrossRefGoogle Scholar
  10. 10.
    G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, T.J. Sejnowski, Classifying facial actions. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 974–989 (1999)Google Scholar
  11. 11.
    J.G. Dy, Unsupervised feature selection, in Computational Methods of Feature Selection, ed. by H. Liu, H. Motoda (Taylor & Francis, Florida, 2008)Google Scholar
  12. 12.
    P. Ekman, W.V. Friesen, Constants across cultures in the face and emotion. J. Personal. Soc. Psychol. 17(2), 124–129 (1971)CrossRefGoogle Scholar
  13. 13.
    D. Ghimire, J. Lee, Z.N. Li, S. Jeong, S.H. Park, H.S. Choi, Recognition of facial expressions based on tracking and selection of discriminative geometric features. Int. J. Multimed. Ubiquitous Eng. 10(3), 35–44 (2015)CrossRefGoogle Scholar
  14. 14.
    G. Guo, R. Guo, X. Li, Facial expression recognition influenced by human aging. IEEE Trans. Affect. Comput. 4(3), 291–298 (2013)CrossRefGoogle Scholar
  15. 15.
    S.L. Happy, A. Routray, Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2014)CrossRefGoogle Scholar
  16. 16.
    Y. Hu, Z. Zeng, L. Yin, X. Wei, X. Zhou, T.S. Huang, Multi-view facial expression recognition, in Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (Amsterdam, Netherlands 2008), pp. 1–6Google Scholar
  17. 17.
    S.M. Imran, S.M.M. Rahman, D. Hatzinakos, Differential components of discriminative 2D Gaussian-Hermite moments for recognition of facial expressions. Pattern Recognit. 56, 100–115 (2016)CrossRefGoogle Scholar
  18. 18.
    A. Jain, D. Zongker, Feature selection: evaluation, application, and small sample performance. IEEE Trans. Pattern Anal. Mach. Intell. 19(2), 153–158 (1997)Google Scholar
  19. 19.
    V. Jain, J. Crowley, Smile detection using multi-scale gaussian derivatives, in WSEAS International Conference on Signal Processing, Robotics and Automation (Cambridge, United Kingdom, 2013), pp. 149–154Google Scholar
  20. 20.
    Y. Ji, K. Idrissi, Automatic facial expression recognition based on spatiotemporal descriptors. Pattern Recognit. Lett. 33(10), 1373–1380 (2012)CrossRefGoogle Scholar
  21. 21.
    M.H. Kabir, T. Jabid, O. Chae, A local directional pattern variance (LDPv) based face descriptor for human facial expression recognition, in Proceedings of the IEEE International Conference on Advanced Video and Signal Based Surveillance (Boston, MA, 2010), pp. 526–532Google Scholar
  22. 22.
    S.E. Kahou, P. Froumenty, C. Pal, Facial expression analysis based on high dimensional binary features, in Lecture Notes in Computer Science: European Conference on Computer Vision (Zurich, Switzerland, 2014), pp. 135–147Google Scholar
  23. 23.
    T. Kanade, J.F. Cohn, Y. Tian, Comprehensive database for facial expression analysis, in Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (Grenoble, France, 2000), pp. 484–490Google Scholar
  24. 24.
    S.B. Kazmi, Q. Ain, M.A. Jaffar, Wavelets-based facial expression recognition using a bank of support vector machines. Soft Comput. 16(3), 369–379 (2012)CrossRefGoogle Scholar
  25. 25.
    K.J. Kelly, J. Metcalfe, Metacognition of emotional face recognition. Emotion 11(4), 896–906 (2011)CrossRefGoogle Scholar
  26. 26.
    D.H. Kim, W.J. Baddar, J. Jang, Y.M. Ro, Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 1–15 (2017).
  27. 27.
    S.R.V. Kittusamy, V. Chakrapani, Facial expressions recognition using eigenspaces. J. Comput. Sci. 8(10), 1674–1679 (2012)CrossRefGoogle Scholar
  28. 28.
    B.C. Ko, A brief review of facial emotion recognition based on visual information. Sensors 18(2)Google Scholar
  29. 29.
    S.M. Lajevardi, Z.M. Hussain, Higher order orthogonal moments for invariant facial expression recognition. Digit. Signal Process. 20(6), 1771–1779 (2010)CrossRefGoogle Scholar
  30. 30.
    S.M. Lajevardi, H.R. Wu, Facial expression recognition in perceptual color space. IEEE Trans. Image Process. 21(8), 3721–3733 (2012)MathSciNetCrossRefGoogle Scholar
  31. 31.
    S.M. Lajeverdi, Z.M. Hussain, Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6(1), 159–169 (2012)CrossRefGoogle Scholar
  32. 32.
    A. Lanitis, C.J. Taylor, T.F. Cootes, Automatic interpretation and coding of face images using flexible models. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 743–756 (1997)Google Scholar
  33. 33.
    H. Li, J.M. Morvan, L. Chen, 3D facial expression recognition based on histograms of surface differential quantities, in Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems, vol. 6915 (Ghent, Belgium, 2011), pp. 483–494Google Scholar
  34. 34.
    Y. Li, S. Wang, Y. Zhao, Q. Ji, Simultaneous facial feature tracking and facial expression recognition. IEEE Trans. Image Process. 22(7), 2559–2573 (2013)CrossRefGoogle Scholar
  35. 35.
    Z. Li, J. Imai, M. Kaneko, Face and expression recognition based on bag of words method considering holistic and local image features, in Proceedings of the International Symposium on Communications and Information Technologies (Tokyo, Japan, 2010), pp. 1–6Google Scholar
  36. 36.
    R. Londhe, V. Pawar, Facial expression recognition based on affine moment invariants. IJCSI Int. J. Comput. Sci. Issues 9(2), 388–392 (2012)Google Scholar
  37. 37.
    M.J. Lyons, J. Budynek, S. Akamatsu, Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)Google Scholar
  38. 38.
    O. Maimon, L. Rokach, Data Mining and Knowledge Discovery Handbook, 2nd edn. (Springer, New York, 2010)CrossRefGoogle Scholar
  39. 39.
    A. Mehrabian, Communication without words. Psychol. Today 2(4), 53–56 (1968)Google Scholar
  40. 40.
    S. Mitra, N.A. Lazar, Y. Liu, Understanding the role of facial asymmetry in human face identification. Stat. Comput. 17(1), 57–70 (2007)MathSciNetCrossRefGoogle Scholar
  41. 41.
    L. Oliveira, A.L. Koerich, M. Mansano, A.S. Britto, 2D principal component analysis for face and facial-expression recognition. Comput. Sci. Eng. 13(3), 9–13 (2011)CrossRefGoogle Scholar
  42. 42.
    M. Pantic, M. Valstar, R. Rademaker, L. Maat, Web-based database for facial expression analysis, in Proceedings of the IEEE International Conference on Multimedia and Expo (Amsterdam, The Netherlands 2005), pp. 1–5Google Scholar
  43. 43.
    P.J. Phillips, P.J. Flynn, T. Scruggs, K.W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, W. Worek, Overview of the face recognition grand challenge, in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (San Diego, CA, USA, 2005), pp. 947–954Google Scholar
  44. 44.
    S.M.M. Rahman, T. Howlader, D. Hatzinakos, On the selection of 2D Krawtchouk moments for face recognition. Pattern Recognit. 54, 83–93 (2016)CrossRefGoogle Scholar
  45. 45.
    A.R. Rivera, J.R. Castillo, O. Chae, Local directional number pattern for face analysis: face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2013)MathSciNetCrossRefGoogle Scholar
  46. 46.
    H. Rodger, L. Vizioli, X. Ouyang, R. Caldara, Mapping the development of facial expression recognition. Dev. Sci. 18(6), 926–939 (2015)CrossRefGoogle Scholar
  47. 47.
    E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: a Survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)Google Scholar
  48. 48.
    C. Shan, Smile detection by boosting pixel differences. IEEE Trans. Image Process. 21(1), 431–436 (2012)MathSciNetCrossRefGoogle Scholar
  49. 49.
    C. Shan, S. Gong, P.W. McOwan, Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  50. 50.
    Y. Sun, X. Wang, X. Tang, Deep convolutional network cascade for facial point detection, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Portland, OR, 2013), pp. 3476–3483Google Scholar
  51. 51.
    C.E. Thomaz, D.F. Gillies, R.Q. Feitosa, Using mixture covariance matrices to improve face and facial expression recognitions. Pattern Recognit. Lett. 24(13), 2159–2165 (2003)CrossRefGoogle Scholar
  52. 52.
    M. Turk, Multimodal human-computer interaction, in Real-Time Vision for Human-Computer Interaction (Springer, New York, 2005), pp. 269–283Google Scholar
  53. 53.
    M.Z. Uddin, W. Khaksar, J. Torresen, Facial expression recognition using salient features and convolutional neural network. IEEE Access 5, 26146–26161 (2017)CrossRefGoogle Scholar
  54. 54.
    L. Wang, R.F. Li, K. Wang, J. Chen, Feature representation for facial expression recognition based on FACS and LBP. Int. J. Autom. Comput. 11(5), 459–468 (2014)CrossRefGoogle Scholar
  55. 55.
    S. Wang, Q. Ji, Video affective content analysis: a survey of state-of-the-art methods. IEEE Trans. Affect. Comput. 6(4), 410–430 (2015)CrossRefGoogle Scholar
  56. 56.
    Z. Wang, Y. Li, S. Wang, Q. Ji, Capturing global semantic relationships for facial action unit recognition, in Proceedings of the IEEE Conference on Computer Vision (Sydney, NSW, 2013), pp. 3304–3311Google Scholar
  57. 57.
    Z. Wang, S. Wang, Q. Ji, Capturing complex spatio-temporal relations among facial muscles for facial expression recognition, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (Portland, OR, 2013), pp. 3422–3429Google Scholar
  58. 58.
    J. Whitehill, G. Littlewort, I. Fasel, M. Bartlett, J. Movellan, Toward practical smile detection. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 2106–2111 (2009)Google Scholar
  59. 59.
    W.M. Wundt, Grundzüge de Physiologischen Psychologie (Engelman, Leipzig, 1905)Google Scholar
  60. 60.
    R. Xiao, Q. Zhao, D. Zhang, P. Shi, Facial expression recognition on multiple manifolds. Pattern Recognit. 44(1), 107–116 (2011)CrossRefGoogle Scholar
  61. 61.
    J. Xibin, B. Xiyuan, D.M.W. Powers, L. Yujian, Facial expression recognition based on block Gabor wavelet fusion feature. J. Converg. Inf. Technol. 8(5), 282–289 (2013)Google Scholar
  62. 62.
    B. Yang, M. Dai, Image analysis by Gaussian-Hermite moments. Signal Process. 91(10), 2290–2303 (2011)CrossRefGoogle Scholar
  63. 63.
    B. Yang, G. Li, H. Zhang, M. Dai, Rotation and translation invariants of Gaussian-Hermite moments. Pattern Recognit. Lett. 32(9), 1283–1298 (2011)CrossRefGoogle Scholar
  64. 64.
    S. Yang, B. Bhanu, Facial expression recognition using emotion avatar image, in Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (Santa Barbara, CA, 2011), pp. 866–871Google Scholar
  65. 65.
    L. Zhang, D. Tjondronegoro, Facial expression recognition using facial movement features. IEEE Trans. Affect. Comput. 2(4), 219–229 (2011)CrossRefGoogle Scholar
  66. 66.
    W. Zhang, Y. Zhang, L. Ma, J. Guan, S. Gong, Multimodal learning for facial expression recognition. Pattern Recognit. 48(10), 3191–3202 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  • S. M. Mahbubur Rahman
    • 1
    Email author
  • Tamanna Howlader
    • 2
  • Dimitrios Hatzinakos
    • 3
  1. 1.Department of Electrical and Electronic EngineeringBangladesh University of Engineering and TechnologyDhakaBangladesh
  2. 2.Institute of Statistical Research and TrainingUniversity of DhakaDhakaBangladesh
  3. 3.Department of Electrical and Computer EngineeringUniversity of TorontoTorontoCanada

Personalised recommendations