Skip to main content
SpringerLink
Log in

Anubhav: recognizing emotions through facial expression

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

We present a computer vision-based system named Anubhav (a Hindi word meaning feeling) which recognizes emotional facial expressions from streaming face videos. Our system runs at a speed of 10 frames per second (fps) on a 3.2-GHz desktop and at 3 fps on an Android mobile device. Using entropy and correlation-based analysis, we show that some particular salient regions of face image carry major expression-related information compared with other face regions. We also show that spatially close features within a salient face region carry correlated information regarding expression. Therefore, only a few features from each salient face region are enough for expression representation. Extraction of only a few features considerably saves response time. Exploitation of expression information from spatial as well as temporal dimensions gives good recognition accuracy. We have done extensive experiments on two publicly available data sets and also on live video streams. The recognition accuracies on benchmark CK\(+\) data set and on live video stream by our system are at least 13 and 20 % better, respectively, compared to competing approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. https://www.youtube.com/watch?v=pLq9H83Dd40.

References

  1. http://bit.ly/2cjs1Sm. Accessed on 14 Sept 2016

  2. Aldebaran: https://www.aldebaran.com/en/cool-robots/pepper. Accessed on 26 March 2016

  3. Agarwal, S., Chatterjee, M., Mukherjee, D.P.: Recognizing facial expressions using a novel shape motion descriptor. In: Indian Conference on Vision, Graphics and Image Processing. ACM (2012). doi:10.1145/2425333.2425362

  4. Agarwal, S., Mukherjee, D.P.: Decoding mixed emotions from expression map of face images. In: International Conference and Workshops on Automatic Face and Gesture Recognition. IEEE (2013). doi:10.1109/FG.2013.6553731

  5. Agarwal, S., Umer, S., Mukherjee, D.P.: Mp-feg: media player controlled by facial expressions and gestures. In: National Conference on Computer Vision, Pattern Recognition, Image Processing and Graphics. IEEE (2015)

  6. Aifanti, N., Papachristou, C., Delopoulos, A.: The mug facial expression database. In: International Workshop on Image Analysis for Multimedia Interactive Services, pp. 76–84 (2010)

  7. An, K.H., Chung, M.J.: Cognitive face analysis system for future interactive tv. IEEE Trans. Consum. Electron. 55(4), 2271–2279 (2009)

    Article  Google Scholar 

  8. Anand, B., Navathe, B., Velusamy, S., Kannan, H., Sharma, A., Gopalakrishnan, V.: Beyond touch: natural interactions using facial expressions. In: Consumer Communications and Networking Conference, pp. 255–259. IEEE (2012). doi:10.1109/CCNC.2012.6181097

  9. Bacivarov, I., Corcoran, P., Ionita, M.: Smart cameras: 2d affine models for determining subject facial expressions. IEEE Trans. Consum. Electron. 56(2), 289–297 (2010)

    Article  Google Scholar 

  10. Baltrusaitis, T., McDuff, D., Banda, N., Mahmoud, M., El Kaliouby, R., Robinson, P., Picard, R.: Real-time inference of mental states from facial expressions and upper body gestures. In: International Conference on Automatic Face Gesture Recognition and Workshops, pp. 909–914. IEEE (2011). doi:10.1109/FG.2011.5771372

  11. Bejani, M., Gharavian, D., Charkari, N.M.: Audiovisual emotion recognition using anova feature selection method and multi-classifier neural networks. Neural Comput. Appl. 24(2), 399–412 (2014)

    Article  Google Scholar 

  12. Berretti, S., Amor, B.B., Daoudi, M., Del Bimbo, A.: 3d facial expression recognition using sift descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021–1036 (2011)

    Article  Google Scholar 

  13. Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3d face scans. Vis. Comput. 29(12), 1333–1350 (2013)

    Article  Google Scholar 

  14. Boughrara, H., Chtourou, M., Amar, C.B., Chen, L.: Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed. Tools Appl. 75(2), 709–731 (2016)

    Article  Google Scholar 

  15. Bradski, G.: The opencv library. Dr. Dobb’s J. 25(11), 120–126 (2000)

  16. Chew, S.W., Rana, R., Lucey, P., Lucey, S., Sridharan, S.: Sparse temporal representations for facial expression recognition. In: Yo-sung, H. (ed.) Advances in Image and Video Technology, pp. 311–322. Springer (2012)

  17. Danelakis, A., Theoharis, T., Pratikakis, I.: A spatio-temporal wavelet-based descriptor for dynamic 3d facial expression retrieval and recognition. Vis. Comput. 1–11 (2016). doi:10.1007/s00371-016-1243-y

  18. Darwin, C.: The Expression of the Emotions in Man and Animals, anniversary edn. Harper Perennial (1872/2009). http://www.worldcat.org/isbn/0195158067

  19. Duchenne, G.B.: Mecanisme de la Physionomie Humaine, ou analyse electro-physiologique de I’expression des passions. Jules Renouard, Paris (1862)

    Google Scholar 

  20. Ekman, P., Friesen, W.V.: The Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press Inc., San Francisco (1978)

    Google Scholar 

  21. Erden, F., Cetin, A.: Hand gesture based remote control system using infrared sensors and a camera. IEEE Trans. Consum. Electron. 60(4), 675–680 (2014)

    Article  Google Scholar 

  22. Esau, N., Wetzel, E., Kleinjohann, L., Kleinjohann, B.: Real-time facial expression recognition using a fuzzy emotion model. In: International Fuzzy Systems Conference, pp. 1–6. IEEE (2007)

  23. Hou, X.N., Ding, S.H., Ma, L.Z., Wang, C.J., Li, J.L., Huang, F.Y.: Similarity metric learning for face verification using sigmoid decision function. Vis. Comput. 32(4), 479–490 (2016)

  24. Hsu, F.S., Lin, W.Y., Tsai, T.W.: Facial expression recognition using bag of distances. Multimed. Tools Appl. 73(1), 309–326 (2014)

    Article  Google Scholar 

  25. Jeong, J.W., Lee, D.H.: Inferring search intents from remote control movement patterns: a new content search method for smart tv. IEEE Trans. Consum. Electron. 60(1), 92–98 (2014)

    Article  Google Scholar 

  26. Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection uing sparse appearance descriptors in space-time video volumes. In: International Conference on Automatic Face and Gesture Recognition and Workshops, pp. 314–321. IEEE (2011). doi:10.1109/FG.2011.5771416

  27. Kanade, T., Cohn, J., Tian, Y.L.: Comprehensive database for facial expression analysis. In: International Conference on Automatic Face and Gesture Recognition, pp. 46–53. IEEE (2000). doi:10.1109/AFGR.2000.840611

  28. Khan, R.A., Meyer, A., Konik, H., Bouakaz, S.: Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit. Lett. 34(10), 1159–1168 (2013)

    Article  Google Scholar 

  29. Kotsia, I., Pitas, I.: Real time facial expression recognition from image sequences using support vector machines. In: Visual Communications and Image Processing 2005, pp. 59,602E–59,602E. International Society for Optics and Photonics (2005)

  30. Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6(1), 159–169 (2012)

    Article  Google Scholar 

  31. Li, H., Buenaposada, J.M., Baumela, L.: Real-time facial expression recognition with illumination-corrected image sequences. In: International Conference on Automatic Face & Gesture Recognition, pp. 1–6. IEEE (2008). doi:10.1109/AFGR.2008.4813328

  32. Lian, S., Hu, W., Wang, K.: Automatic user state recognition for hand gesture based low-cost television control system. IEEE Trans. Consum. Electron. 60(1), 107–115 (2014)

    Article  Google Scholar 

  33. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: International Conference on Automatic Face Gesture Recognition and Workshops, pp. 298–305 (2011). doi:10.1109/FG.2011.5771414

  34. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Computer Vision and Pattern Recognition, pp. 94–101 (2010)

  35. Maeda, A., Kobayashi, M.: Trace select: acquiring remote targets with gesture guides generated from glyphs. IEEE Trans. Consum. Electron. 60(3), 453–460 (2014)

    Article  Google Scholar 

  36. Rashid, M., Abu-Bakar, S., Mokji, M.: Human emotion recognition from videos using spatio-temporal and audio features. Vis. Comput. 29(12), 1269–1275 (2013)

    Article  Google Scholar 

  37. Sanchez, A., Ruiz, J.V., Moreno, A.B., Montemayor, A.S., Hernndez, J., Pantrigo, J.J.: Differential optical flow applied to automatic facial expression recognition. Neurocomputing 74, 1272–1282 (2011)

    Article  Google Scholar 

  38. Shan, C., Gong, S., McOwan, P.W.: Dynamic facial expression recognition using a bayesian temporal manifold model. In: British Machine Vision Conference, pp. 297–306. Citeseer (2006)

  39. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009). doi:10.1016/j.imavis.2008.08.005

    Article  Google Scholar 

  40. Suk, M., Prabhakaran, B.: Real-time mobile facial expression recognition system—a case study, pp. 132–137. IEEE (2014)

  41. de la Torre, F., Chu, W.S., Xiong, X., Vicente, F., Ding, X., Cohn, J.: Intraface. In: International Conference and Workshops on Automatic Face and Gesture Recognition, vol. 1, pp. 1–8. IEEE (2015). doi:10.1109/FG.2015.7163082

  42. Truong, A., Boujut, H., Zaharia, T.: Laban descriptors for gesture recognition and emotional analysis. Vis. Comput. 32(1), 83–98 (2016)

  43. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  44. Wang, J., Yin, L.: Static topographic modeling for facial expression recognition and analysis. Comput. Vis. Image Underst. 108(1), 19–34 (2007)

    Article  Google Scholar 

  45. Wang, Z., Miao, Z., Wu, Q.J., Wan, Y., Tang, Z.: Low-resolution face recognition: a review. Vis. Comput. 30(4), 359–386 (2014)

    Article  Google Scholar 

  46. Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Trans. Pattern Anal. Mach. Intell. 31(11), 2106–2111 (2009). doi:10.1109/TPAMI.2009.42

    Article  Google Scholar 

  47. Xiao, R., Zhao, Q., Zhang, D., Shi, P.: Facial expression recognition on multiple manifolds. Pattern Recognit. 44(1), 107–116 (2011)

    Article  MATH  Google Scholar 

  48. Xiong, X., Torre, F.: Supervised descent method and its applications to face alignment. In: Conference on Computer Vision and Pattern Recognition, pp. 532–539. IEEE (2013)

  49. Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86,041 (2014)

    Article  Google Scholar 

  50. Zhang, X., Yin, L., Hipp, D., Gerhardstein, P.: Evaluation of perceptual biases in facial expression recognition by humans and machines. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., McMahan, R.,Jerald, J., Zhang, H., Drucker, S. M., Kambhamettu, C., ElChoubassi, M., Deng, Z., Carlson, M. (eds.) Advances in Visual Computing, pp. 809–819. Springer Switzerland (2014)

  51. Zhao, G., Pietikinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

Download references

Acknowledgments

The work of Swapna Agarwal was supported by Department of Science and Technology, Government of India Project No. SR/WOS-A/ET-53/2012(G).

Author information

Authors and Affiliations

  1. Electronics and Communication Sciences Unit, Indian Statistical Institute, 203 B T Road, Kolkata, India

    Swapna Agarwal, Bikash Santra & Dipti Prasad Mukherjee

Authors
  1. Swapna Agarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bikash Santra
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dipti Prasad Mukherjee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Swapna Agarwal.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (avi 11948 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Agarwal, S., Santra, B. & Mukherjee, D.P. Anubhav: recognizing emotions through facial expression. Vis Comput 34, 177–191 (2018). https://doi.org/10.1007/s00371-016-1323-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-016-1323-z

Keywords

Search

Navigation