Skip to main content
SpringerLink
Log in

Recognizing Subtle Micro-facial Expressions Using Fuzzy Histogram of Optical Flow Orientations and Feature Selection Methods

  • Chapter
  • First Online:
Computational Intelligence for Pattern Recognition

Part of the book series: Studies in Computational Intelligence ((SCI,volume 777))

Abstract

Micro-expressions are the subtle and short-lived facial deformations that convey the inner feelings of a person. Automatic recognition of micro-expressions has potential applications in many areas. However, extraction of the appropriate feature, for encoding the subtle movements during the micro-expressions, is a very challenging work. The use of spatial and spatio-temporal features are studied extensively for this problem. However, the face appearance does not change appreciably during a micro-expression. Moreover, the muscle movements are also very small, almost indistinguishable. Rather, these changes possess a temporal pattern. We use the fuzzy histogram of optical flow orientation (FHOFO) features to encode the temporal patterns associated with facial micro-movements. The FHOFO constructs fuzzified angular histograms from the facial movement vectors. The feature descriptors of a micro-expression clip usually possess high dimension and suffer from the curse of dimensionality. To this end, we explore different feature selection methods to reduce the dimension of the descriptor. Experimentally we found that FHOFO achieves significant accuracy on the publicly available databases and its performance is consistently well across the databases.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. R.W. Picard, R. Picard, Affective Computing, vol. 252 (MIT Press, Cambridge, 1997)

    Google Scholar 

  2. P. Ekman, Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life (Times Books, New York, 2003)

    Google Scholar 

  3. P. Ekman, Lie catching and microexpressions, The Philosophy of Deception (2009), pp. 118–133

    Google Scholar 

  4. W.-J. Yan, Q. Wu, J. Liang, Y.-H. Chen, X. Fu, How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behavior 37(4), 217–230 (2013)

    Article  Google Scholar 

  5. M. Frank, M. Herbasz, K. Sinuk, A. Keller, C. Nolan, I see how you feel: training laypeople and professionals to recognize fleeting emotions, in The Annual Meeting of the International Communication Association, New York City (2009)

    Google Scholar 

  6. E. Sariyanidi, H. Gunes, A. Cavallaro, Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015)

    Article  Google Scholar 

  7. Y. Tian, T. Kanade, J.F. Cohn, Facial expression recognition, Handbook of Face Recognition (Springer, Berlin, 2011), pp. 487–519

    Chapter  Google Scholar 

  8. M. Valstar, M. Pantic, Combined support vector machines and hidden Markov models for modeling facial action temporal dynamics, Human–Computer Interaction (2007), pp. 118–127

    Google Scholar 

  9. S.-T. Liong, J. See, K. Wong, R.C.-W. Phan, Automatic micro-expression recognition from long video using a single spotted apex, in Asian Conference on Computer Vision (Springer, 2016), pp. 345–360

    Chapter  Google Scholar 

  10. M. Shreve, J. Brizzi, S. Fefilatyev, T. Luguev, D. Goldgof, S. Sarkar, Automatic expression spotting in videos. Image Vis. Comput. 32(8), 476–486 (2014)

    Article  Google Scholar 

  11. D. Patel, G. Zhao, M. Pietikäinen, Spatiotemporal integration of optical flow vectors for micro-expression detection, in International Conference on Advanced Concepts for Intelligent Vision Systems (Springer, 2015), pp. 369–380

    Chapter  Google Scholar 

  12. S.-J. Wang, S. Wu, X. Qian, J. Li, X. Fu, A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing 230, 382–389 (2017)

    Article  Google Scholar 

  13. Z. Xia, X. Feng, J. Peng, X. Peng, G. Zhao, Spontaneous micro-expression spotting via geometric deformation modeling. Comput. Vis. Image Underst. 147, 87–94 (2015)

    Article  Google Scholar 

  14. S. Polikovsky, Y. Kameda, Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Trans. Inf. Syst. 96(1), 81–92 (2013)

    Article  Google Scholar 

  15. X. Li, H. Xiaopeng, A. Moilanen, X. Huang, T. Pfister, G. Zhao, M. Pietikainen, Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 8, 29–42 (2017)

    Google Scholar 

  16. A.K. Davison, M.H. Yap, C. Lansley, Micro-facial movement detection using individualised baselines and histogram-based descriptors, in IEEE International Conference on Systems, Man, and Cybernetics (SMC) (2015), pp. 1864–1869

    Google Scholar 

  17. A.C. Le Ngo, Y.-H. Oh, R.C.-W. Phan, J. See, Eulerian emotion magnification for subtle expression recognition, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2016), pp. 1243–1247

    Google Scholar 

  18. K. Sumi, T. Ueda, Micro-expression recognition for detecting human emotional changes, in International Conference on Human-Computer Interaction (Springer, 2016), pp. 60–70

    Chapter  Google Scholar 

  19. H. Zheng, Micro-expression recognition based on 2D Gabor filter and sparse representation. J. Phys.: Conf. Ser. 787(1), 012013 (2017) (IOP Publishing)

    Google Scholar 

  20. Q. Wu, X. Shen, X. Fu, The machine knows what you are hiding: an automatic micro-expression recognition system, in Affective Computing and Intelligent Interaction (2011), pp. 152–162

    Chapter  Google Scholar 

  21. M. Owayjan, A. Kashour, N. Al Haddad, M. Fadel, G. Al Souki, The design and development of a lie detection system using facial micro-expressions, in 2012 2nd International Conference on Advances in Computational Tools for Engineering Applications (ACTEA) (IEEE, 2012), pp. 33–38

    Google Scholar 

  22. S. Yao, N. He, H. Zhang, O. Yoshie, Micro-expression recognition by feature points tracking, in 2014 10th International Conference on Communications (COMM) (IEEE, 2014), pp. 1–4

    Google Scholar 

  23. M. Shreve, S. Godavarthy, D. Goldgof, S. Sarkar, Macro-and micro-expression spotting in long videos using spatio-temporal strain, in 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011) (IEEE, 2011), pp. 51–56

    Google Scholar 

  24. S.-T. Liong, J. See, R.C.-W. Phan, A.C. Le Ngo, Y.-H. Oh, K. Wong, Subtle expression recognition using optical strain weighted features, in Asian Conference on Computer Vision (Springer, 2014), pp. 644–657

    Google Scholar 

  25. Y.-J. Liu, J.-K. Zhang, W.-J. Yan, S.-J. Wang, G. Zhao, X. Fu, A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7, 299–310 (2016). https://doi.org/10.1109/TAFFC.2015.2485205

    Article  Google Scholar 

  26. F. Xu, J. Zhang, J.Z. Wang, Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 254–267 (2017)

    Article  Google Scholar 

  27. X. Huang, G. Zhao, X. Hong, W. Zheng, M. Pietikäinen, Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175, 564–578 (2016)

    Article  Google Scholar 

  28. S.-J. Wang, W.-J. Yan, G. Zhao, X. Fu, C.-G. Zhou, Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features, in Computer Vision-ECCV Workshops (2014), pp. 325–338

    Chapter  Google Scholar 

  29. G. Zhao, M. Pietikainen, Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  30. T. Ojala, M. Pietikainen, T. Maenpaa, Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  31. J. He, J.-F. Hu, X. Lu, W.-S. Zheng, Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognit. 66, 44–52 (2017)

    Article  Google Scholar 

  32. T. Pfister, X. Li, G. Zhao, M. Pietikäinen, Recognising spontaneous facial micro-expressions, in IEEE International Conference on Computer Vision (ICCV) (2011), pp. 1449–1456

    Google Scholar 

  33. W.-J. Yan, X. Li, S.-J. Wang, G. Zhao, Y.-J. Liu, Y.-H. Chen, X. Fu, CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PloS one 9(1), e86041 (2014)

    Article  Google Scholar 

  34. B. Jiang, M. Valstar, B. Martinez, M. Pantic, A dynamic appearance descriptor approach to facial actions temporal modeling. IEEE Trans. Cybern. 44(2), 161–174 (2014)

    Article  Google Scholar 

  35. X. Fan, T. Tjahjadi, A spatial-temporal framework based on histogram of gradients and optical flow for facial expression recognition in video sequences. Pattern Recognit. 48(11), 3407–3416 (2015)

    Article  Google Scholar 

  36. X. Huang, G. Zhao, W. Zheng, M. Pietikäinen, Spatiotemporal local monogenic binary patterns for facial expression recognition. IEEE Signal Process. Lett. 19(5), 243–246 (2012)

    Article  Google Scholar 

  37. S. Polikovsky, Y. Kameda, Y. Ohta, Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Trans. Inf. Syst. 96(1), 81–92 (2013)

    Article  Google Scholar 

  38. Y. Guo, C. Xue, Y. Wang, M. Yu, Micro-expression recognition based on CBP-TOP feature with ELM. Optik-Int. J. Light Electron Opt. 126(23), 4446–4451 (2015)

    Article  Google Scholar 

  39. X. Ben, X. Jia, R. Yan, X. Zhang, W. Meng, Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recognit. Lett. (2017)

    Google Scholar 

  40. S.-J. Wang, H.-L. Chen, W.-J. Yan, Y.-H. Chen, X. Fu, Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process. Lett. 39(1), 25–43 (2014)

    Article  Google Scholar 

  41. S.-J. Wang, W.-J. Yan, X. Li, G. Zhao, C.-G. Zhou, X. Fu, M. Yang, J. Tao, Micro-expression recognition using color spaces. IEEE Trans. Image Process. 24(12), 6034–6047 (2015)

    Article  MathSciNet  Google Scholar 

  42. D. Patel, X. Hong, G. Zhao, Selective deep features for micro-expression recognition, in 2016 23rd International Conference on Pattern Recognition (ICPR) (IEEE, 2016), pp. 2258–2263

    Google Scholar 

  43. D.H. Kim, W. Baddar, J. Jang, Y.M. Ro, Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. (2017)

    Google Scholar 

  44. R. Chaudhry, A. Ravichandran, G. Hager, R. Vidal, Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions, in IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 1932–1939

    Google Scholar 

  45. X. Li, T. Pfister, X. Huang, G. Zhao, M. Pietikainen, A spontaneous micro-expression database: inducement, collection and baseline, in IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013), pp. 1–6

    Google Scholar 

  46. W.-J. Yan, Q. Wu, Y.-J. Liu, S.-J. Wang, X. Fu, CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces, in IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013), pp. 1–7

    Google Scholar 

  47. J. Tang, S. Alelyani, H. Liu, Feature selection for classification: a review, Data Classification: Algorithms and Applications (2014), p. 37

    Google Scholar 

  48. Z. Zhao, L. Wang, H. Liu, Efficient spectral feature selection with minimum redundancy, in 24th AAAI Conference on Artificial Intelligence (2010)

    Google Scholar 

  49. L. Yu, H. Liu, Efficient feature selection via analysis of relevance and redundancy. J. Mach. Learn. Res. 5, 1205–1224 (2004)

    Google Scholar 

  50. L. Yin, Y. Ge, K. Xiao, X. Wang, X. Quan, Feature selection for high-dimensional imbalanced data. Neurocomputing 105, 3–11 (2013)

    Article  Google Scholar 

  51. B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereo vision, in International Joint Conference on Artificial Intelligence, vol. 81 (1981), pp. 674–679

    Google Scholar 

  52. J. Han, K.-K. Ma, Fuzzy color histogram and its use in color image retrieval. IEEE Trans. Image Process. 11(8), 944–952 (2002)

    Article  Google Scholar 

  53. P.S. Bradley, O.L. Mangasarian, Feature selection via concave minimization and support vector machines, in International Conference on Machine Learning (ICML), vol. 98 (1998), pp. 82–90

    Google Scholar 

  54. X. He, D. Cai, P. Niyogi, Laplacian score for feature selection, in NIPS, vol. 186 (2005), p. 189

    Google Scholar 

  55. S.L. Happy, R. Mohanty, A. Routray, An effective feature selection method based on pair-wise feature proximity for high dimensional low sample size data, in European Signal Processing Conference (EUSIPCO) (2017)

    Google Scholar 

  56. S.L. Happy, A. Routray, Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)

    Article  Google Scholar 

  57. M. Shreve, S. Godavarthy, D. Goldgof, S. Sarkar, Macro-and micro-expression spotting in long videos using spatio-temporal strain, in IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (2011), pp. 51–56

    Google Scholar 

  58. S.L. Happy, A. Routray, Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. (2017)

    Google Scholar 

  59. A. Asthana, S. Zafeiriou, G. Tzimiropoulos, S. Cheng, M. Pantic, From pixels to response maps: discriminative image filtering for face alignment in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1312–1320 (2015)

    Article  Google Scholar 

  60. M. Sokolova, G. Lapalme, A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45(4), 427–437 (2009)

    Article  Google Scholar 

  61. A.C. Le Ngo, R.C.-W. Phan, J. See, Spontaneous subtle expression recognition: imbalanced databases and solutions, in Asian Conference on Computer Vision (Springer, 2014), pp. 33–48

    Google Scholar 

  62. X. Huang, S.-J. Wang, G. Zhao, M. Piteikainen, Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection, in IEEE International Conference on Computer Vision (ICCV) Workshops (2015), pp. 1–9

    Google Scholar 

  63. S.-T. Liong, J. See, R.C.-W. Phan, Y.-H. Oh, A.C. Le Ngo, K. Wong, S.-W. Tan, Spontaneous subtle expression detection and recognition based on facial strain. Signal Process.: Image Commun. 47, 170–182 (2016)

    Google Scholar 

  64. S.-J. Wang, W.-J. Yan, T. Sun, G. Zhao, X. Fu, Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214, 218–232 (2016)

    Article  Google Scholar 

  65. W.-J. Yan, S.-J. Wang, Y.-J. Liu, Q. Wu, X. Fu, For micro-expression recognition: database and suggestions. Neurocomputing 136, 82–87 (2014)

    Article  Google Scholar 

  66. Y.-H. Oh, A.C. Le Ngo, J. See, S.-T. Liong, R.C.-W. Phan, H.-C. Ling, Monogenic Riesz wavelet representation for micro-expression recognition, in IEEE International Conference on Digital Signal Processing (DSP) (IEEE, 2015), pp. 1237–1241

    Google Scholar 

  67. Y. Wang, J. See, R.C.-W. Phan, Y.-H. Oh, Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS one 10(5), e0124674 (2015)

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Ministry of Human Resource Development, Govt. of India for funding this research.

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Indian Institute of Technology, Kharagpur, India

    S. L. Happy & Aurobinda Routray

Authors
  1. S. L. Happy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aurobinda Routray
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. L. Happy .

Editor information

Editors and Affiliations

  1. University of Alberta, Edmonton, Alberta, Canada

    Witold Pedrycz

  2. National Taiwan University of Science and Technology, Taipei, Taiwan

    Shyi-Ming Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Happy, S.L., Routray, A. (2018). Recognizing Subtle Micro-facial Expressions Using Fuzzy Histogram of Optical Flow Orientations and Feature Selection Methods. In: Pedrycz, W., Chen, SM. (eds) Computational Intelligence for Pattern Recognition. Studies in Computational Intelligence, vol 777. Springer, Cham. https://doi.org/10.1007/978-3-319-89629-8_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-89629-8_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-89628-1

  • Online ISBN: 978-3-319-89629-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation