Skip to main content
SpringerLink
Log in

A comprehensive survey on facial micro-expression: approaches and databases

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Over the past few years, the importance of the facial micro-expression (FME) has garnered increasing attention from experts because of its potential applications from the judgment court to the psychology research centers. A real challenge for developing an extensive system of the FME analysis is to select a suitable method and a database. In this manuscript, we have conducted a comprehensive and comparative survey to address the aforementioned challenge, and to give clear guidelines to alleviate further researches. To come up with this task, we have justified each method in terms of its pros and cons, which are meant to be beneficial for researchers choosing a method or a database, which suits their context application. Also, we have exhaustively analyzed the whole framework of the FME system by decomposing its pipeline into the pre-processing, the feature extraction, and the classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Abbreviations

FME:

Facial Micro-Expressions

MEs:

Micro-Expressions

MESR:

ME analysis System for Recognition

DDT:

Deception Detection Test

HS:

High Speed

VIS:

normal Visual

NIR:

Near-InfraRed

AU:

Action Unit

FACS:

Facial Action Coding System

ROI:

Region of Interest

DTCM:

Delaunay-based Temporal Coding Model

CASME:

Chinese Academy of Sciences Micro-Expressions

SAMM:

Spontaneous Actions and Micro-Movements

MEVIEW:

Micro-Expression VIdEos in the Wild

SVM:

Support-Vector Machine

RF:

Random Forest

NN:

Nearest Neighbour

MKL:

Multiple Kernel Learning

UFO-MKL:

Ultra-Fast Optimization-MKL

RK-SVD:

Relaxed K-Singular Value Decomposition

LBP:

Local Binary Pattern

LBP-TOP:

Local Binary Pattern on Three Orthogonal Planes

ELBPTOP:

Extended LBP-TOP

RDLBP-TOP:

Radial Difference LBP-TOP

LBP-SIP:

LBP with Six Intersection Points;

LBP-SIPl :

Local Binary Pattern from Six Intersection Planes

CBP-TOP:

Centralized Binary Patterns from Three Orthogonal Panels

STCLQP:

Spatio-Temporal Completed Local Quantization Patterns

AAM:

Active Appearance Models

ASM:

Active Shape Model

CLM:

Constraint Local Model

DRMF:

Discriminative Response Maps Fitting

HOG:

Histograms of Oriented Gradients

HIGO:

Histogram of Image Gradient Orientation

OF:

Optical Flow

HOOF:

Histograms of Oriented Optical Flow

RHOOF:

Region Histogram of Oriented Optical Flow

FDM:

Facial Dynamics Map

MDMO:

Main Directional Mean Optical-flow

Bi-WOOF:

Bi-Weighted Oriented Optical Flow

MDMD:

Main Directional Maximal Difference Analysis

DS-OMMA:

Dense Sampling Optical-flow’s Mean Magnitude and Angle

FHOFO:

Fuzzy Histogram of Optical Flow Orientations

CNN:

Convolutional Neural Network

DTSCNN:

Dual Temporal Scale CNN

RCNN:

Recurrent CNN

ELRCN:

Enriched Long-term Recurrent Convolutional Network

BiVACNN:

Bi-directional Vectors from Apex in CNN

TSCNN:

Three-Stream CNN

STSTNet:

Shallow Triple Stream Three-dimensional CNN

EMM:

Eulerian Motion Magnification

TIM:

Temporal Interpolation Model

MAE:

Mean Absolute Error

SE:

Standard Error

WPCA:

Whitened Principal Component Analysis

HBF:

High-frequency Band Filter

3D FFT:

3D Fast Fourier Transform

DCT:

Discrete Curvelet Transform

LWM:

Local Weighted Mean

PHOG-WEE:

Pyramid of Histograms of Orientation Gradients without Edge Extraction

References

  1. Adegun IP, Vadapalli HB, editors (2016) Automatic recognition of micro-expressions using local binary patterns on three orthogonal planes and extreme learning machine. 2016 Pattern Recognition Association of South Africa and Robotics and Mechatronics International Conference (PRASA-RobMech): IEEE

  2. Al-Sumaidaee SA, Abdullah MA, Al-Nima RRO, Dlay SS, Chambers JAJPR (2017) Multi-gradient features and elongated quinary pattern encoding for image-based facial expression recognition. Pattern Recogn 71:249–263

    Article  Google Scholar 

  3. Asthana A, Zafeiriou S, Cheng S, Pantic M, editors (2013) Robust discriminative response map fitting with constrained local models. Proceedings of the IEEE conference on computer vision and pattern recognition

  4. Ben X, Jia X, Yan R, Zhang X, Meng WJPRL (2018) Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn Lett 107:50–58

    Article  Google Scholar 

  5. Chang T, Long F, Huang J, editors (2019) Micro-expression recognition using optical flow and local binary patterns on three orthogonal planes. Proceedings of the Seventh International Symposium of Chinese CHI

  6. Chaudhry R, Ravichandran A, Hager G, Vidal R, editors (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. Computer Vision and Pattern Recognition, 2009 CVPR 2009 IEEE Conference on: IEEE

  7. Choi DY, Song BC (2020) Facial Micro-expression recognition using two-dimensional landmark feature maps. IEEE Access 8:121549–121563

    Article  Google Scholar 

  8. Chowdhary CL, Patel PV, Kathrotia KJ, Attique M, Perumal K, Ijaz MF (2020) Analytical study of hybrid techniques for image encryption and decryption. Sensors. 20(18):5162

    Article  Google Scholar 

  9. Chowdhary CL, Mittal M, Pattanaik PA, Marszalek Z (2020) An efficient segmentation and classification system in medical images using intuitionist possibilistic fuzzy C-mean clustering and fuzzy SVM algorithm. Sensors. 20(14):3903

    Article  Google Scholar 

  10. Cootes TF, Taylor CJ, Cooper DH, Graham J (1995 Jan 1) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59

    Article  Google Scholar 

  11. Das TK, Chowdhary CL, Gao XZ (2020) Chest X-ray investigation: a convolutional neural network approach. InJournal of biomimetics, biomaterials and biomedical engineering 45, 57-70. Trans Tech Publications Ltd

  12. Davison AK, Yap MH, Costen N, Tan K, Lansley C, Leightley D editors. (2014) Micro-facial movements: An investigation on spatio-temporal descriptors. European conference on computer vision; Springer

  13. Davison AK, Lansley C, Costen N, Tan K (2016) Yap MHJItoac. Samm: a spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129

    Article  Google Scholar 

  14. Davison AK, Lansley C, Costen N, Tan K (2018) Yap MHJIToAC. Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129

    Article  Google Scholar 

  15. Davison A, Merghani W, Lansley C, Ng C-C, Yap MH, editors (2018) Objective micro-facial movement detection using facs-based regions and baseline evaluation. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018): IEEE

  16. Davison AK, Merghani W, Yap M (2018) Objective classes for micro-facial expression recognition. Journal of Imaging 4(10):119

    Article  Google Scholar 

  17. Duque CA, Alata O, Emonet R, Legrand A-C, Konik H, editors (2018) Micro-expression spotting using the Riesz pyramid. 2018 IEEE Winter Conference on Applications of Computer Vision (WACV): IEEE

  18. Ekman P (1978) Facial action coding system. Consulting Psychologists Palo Alto

  19. Ekman P (2002) Microexpression training tool (METT): San Francisco: University of California

  20. Ekman P, Friesen WVJP (1969) Nonverbal leakage and clues to deception. Psychiatry. 32(1):88–106

    Article  Google Scholar 

  21. Esmaeili V, Shahdi SO (2020) Automatic micro-expression apex spotting using cubic-LBP. Multimed Tools Appl 15:1–9

    Google Scholar 

  22. Esmaeili V, Mohassel Feghhi M, Shahdi SO (2020) Autonomous apex detection and Micro-expression recognition using proposed diagonal Planes. International Journal of Nonlinear Analysis and Applications 11(Special Issue):483–497

    Google Scholar 

  23. Esmaeili V, Mohassel Feghhi M, Shahdi SO (2020) Automatic Micro-Expression Apex Frame Spotting using Local Binary Pattern from Six Intersection Planes. accepted at the 2020 International Conference on Machine Vision and Image Processing (MVIP). Faculty of Engineering, College of Farabi, University of Tahran, Iran. 19 & 20

  24. Frank MG, Ekman P,The ability to detect deceit generalizes across different types of high-stake lies. J Pers Soc Psychol. 1997;72(6):1429.

  25. Friesen E, Ekman PJPA (1978) Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3

  26. Gan Y, Liong S-T, editors (2018) Bi-directional vectors from apex in cnn for micro-expression recognition. 2018 IEEE 3rd International Conference on Image, Vision and Computing (ICIVC): IEEE

  27. Gan Y, Liong S-T, Yau W-C, Huang Y-C, Tan L-KJSPIC (2019) Off-apexnet on micro-expression recognition system. Signal Process Image Commun 74:129–139

    Article  Google Scholar 

  28. Goh KM, Ng CH, Lim LL, Sheikh UJTVC (2020) Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis Comput 36(3):445–468

    Article  Google Scholar 

  29. Goshtasby A (1988 Nov 1) Image registration by local approximation methods. Image Vis Comput 6(4):255–261

    Article  Google Scholar 

  30. Grobova J, Colovic M, Marjanovic M, Njegus A, Demire H, Anbarjafari G, editors (2017) Automatic hidden sadness detection using micro-expressions. 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017): IEEE

  31. Guo Y, Tian Y, Gao X, Zhang X, editors (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. 2014 international joint conference on neural networks (IJCNN): IEEE

  32. Guo Y, Xue C, Wang Y, Yu MJO (2015) Micro-expression recognition based on CBP-TOP feature with ELM. Optik. 126(23):4446–4451

    Article  Google Scholar 

  33. Guo C, Liang J, Zhan G, Liu Z, Pietikäinen M, Liu LJIA (2019) Extended local binary patterns for efficient and robust spontaneous facial Micro-expression recognition. IEEE Access 7:174517–174530

    Article  Google Scholar 

  34. Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy: Springer, 154–65

  35. Happy S (2017) Routray AJIToAC. Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Transactions on Affective Computing

  36. Happy S, Routray A (2018) Recognizing subtle micro-facial expressions using fuzzy histogram of optical flow orientations and feature selection methods. Computational Intelligence for Pattern Recognition: Springer, 341–68

  37. He J, Hu J-F, Lu X, Zheng W-SJPR (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52

    Article  Google Scholar 

  38. He Y, Wang S-J, Li J (2019) Yap MHJapa. Spotting Macro-and Micro-expression Intervals in Long Video Sequences. arXiv preprint arXiv:1912.11985

  39. House C, Meyer R (2015) Preprocessing and descriptor features for facial micro-expression recognition. IEEE transaction

  40. Huang W, Yin HJPR (2017) Robust face recognition with structural binary gradient patterns. Pattern Recogn 68:126–140

    Article  Google Scholar 

  41. Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE transactions on Systems, Man, and Cybernetics Part B (Cybernetics) 42(2):513–529

    Article  Google Scholar 

  42. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen MJN (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. In International Conference on the Frontiers and Advances in Data Science (FADS) 175:564–578

    Google Scholar 

  43. Huang X, Wang S-J, Liu X, Zhao G, Feng X (2017) Pietikainen MJIToAC. Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans Affect Comput 10(1):32–47

    Article  Google Scholar 

  44. Husák P, Cech J, Matas J, editors (2017) Spotting facial micro-expressions “in the wild”. 22nd Computer Vision Winter Workshop (Retz)

  45. Khor H-Q, See J, Phan RCW, Lin W, editors (2018) Enriched long-term recurrent convolutional network for facial micro-expression recognition. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018: IEEE

  46. Kim DH, Baddar WJ, Ro YM, editors (2016) Micro-expression recognition with expression-state constrained spatio-temporal feature representations. Proceedings of the 24th ACM international conference on Multimedia

  47. Krizhevsky A, Sutskever I, Hinton GE, editors (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems

  48. Le Ngo AC, See J (2016) Phan RC-WJIToAC. Sparsity in dynamics of spontaneous subtle emotions: analysis and application. IEEE Trans Affect Comput 8(3):396–411

    Article  Google Scholar 

  49. Le Ngo AC, Oh Y-H, Phan RC-W, See J, editors (2016) Eulerian emotion magnification for subtle expression recognition. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): IEEE

  50. Le Ngo AC, Johnston A, Phan RC-W, See J, editors (2018) Micro-expression motion magnification: Global Lagrangian vs. local Eulerian approaches. 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018): IEEE

  51. LeCun Y, Bottou L, Bengio Y (1998) Haffner PJPotI. Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  52. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M, editors (2013) A spontaneous micro-expression database: Inducement, collection and baseline. 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG: IEEE

  53. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, et al. (2015) Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv preprint arXiv:1511.00423 2(6):7

  54. Li X, Yu J, Zhan S, editors (2016) Spontaneous facial micro-expression detection based on deep learning. 2016 IEEE 13th International Conference on Signal Processing (ICSP): IEEE

  55. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikainen M (2018) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  56. Li Y, Huang X, Zhao G, editors (2018) Can micro-expression be recognized based on single apex frame? 2018 25th IEEE International Conference on Image Processing (ICIP): IEEE

  57. Li J, Wang Y, See J, Liu WJPA (2019) Applications. Micro-expression recognition based on 3D flow convolutional neural network. Pattern Anal Applic 22(4):1331–1339

    Article  Google Scholar 

  58. Li J, Soladie C, Seguier R, Wang SJ, Yap MH (2019) Spotting micro-expressions on long videos sequences. In2019 14th IEEE international conference on Automatic Face & Gesture Recognition (FG 2019) (pp. 1-5). IEEE

  59. Li Q, Yu J, Kurihara T, Zhang H, Zhan S (2020) Deep convolutional neural network with optical flow for facial micro-expression recognition. Journal of Circuits, Systems and Computers 29(01):2050006

  60. Liong S-T, See J, Wong K, Le Ngo AC, Oh Y-H, Phan R, editors (2015) Automatic apex frame spotting in micro-expression database. 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR): IEEE

  61. Liong S-T, See J, Wong K, Phan RC-W, editors (2016) Automatic micro-expression recognition from long video using a single spotted apex. Asian Conference on Computer Vision: Springer

  62. Liong S-T, See J, Wong K, Phan RC-WJSPIC (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  63. Liong S-T, See J, Phan RC-W, Wong K (2018) Tan S-WJJoSPS. Hybrid facial regions extraction for micro-expression recognition system. Journal of Signal Processing Systems 90(4):601–617

    Article  Google Scholar 

  64. Liong S-T, Gan Y, See J, Khor H-Q, Huang Y-C, editors (2019) Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019): IEEE

  65. Liu Y-J, Zhang J-K, Yan W-J, Wang S-J, Zhao G (2016) Fu XJIToAC. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  66. Liu Y-J, Zhang J-K, Yan W-J, Wang S-J, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  67. Liu Y-J, Li B-J, Lai Y-KJIToAC (2018) Sparse MDMO: Learning a Discriminative Feature for Spontaneous Micro-Expression Recognition. IEEE Transactions on Affective Computing

  68. Liu Y, Du H, Zheng L, Gedeon T, editors (2019) A neural micro-expression recognizer. 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019): IEEE

  69. Lu H, Kpalma K, Ronsin JJSPIC (2018) Motion descriptors for micro-expression recognition. Signal Process Image Commun 67:108–117

    Article  Google Scholar 

  70. Ma H, An G, Wu S, Yang F, editors (2017) A region histogram of oriented optical flow (RHOOF) feature for apex frame spotting in micro-expression. 2017 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS): IEEE

  71. McCabe M (2009) Available from: http://www.itl.nist.gov/iaui/894.03/face/bpr_mug3.htm

  72. Merghani W, Davison AK, Yap MHJapa (2018) A Review on facial micro-expressions analysis: Datasets, Features and Metrics. Computer Vision and Pattern Recognition

  73. Oh Y-H, Le Ngo AC, See J, Liong S-T, Phan RC-W, Ling H-C, editors (2015) Monogenic Riesz wavelet representation for micro-expression recognition. 2015 IEEE International Conference on Digital Signal Processing (DSP): IEEE

  74. Oh Y-H, Le Ngo AC, Phari RC-W, See J, Ling H-C, editors (2016) Intrinsic two-dimensional local structures for micro-expression recognition. 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP): IEEE

  75. Oh Y-H, See J, Le Ngo AC, Phan RC-W, Baskaran VMJF (2018) A Survey of automatic facial micro-expression analysis: Databases, Methods and Challenges. Front Psychol 9:1128

    Article  Google Scholar 

  76. Oh Y-H, See J, Ngo ACL, Phan RC-W, Baskaran VM (2018) A survey of automatic facial micro-expression analysis: Databases, Methods and Challenges. Frontiers in psychology

  77. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  78. Patel D, Hong X, Zhao G, editors (2016) Selective deep features for micro-expression recognition. 2016 23rd International Conference on Pattern Recognition (ICPR): IEEE

  79. Peng M, Wang C, Chen T, Liu G (2017) Fu XJFip. Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8:1745

    Article  Google Scholar 

  80. Pfister T, Li X, Zhao G, Pietikäinen M, editors (2011) Recognising spontaneous facial micro-expressions. Computer Vision (ICCV), 2011 IEEE International Conference on: IEEE.

  81. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In 3rd international conference on crime detection and prevention. IET. 1–6

  82. Qu F, Wang S-J, Yan W-J, Li H, Wu S (2018) Fu XJIToAC. CAS (ME) ^ 2: a database for spontaneous macro-expression and Micro-expression spotting and recognition. IEEE Trans Affect Comput 9(4):424–436

    Article  Google Scholar 

  83. Reddy GT, Bhattacharya S, Ramakrishnan SS, Chowdhary CL, Hakak S, Kaluri R, Reddy MP (2020) An ensemble based machine learning model for diabetic retinopathy classification. In2020 international conference on emerging trends in information technology and engineering (ic-ETITE) (pp. 1-6). IEEE

  84. Rinn WEJP (1984) The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions. Psychol Bull 95(1):52–77

    Article  Google Scholar 

  85. RM SP, Maddikunta PK, Parimala M, Koppu S, Gadekallu TR, Chowdhary CL, Alazab M (2020) An effective feature engineering for DNN using hybrid PCA-GWO for intrusion detection in IoMT architecture. Comput Commun 160:139–149

    Article  Google Scholar 

  86. Ruiz-Hernandez JA, Pietikäinen M, editors (2013) Encoding local binary patterns using the re-parametrization of the second order gaussian jet. Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on: IEEE

  87. Shahdi SO, Abu-Bakar SA (2012) Varying pose face recognition using combination of discrete cosine & wavelet transforms. In2012 4th international conference on intelligent and advanced systems (ICIAS2012) (2, 642-647). IEEE

  88. Shahdi SO, Pooyan M, Abu-Bakar SA (2010) Facial expression recognition using image orientation field in limited regions and MLP neural network. In10th international conference on information Science, signal processing and their applications (ISSPA 2010), (85-88). IEEE

  89. Shreve M, Godavarthy S, Manohar V, Goldgof D, Sarkar S, editors (2009) Towards macro-and micro-expression spotting in video using strain patterns. 2009 Workshop on Applications of Computer Vision (WACV): IEEE

  90. Shreve M, Godavarthy S, Goldgof D, Sarkar S, editors (2011) Macro-and micro-expression spotting in long videos using spatio-temporal strain. Face and Gesture 2011: IEEE

  91. Song B, Li K, Zong Y, Zhu J, Zheng W, Shi J, Zhao L (2019) Recognizing spontaneous Micro-expression using a three-stream convolutional neural network. IEEE Access 7:184537–184551

    Article  Google Scholar 

  92. Su W, Wang Y, Su F, Zhao Z, editors (2018) Micro-Expression Recognition Based on the Spatio-Temporal Feature. 2018 IEEE International Conference on Multimedia & Expo Workshops (ICMEW): IEEE

  93. Sukno FM, Butakoff C, Cruz S (2007) Frangi AFJIToPA, Intelligence M Active shape models with invariant optimal features: Application to facial analysis IEEE Transactions on Pattern Analysis and Machine Intelligence (7):1105–17

  94. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, et al. editors (2015) Going deeper with convolutions. Proceedings of the IEEE conference on computer vision and pattern recognition

  95. Takalkar MA, Xu M, editors (2017) Image based facial micro-expression recognition using deep learning on small datasets. 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA): IEEE

  96. Takalkar M, Xu M, Wu Q, Chaczko Z (2017) A survey: facial micro-expression recognition. Multimed Tools Appl 77(15):19301–19325

    Article  Google Scholar 

  97. Verma GK (2017) Facial micro-expression recognition using discrete curvelet transform. In2017 conference on information and communication technology (CICT), (pp. 1-6). IEEE

  98. Wang S-J, Yan W-J, Zhao G, Fu X, Zhou C-G, editors (2014) Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. European Conference on Computer Vision: Springer

  99. Wang Y, See J, Phan RC-W, Oh Y-H, editors (2014) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. Asian Conference on Computer Vision: Springer

  100. Wang Y, See J, Phan R, Oh YJPO (2015) Efficient Spatio-temporal local binary patterns for spontaneous facial Micro-expression recognition. PLoS One 10(5):e0124674

    Article  Google Scholar 

  101. Wang S-J, Wu S, Qian X, Li J, Fu XJN (2017) A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing. 230:382–389

    Article  Google Scholar 

  102. Wang Y, See J, Oh Y-H, Phan RC-W, Rahulamathavan Y, Ling H-C, Tan SW, Li X (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690

    Article  Google Scholar 

  103. Wang S-J, Li B-J, Liu Y-J, Yan W-J, Ou X, Huang X, Xu F, Fu X (2018) Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing. 312:251–262

    Article  Google Scholar 

  104. Wang L, Jia J, Mao N (2020) Micro-expression recognition based on 2D-3D CNN. In2020 39th Chinese control conference (CCC), (pp. 3152-3157). IEEE

  105. Warren G, Schertler E (2009) Bull PJJoNB. Detecting deception from emotional and unemotional cues. J Nonverbal Behav 33(1):59–69

    Article  Google Scholar 

  106. Weber R, Li J, Soladie C, Seguier R, editors (2018) A survey on databases of facial macro-expression and micro-expression. International Joint Conference on Computer Vision, Imaging and Computer Graphics: Springer.

  107. Wu C, Guo F (2021) TSNN: three-stream combining 2D and 3D convolutional neural network for Micro-expression recognition. IEEJ Trans Electr Electron Eng 16(1):98–107

    Article  Google Scholar 

  108. Wu Q, Shen X, Fu X, editors (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. international conference on affective computing and intelligent Interaction: Springer

  109. Wu H-Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph 31:1–8

    Article  Google Scholar 

  110. Xia Z, Feng X, Hong X, Zhao G, editors (2018) Spontaneous facial micro-expression recognition via deep convolutional network. 2018 Eighth International Conference on Image Processing Theory, Tools and Applications (IPTA): IEEE

  111. Xie L, Liu X, Wang ZJJOI, SCIENCE C (2015) Micro-expression cognition and emotion modeling based on gross reappraisal strategy. Journal of Information Computational Science 12(6):2117–2132

    Article  Google Scholar 

  112. Xu F, Zhang J (2017) Wang JZJIToAC. Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  113. Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X editors (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG); IEEE

  114. Yan W-J, Wang S-J, Liu Y-J, Wu Q, Fu XJN (2014) For micro-expression recognition: database and suggestions. Neurocomputing. 136:82–87

    Article  Google Scholar 

  115. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041

    Article  Google Scholar 

  116. Yan W-J, Wang S-J, Chen Y-H, Zhao G, Fu X, editors (2014) Quantifying micro-expressions with constraint local model and local binary pattern. European Conference on Computer Vision: Springer

  117. Yu Y, Duan H, Yu M (2018) Spatiotemporal features selection for spontaneous micro-expression recognition. J Intell Fuzzy Syst 35(4):4773–4784

    Article  MathSciNet  Google Scholar 

  118. Yu M, Guo Z, Yu Y, Wang Y, Cen SJIA (2019) Spatiotemporal feature descriptor for Micro-expression recognition using local cube binary pattern. IEEE Access 7:159214–159225

    Article  Google Scholar 

  119. Zarezadeh E (2016) Rezaeian MJBBRiAI, neuroscience. Micro expression recognition using the eulerian video magnification method. Broad Research in Artificial Intelligence and Neuroscience 7(3):43–54

    Google Scholar 

  120. Zhao G, Pietikainen M (2007 Apr 23) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

  121. Zheng H, Geng X, Yang Z, editors (2016) A relaxed K-SVD algorithm for spontaneous micro-expression recognition. Pacific Rim International Conference on Artificial Intelligence: Springer

  122. Zheng H, Zhu J, Yang Z, Jin Z (2017) Effective micro-expression recognition using relaxed K-SVD algorithm. Int J Mach Learn Cybern 8(6):2043–2049

    Article  Google Scholar 

  123. Zhu J-Y, Zheng W-S, Lai J-H, editors (2012) Complete gradient face: a novel illumination invariant descriptor. Chinese Conference on Biometric Recognition: Springer

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical and Computer Engineering, University of Tabriz, 29 Bahman Blvd, Tabriz, 5166616471, Iran

    Vida Esmaeili & Mahmood Mohassel Feghhi

  2. Department of Electrical Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran

    Seyed Omid Shahdi

Authors
  1. Vida Esmaeili
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahmood Mohassel Feghhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyed Omid Shahdi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors took part in the work described in this manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Mahmood Mohassel Feghhi.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Esmaeili, V., Mohassel Feghhi, M. & Shahdi, S.O. A comprehensive survey on facial micro-expression: approaches and databases. Multimed Tools Appl 81, 40089–40134 (2022). https://doi.org/10.1007/s11042-022-13133-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13133-2

Keywords

Search

Navigation