Multimedia Tools and Applications

, Volume 77, Issue 3, pp 3105–3124 | Cite as

Coupled source domain targetized with updating tag vectors for micro-expression recognition

  • Xuena Zhu
  • Xianye BenEmail author
  • Shigang Liu
  • Rui Yan
  • Weixiao Meng


Micro-expression has raised increasing attention for analyzing human inner emotions. However, most micro-expression recognition methods are developed with specific feature representations and extraction methods, such as local binary pattern on three orthogonal planes (LBP-TOP) and optical flow. The performance in such micro-expression recognition models is not high due to the limited training samples and the unequal size of the sample category. To improve the performance, we present a novel algorithm, named coupled source domain targetized with updating tag vectors, and we apply it to the micro-expression recognition. This method leverages rich speech data to enhance micro-expression recognition by transferring learning from the speech to the micro-expression recognition. The method highlights are: it simultaneously projects micro-expression samples and speech samples into a common space, then minimizes the reconstruction error between the speech and micro-expression samples, with an updating tag vectors added in the reconstruction process. It performs recognition by using dictionary learning together with support vector machine (SVM). Experimental results on the CASIA Chinese emotional corpus and CASME II micro-expression database demonstrate the effectiveness of our method.


Micro-expression recognition Coupled source domain targetized Tag vectors Transfer learning 



We sincerely thank the Institute of Psychology, Chinese Academy of Sciences for granting us permission to use the CASME database. This project is supported by the Natural Science Foundation of China (Grant No. 61571275, 61672333), the Young Scholars Program of Shandong University, and the National Key Research and Development Program of China (Grant No. 2017YFC0803400).


  1. 1.
    Aharon M, Elad M, Bruckstein A (2006) KSVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322zbMATHCrossRefGoogle Scholar
  2. 2.
    Ben X, Meng W, Yan R, Wang K (2013) Kernel coupled distance metric learning for gait recognition and face recognition. Neurocomputing 120(10):577–589CrossRefGoogle Scholar
  3. 3.
    Ben X, Zhang P, Meng W, Yan R, Yang M, Liu W et al (2016) On the distance metric learning between cross-domain gaits. Neurocomputing 208:153–164CrossRefGoogle Scholar
  4. 4.
    Ben X, Zhang P, Yan R, Yang M, Ge G (2016) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput & Applic 27(8):2629–2646CrossRefGoogle Scholar
  5. 5.
    Bryt O, Elad M (2008) Compression of facial images using the k-svd algorithm. J Visual Commun Image Represent 19(4):270–282CrossRefGoogle Scholar
  6. 6.
    CASIA Chinese emotional corpus, (2006),
  7. 7.
    Chang X, Yang Y (2016) Semisupervised feature analysis by mining correlations among multiple tasks. IEEE Trans Neural Netw Learn Syst. doi: 10.1109/TNNLS.2016.2582746 MathSciNetCrossRefGoogle Scholar
  8. 8.
    Chang X, Yu YL, Yang Y (2016) Semantic pooling for complex event analysis in untrimmed videos. IEEE Trans Pattern Anal Mach Intell. doi: 10.1109/TPAMI.2016.2608901 CrossRefGoogle Scholar
  9. 9.
    Chang X, Ma Z, Yang Y (2017) Bi-level semantic representation analysis for multimedia event detection. IEEE Trans Cybern 47(5):1180–1197CrossRefGoogle Scholar
  10. 10.
    Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, Tao D (2017) Exploring representativeness and informativeness for active learning. IEEE Trans Cybern 47(1):14–26CrossRefGoogle Scholar
  11. 11.
    Duan X, Dai Q, Wang X, Wang Y, Hua Z (2016) Recognizing spontaneous micro-expression from eye region. Neurocomputing 217:27–36CrossRefGoogle Scholar
  12. 12.
    Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Mathematics 32(2):407–451MathSciNetzbMATHGoogle Scholar
  13. 13.
    Ekman, P (2009) Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company, New YorkGoogle Scholar
  14. 14.
    Endres J, Laidlaw A (2009) Micro-expression recognition training in medical students: a pilot study. BMC Med Educ 9(1):47CrossRefGoogle Scholar
  15. 15.
    Engan K, Aase SO, Hakon Husoy J (1999) Method of optimal directions for frame design. IEEE International Conference on Acoustics, Speech, and Signal Processing 5:2443–2446Google Scholar
  16. 16.
    Gong C, Tao D, Fu K, Yang J (2014) Fick's law assisted propagation for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(9):2148–2162MathSciNetCrossRefGoogle Scholar
  17. 17.
    Gong C, Liu T, Tao D, Fu K, Tu E, Yang J (2015) Deformed graph Laplacian for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(10):2261–2274MathSciNetCrossRefGoogle Scholar
  18. 18.
    Gong C, Tao D, Maybank SJ, Liu W, Kang G, Yang J (2016) Multi-modal curriculum learning for semi-supervised image classification. IEEE Trans Image Process 25(7):3249–3260MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Gu B, Sheng VS (2017) A robust regularization path algorithm for v-support vector classification. IEEE Trans Neural Netw Learn Syst 28(5):1241–1248CrossRefGoogle Scholar
  20. 20.
    Gu B, Sun X, Sheng VS (2016) Structural minimax probability machine. IEEE Trans Neural Netw Learn Syst PP(99):1–11Google Scholar
  21. 21.
    Guo Y, Tian Y, Gao X, Zhang X (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. The 2014 International Joint Conference on Neural Networks (IJCNN). IEEE, Beijing, p 3473–3479Google Scholar
  22. 22.
    Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy. Springer US. 154–165Google Scholar
  23. 23.
    Han Y, Yang Y, Zhou X (2013) Co-regularized ensemble for feature selection. IJCAI 13:1380–1386Google Scholar
  24. 24.
    Han Y, Yang Y, Yan Y, Ma Z (2015) Semi-supervised feature selection via spline regression for video semantic recognition. IEEE Trans Neural Netw Learn Syst 26(2):252–264MathSciNetCrossRefGoogle Scholar
  25. 25.
    He J, Hu JF, Lu X, Zheng WS (2016) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn:44–52CrossRefGoogle Scholar
  26. 26.
    Jia X, Ben X, Yuan H, Kpalma K, Meng W Macro-to-micro transformation model for micro-expression recognition. J Comput Sci. doi: 10.1016/j.jocs.2017.03.016 CrossRefGoogle Scholar
  27. 27.
    Kan M, Wu J, Shan S, Chen X (2014) Domain adaptation for face recognition: targetize source domain bridged by common subspace. Int J Comput Vis 109(1):94–109zbMATHCrossRefGoogle Scholar
  28. 28.
    Li H, Liu F (2009) Image denoising via sparse and redundant representations over learned dictionaries in wavelet domain. International Conference on Image and Graphics. IEEE, Xi'an, p 754–758Google Scholar
  29. 29.
    Li Z, Yang Y, Liu J, Zhou X, Lu H (2012) Unsupervised feature selection using nonnegative spectral analysis. Twenty-Sixth AAAI Conference on Artificial Intelligence 2:1026–1032Google Scholar
  30. 30.
    Li Z, Liu J, Yang Y, Zhou X (2014) Clustering-guided sparse structural learning for unsupervised feature selection. IEEE Trans Knowl Data Eng 26(9):2138–2150CrossRefGoogle Scholar
  31. 31.
    Liao S, Jain AK, Li SZ (2013) Partial face recognition: alignment-free approach. IEEE Trans Pattern Anal Mach Intell 35(5):1193–1205CrossRefGoogle Scholar
  32. 32.
    Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310CrossRefGoogle Scholar
  33. 33.
    Mccree AV, Barnwell TPI (1995) A mixed excitation LPC vocoder model for low bit rate speech coding. IEEE Transactions on Speech & Audio Processing 3(4):242–250CrossRefGoogle Scholar
  34. 34.
    Miao Y, Gowayyed M, Metze F (2015) EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. In Automatic Speech Recognition and Understanding (ASRU), Scottsdale, 167–174Google Scholar
  35. 35.
    Muda L, Begam M, Elamvazuthi I (2010) Voice recognition algorithms using MEL frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques. arXiv preprint arXiv:1003–4083Google Scholar
  36. 36.
    Murty KSR, Yegnanarayana B (2006) Combining evidence from residual phase and MFCC features for speaker recognition. IEEE Signal Process Lett 13(1):52–55CrossRefGoogle Scholar
  37. 37.
    Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359CrossRefGoogle Scholar
  38. 38.
    Pati YC, Rezaiifar R, Krishnaprasad PS (1993) Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition, Signals, Systems and Computers, 1993. 1993 Conference Record of The Twenty-Seventh Asilomar Conference on. Conference Record of The Twenty-Seventh Asilomar Conference on. IEEE, Pacific Grove, 40–44Google Scholar
  39. 39.
    Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS (ME)^ 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput. doi: 10.1109/TAFFC.2017.2654440 CrossRefGoogle Scholar
  40. 40.
    Ren CX, Dai DQ (2010) Incremental learning of bidirectional principal components for face recognition. Pattern Recogn 43(1):318–330zbMATHCrossRefGoogle Scholar
  41. 41.
    Sobin C, Alpert M (1999) Emotion in speech: the acoustic attributes of fear, anger, sadness, and joy. J Psycholinguist Res 28(4):347–365CrossRefGoogle Scholar
  42. 42.
    Song L, Smola A, Gretton A, Borgwardt KM, Bedo J (2007) Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning, 823–830. Corvalis, Oregon, USA — June 20–24, 2007Google Scholar
  43. 43.
    Vergyri D, Stolcke A, Gadde VRR, Ferrer L (2003) Prosodic knowledge sources for automatic speech recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing 1:208–211Google Scholar
  44. 44.
    Wang SJ, Yan WJ, Li X, Zhao G, Fu X (2014) Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space. International Conference on Pattern Recognition. IEEE, Stockholm, 4678–4683Google Scholar
  45. 45.
    Wang SJ, Chen HL, Yan WJ, Chen YH, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43CrossRefGoogle Scholar
  46. 46.
    Wang Z, Yang W, Ben X (2015) Low-resolution degradation face recognition over long distance based on cca. Neural Comput & Applic 26(7):1645–1652CrossRefGoogle Scholar
  47. 47.
    Wang SJ, Yan WJ, Sun T, Zhao G, Fu X (2016) Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214:218–232CrossRefGoogle Scholar
  48. 48.
    Xia Z, Feng X, Peng J, Peng X, Zhao G (2016) Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Underst 147:87–94CrossRefGoogle Scholar
  49. 49.
    Xu F, Zhang J, Wang J (2016) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput. doi: 10.1109/TAFFC.2016.2518162 CrossRefGoogle Scholar
  50. 50.
    Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041CrossRefGoogle Scholar
  51. 51.
    Yan Y, Nie F, Li W, Gao C, Yang Y, Xu D (2016) Image classification by cross-media active learning with privileged information. IEEE Trans Multimedia 18(12):2494–2502 57CrossRefGoogle Scholar
  52. 52.
    Yang Y, Yang Y, Shen HT (2011) Effective transfer tagging from image to video. Int Conf Multimedea 9:1137–1140Google Scholar
  53. 53.
    Yang Y, Shen HT, Ma Z, Huang Z, Zhou X (2011) L2, 1-norm regularized discriminative feature selection for unsupervised learning. International Joint Conference on Artificial Intelligence 22(1):1589–1594Google Scholar
  54. 54.
    Yang Y, Ma Z, Hauptmann AG et al (2013) Feature selection for multimedia analysis by sharing information among multiple tasks. IEEE Trans Multimedia 15(3):661–669CrossRefGoogle Scholar
  55. 55.
    Yang W, Wang Z, Sun C (2015) A collaborative representation based projections method for feature extraction. Pattern Recogn 48(1):20–27CrossRefGoogle Scholar
  56. 56.
    Yang Y, Ma Z, Nie F, Chang X, Hauptmann AG (2015) Multi-class active learning by uncertainty sampling with diversity maximization. Int J Comput Vis 113(2):113–127MathSciNetCrossRefGoogle Scholar
  57. 57.
    Yeh YR, Huang CH, Wang YCF (2014) Heterogeneous domain adaptation and classification by exploiting the correlation subspace. IEEE Trans Image Process 23(5):2009–2018MathSciNetzbMATHCrossRefGoogle Scholar
  58. 58.
    Zhang P, Ben X, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik - International Journal for Light and Electron Optics 127:1395–1400CrossRefGoogle Scholar
  59. 59.
    Zhang S, Feng B, Chen Z, Huang X (2017) Micro-Expression Recognition by Aggregating Local Spatio-Temporal Patterns. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) MultiMedia Modeling. MMM 2017. Lecture Notes in Computer Science, vol 10132. Springer, ChamGoogle Scholar
  60. 60.
    Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Xuena Zhu
    • 1
  • Xianye Ben
    • 1
    Email author
  • Shigang Liu
    • 2
  • Rui Yan
    • 3
  • Weixiao Meng
    • 4
  1. 1.School of Information Science and EngineeringShandong UniversityJinanChina
  2. 2.School of Computer ScienceShaanxi Normal UniversityXi’anChina
  3. 3.Computer Science DepartmentRensselaer Polytechnic InstituteTroyUSA
  4. 4.School of Electronics and Information EngineeringHarbin Institute of TechnologyHarbinChina

Personalised recommendations