Advertisement

Weighted-fusion feature of MB-LBPUH and HOG for facial expression recognition

  • Yan Wang
  • Ming LiEmail author
  • Congxuan Zhang
  • Hao Chen
  • Yuming Lu
Focus
  • 10 Downloads

Abstract

Obtaining a useful and discriminative feature for facial expression recognition (FER) is a hot research topic in computer vision. In this paper, we propose a novel facial expression representation for FER. Firstly, we select the appropriate parameter of multi-scale block local binary pattern uniform histogram (MB-LBPUH) operator to filter the facial images for representing the holistic structural features. Then, normalizing the filtered images into a uniform basis reduces the computational complexity and remains the full information. An MB-LBPUH feature and a HOG feature are concatenated to fuse a new feature representation for characterizing facial expressions. At the same time, weighting the MB-LBPUH feature can remove the data unbalance from a fusion feature. The weighted-fusion feature reflects not only global facial expressions structure patterns but also characterizes local expression texture appearance and shape. Finally, we utilize principal component analysis for dimensionality reduction and employ support vector machine to classification. Experimental results demonstrate that the proposed algorithm exhibits superior performance compared with the existing algorithms on JAFFE, CK+, and BU-3DFE datasets.

Keywords

Facial expression recognition Feature extraction Weighted-fusion feature Support vector machine 

Notes

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61866025, 61772255, 61866026, in part by the Key Research and Development Plan of Jiangxi Province under Grant 20161BBE50080, in part by the Advantage Subject Team Project of Jiangxi Province under Grant Nos. 20165BCB19007, 20152BCB24004, and the Science Technique Project of Jiangxi Province under Grant Nos. GJJ170608, GJJ170572.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

References

  1. Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014) Gabor wavelet transform based facial expression recognition using PCA and LBP. In: Proceedings of IEEE conference on signal processing and communications applications, pp 2265–2268Google Scholar
  2. Alsmirat MA, Al-Alem F, Al-Ayyoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimed Tools Appl 78:3649–3688CrossRefGoogle Scholar
  3. Bacivarov I, Corcoran PM (2009) Facial expression modeling using component AAM models—gaming applications. In: Proceedings of IEEE conference on consumer electronics society’s games innovations, pp 1–16Google Scholar
  4. Benitez-Garcia G, Nakamura T, Kaneko M (2017) Facial expression recognition based on local fourier coefficients and facial fourier descriptors. J Signal Inform Process 8:132–151CrossRefGoogle Scholar
  5. Chen Y-S (2019) Performance identification in large-scale class data from advanced facets of computational intelligence and soft computing techniques. Int J High Perform Comput Netw 13(3):283–293CrossRefGoogle Scholar
  6. Chen Y, Hua C, Bai R (2014a) Regression-based active appearance model initialization for facial feature tracking with missing frames. Pattern Recognit Lett 38:113–119CrossRefGoogle Scholar
  7. Chen J, Chen Z, Chi Z, Fu H (2014b) Facial expression recognition based on facial components detection and HOG features. In: Proceedings of IEEE conference on electrical and computer engineering, pp 64–69Google Scholar
  8. Cheng F, Yu J, Xiong H (2010) Facial expression recognition in JAFFE dataset based on Gaussian process classification. IEEE Trans Neural Netw 21:1685–1690CrossRefGoogle Scholar
  9. Cheon Y, Kim D (2009) Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recognit 42:1340–1350CrossRefGoogle Scholar
  10. Dahmane M, Meunier J (2014) Prototype-based modeling for facial expression analysis. IEEE Trans Multimed 16:1574–1584CrossRefGoogle Scholar
  11. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 886–893Google Scholar
  12. Edwards GJ, Taylor CJ, Cootes TF (1998) Interpreting face images using active appearance models. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 300–305Google Scholar
  13. Ekman P, Friesen WV (1978) Facial action coding system (FACS): a technique for the measurement of facial actions. Riv Psichiatr 47:126–138Google Scholar
  14. El-Latif AAA, Abd-El-Atty B, Hossain MS, Rahman MA, Alamri A, Gupta BB (2018) Efficient quantum information hiding for remote medical image sharing. IEEE Access 6:21075–21083CrossRefGoogle Scholar
  15. Fu X, Wei W (2008) Centralized binary patterns embedded with image euclidean distance for facial expression recognition. In: Proceedings of IEEE conference on natural computation, pp 115–119Google Scholar
  16. Girish GN, Shrinivasa NCL, Das PK (2014) Face recognition using MB-LBP and PCA: a comparative study. In: Proceedings of IEEE conference on computer communication and informatics, pp 1–6Google Scholar
  17. Gupta BB, Agrawal DP, Wang HX (2018) Computer and cyber security: principles, algorithm, applications, and perspectives. CRC Press, Taylor & Francis, Boca Raton.  https://doi.org/10.1201/9780429424878 CrossRefGoogle Scholar
  18. Han D, Ming Y (2014) Facial expression recognition with LBP and SLPP combined method. In: Proceedings of IEEE conference on signal processing, pp 1418–1422Google Scholar
  19. Happy SL, Dasgupta A, Patnaik P, Routray A (2013) Automated alertness and emotion detection for empathic feedback during e-learning. In: Proceedings of IEEE conference on technology for education, pp 47–50Google Scholar
  20. He H, Garcia EA (2009) Learning from Imbalanced Data. IEEE Trans Knowl Data Eng 21:1263–1284CrossRefGoogle Scholar
  21. He X, Yan S, Hu Y, Niyogi P, Zhang H-J (2005) Face recognition using Laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27:328–340CrossRefGoogle Scholar
  22. Hsu C-W, Lin C-J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13:415–425CrossRefGoogle Scholar
  23. Hsu C-W, Chang C-C, Lin C-J (2003) A practical guide to support vector classication. https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf. 2016
  24. Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C 41:765–781CrossRefGoogle Scholar
  25. Jangra A (2017) Verification and identification approach to maintain MVCC in cloud computing. Int J Comput Sci Appl 7:41–59Google Scholar
  26. Jia Z, Wang C, Yang G, Ma W (2012) A research of face feature extraction based on MB-LBP. In: Proceedings of IEEE conference on power engineering and automation, pp 69–72Google Scholar
  27. Jia J, Xu Y, Zhang S, Xue X (2016) The facial expression recognition method of random forest based on improved PCA extracting feature. In: Proceedings of IEEE conference on communications and computing, pp 1–5Google Scholar
  28. Jing WP, Jiang T, Zhang XG, Zhu LK (2019) The optimisation of speech recognition based on convolutional neural network. IJHPCN 13(2):222–231CrossRefGoogle Scholar
  29. Kumar P, Happy SL, Routray A (2017) A real-time robust facial expression recognition system using HOG features. In: Proceedings of IEEE conference on computing, analytics and security trends, pp 289–293Google Scholar
  30. Kumari J, Rajesh R, Kumar A (2016) Fusion of features for the effective facial expression recognition. In: Proceedings of IEEE conference on communication and signal processing, pp 0457–0461Google Scholar
  31. Li H, Sun J, Xu Z, Chen L (2017a) Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans Multimed 19:2816–2831CrossRefGoogle Scholar
  32. Li Z, Nie F, Chang X, Yang Y (2017b) Beyond trace ratio: weighted harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans Knowl Data Eng 29(10):2100–2110CrossRefGoogle Scholar
  33. Liao S, Zhu X, Lei Z, Zhang L, Li SZ (2007) Learning multi-scale block local binary patterns for face recognition. In: Proceedings of IEEE conference on advanced biometrics, pp 828–837Google Scholar
  34. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 94–101Google Scholar
  35. Luo Y, Wu C-M, Zhang Y (2013) Facial expression feature extraction using hybrid PCA and LBP. J China Univ Posts Telecommun 20:120–124CrossRefGoogle Scholar
  36. Luo M, Chang X, Nie L, Yang Y, Hauptmann AG, Zheng Q (2017) An adaptive semisupervised feature analysis for video semantic recognition. IEEE Trans Cybern 48:648–660CrossRefGoogle Scholar
  37. Maaten LVD, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605zbMATHGoogle Scholar
  38. Mao Q, Rao Q, Yu Y, Dong M (2017) Hierarchical Bayesian theme models for multipose facial expression recognition. IEEE Trans Multimed 19:861–873CrossRefGoogle Scholar
  39. Nakamura T, Maejima A, Morishima S (2014) Driver drowsiness estimation from facial expression features computer vision feature investigation using a CG model. In: Proceedings of IEEE conference on computer vision theory and applications, pp 207–214Google Scholar
  40. Ngo T (2011) Data mining: practical machine learning tools and technique, third edition by Ian H. Witten, Eibe Frank, Mark A. Hell. ACM, New YorkGoogle Scholar
  41. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24:971–987CrossRefGoogle Scholar
  42. Oliveira ASR, Almeida J, Ferreira J (2017) Fail silence mechanism for dependable vehicular communications. Int J High Perform Comput Netw 10:534CrossRefGoogle Scholar
  43. Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern B 36:433–449CrossRefGoogle Scholar
  44. Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern B 34:1449–1461CrossRefGoogle Scholar
  45. Ren F, Huang Z (2015) Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEEJ Trans Electr Electron Eng 10:713–722CrossRefGoogle Scholar
  46. Revina IM, Emmanuel WRS (2018) A survey on human face expression recognition techniques. J King Saud Univ Comput Inform Sci.  https://doi.org/10.1016/j.jksuci.2018.09.002 CrossRefGoogle Scholar
  47. Sadeghi H, Raie AA, Mohammadi MR (2013) Facial expression recognition using geometric normalization and appearance representation. In: Proceedings of IEEE conference on machine vision and image processing, pp 159–163Google Scholar
  48. Sariyanidi E, Gunes H, Cavallaro A (2017) Learning bases of activity for facial expression recognition. IEEE Trans Image Process 26:1965–1978MathSciNetCrossRefGoogle Scholar
  49. Sharieh A, Albdour L (2019) A heuristic approach for service allocation in cloud computing. Int J Cloud Appl Comput 7:60–74.  https://doi.org/10.4018/IJCAC.2017100104 CrossRefGoogle Scholar
  50. Shi J, Zhu H (2018) Scene image classification algorithm of fusing multi-feature. Comput Syst Appl 28(5):171–175Google Scholar
  51. Shojaeilangari S, Yun YW, Khwang TE (2012) Person independent facial expression analysis using Gabor features and genetic algorithm. In: Proceedings of IEEE conference on communications and signal processing, pp 1–5Google Scholar
  52. Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24:1386–1398MathSciNetCrossRefGoogle Scholar
  53. Sun Y, Yu J (2016) Facial expression recognition by fusing Gabor and local binary pattern features. In: Proceedings of IEEE conference on multimedia modeling, pp 209–220Google Scholar
  54. Tariq U, Huang TS (2012) Features and fusion for expression recognition—a comparative analysis. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 146–152Google Scholar
  55. Turan C, Lam KM (2014) Region-based feature fusion for facial-expression recognition. In: Proceedings of IEEE conference on image processing, pp 5966–5970Google Scholar
  56. Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3:71–86CrossRefGoogle Scholar
  57. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp I-511–I-518Google Scholar
  58. Wang X, Jin C, Liu W, Hu M, Xu L, Ren F (2013) Feature fusion of HOG and WLD for facial expression recognition. In: Proceedings of IEEE conference on system integration, pp 227–232Google Scholar
  59. Wang SH, Yang W, Dong Z, Phillips P, Zhang YD (2017) Facial emotion recognition via discrete wavelet transform, principal component analysis, and cat swarm optimization. In: Proceedings of international conference on intelligent science and big data engineering, pp 203–214Google Scholar
  60. Wu J, Song G, Huang H, Liu W, Yong X (2018) Information and communications technologies for sustainable development goals: state-of-the-art, needs and perspectives. IEEE Commun Surv Tutor 20(3):2389–2406CrossRefGoogle Scholar
  61. Xu X, Quan C, Ren F (2015) Facial expression recognition based on Gabor Wavelet transform and histogram of oriented gradients. In: Proceedings of IEEE conference on mechatronics and automation, pp 2117–2122Google Scholar
  62. Xue M, Liu W, Li L (2013) Person-independent facial expression recognition via hierarchical classification. In: Proceedings of IEEE conference on intelligent sensors, sensor networks and information processing, pp 449–454Google Scholar
  63. Yan S, Xu D, Zhang B, Zhang HJ, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29:40–51CrossRefGoogle Scholar
  64. Yang HY, Liu GD (2012) Face recognition based on MB-LBP operator and multi-linear PCA algorithm. Appl Res Comput 29:4728–4733Google Scholar
  65. Yang J, Yang J-Y (2003) Why can LDA be performed in PCA transformed space? Pattern Recognit 36:563–566CrossRefGoogle Scholar
  66. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 211–216Google Scholar
  67. Zamzami G, Ruiz G, Goldgof D, Kasturi R, Yu S, Ashmeade T (2015) Pain assessment in infants: towards spotting pain expression based on infants’ facial strain. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 1–5Google Scholar
  68. Zhang YD, Yang ZJ, Lu HM, Zhou XX, Phillips P, Liu QM, Wang SH (2017) Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation. IEEE Access 4:8375–8385CrossRefGoogle Scholar
  69. Zhao X, Zhang S (2012) Facial expression recognition using local binary patterns and discriminant kernel locally linear embedding. EURASIP J Adv Signal Process 2012:20CrossRefGoogle Scholar
  70. Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) Learning active facial patches for expression analysis. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2562–2569Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Yan Wang
    • 1
    • 2
  • Ming Li
    • 1
    • 2
    Email author
  • Congxuan Zhang
    • 2
  • Hao Chen
    • 2
  • Yuming Lu
    • 2
  1. 1.College of Automation EngineeringNanjing University of Aeronautics and AstronauticsNanjingChina
  2. 2.Key Laboratory of Jiangxi Province for Image Processing and Pattern RecognitionNanchang Hangkong UniversityNanchangChina

Personalised recommendations