Skip to main content
SpringerLink
Log in

Weighted-fusion feature of MB-LBPUH and HOG for facial expression recognition

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Obtaining a useful and discriminative feature for facial expression recognition (FER) is a hot research topic in computer vision. In this paper, we propose a novel facial expression representation for FER. Firstly, we select the appropriate parameter of multi-scale block local binary pattern uniform histogram (MB-LBPUH) operator to filter the facial images for representing the holistic structural features. Then, normalizing the filtered images into a uniform basis reduces the computational complexity and remains the full information. An MB-LBPUH feature and a HOG feature are concatenated to fuse a new feature representation for characterizing facial expressions. At the same time, weighting the MB-LBPUH feature can remove the data unbalance from a fusion feature. The weighted-fusion feature reflects not only global facial expressions structure patterns but also characterizes local expression texture appearance and shape. Finally, we utilize principal component analysis for dimensionality reduction and employ support vector machine to classification. Experimental results demonstrate that the proposed algorithm exhibits superior performance compared with the existing algorithms on JAFFE, CK+, and BU-3DFE datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  • Abdulrahman M, Gwadabe TR, Abdu FJ, Eleyan A (2014) Gabor wavelet transform based facial expression recognition using PCA and LBP. In: Proceedings of IEEE conference on signal processing and communications applications, pp 2265–2268

  • Alsmirat MA, Al-Alem F, Al-Ayyoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimed Tools Appl 78:3649–3688

    Article  Google Scholar 

  • Bacivarov I, Corcoran PM (2009) Facial expression modeling using component AAM models—gaming applications. In: Proceedings of IEEE conference on consumer electronics society’s games innovations, pp 1–16

  • Benitez-Garcia G, Nakamura T, Kaneko M (2017) Facial expression recognition based on local fourier coefficients and facial fourier descriptors. J Signal Inform Process 8:132–151

    Article  Google Scholar 

  • Chen Y-S (2019) Performance identification in large-scale class data from advanced facets of computational intelligence and soft computing techniques. Int J High Perform Comput Netw 13(3):283–293

    Article  Google Scholar 

  • Chen Y, Hua C, Bai R (2014a) Regression-based active appearance model initialization for facial feature tracking with missing frames. Pattern Recognit Lett 38:113–119

    Article  Google Scholar 

  • Chen J, Chen Z, Chi Z, Fu H (2014b) Facial expression recognition based on facial components detection and HOG features. In: Proceedings of IEEE conference on electrical and computer engineering, pp 64–69

  • Cheng F, Yu J, Xiong H (2010) Facial expression recognition in JAFFE dataset based on Gaussian process classification. IEEE Trans Neural Netw 21:1685–1690

    Article  Google Scholar 

  • Cheon Y, Kim D (2009) Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recognit 42:1340–1350

    Article  Google Scholar 

  • Dahmane M, Meunier J (2014) Prototype-based modeling for facial expression analysis. IEEE Trans Multimed 16:1574–1584

    Article  Google Scholar 

  • Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 886–893

  • Edwards GJ, Taylor CJ, Cootes TF (1998) Interpreting face images using active appearance models. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 300–305

  • Ekman P, Friesen WV (1978) Facial action coding system (FACS): a technique for the measurement of facial actions. Riv Psichiatr 47:126–138

    Google Scholar 

  • El-Latif AAA, Abd-El-Atty B, Hossain MS, Rahman MA, Alamri A, Gupta BB (2018) Efficient quantum information hiding for remote medical image sharing. IEEE Access 6:21075–21083

    Article  Google Scholar 

  • Fu X, Wei W (2008) Centralized binary patterns embedded with image euclidean distance for facial expression recognition. In: Proceedings of IEEE conference on natural computation, pp 115–119

  • Girish GN, Shrinivasa NCL, Das PK (2014) Face recognition using MB-LBP and PCA: a comparative study. In: Proceedings of IEEE conference on computer communication and informatics, pp 1–6

  • Gupta BB, Agrawal DP, Wang HX (2018) Computer and cyber security: principles, algorithm, applications, and perspectives. CRC Press, Taylor & Francis, Boca Raton. https://doi.org/10.1201/9780429424878

    Book  Google Scholar 

  • Han D, Ming Y (2014) Facial expression recognition with LBP and SLPP combined method. In: Proceedings of IEEE conference on signal processing, pp 1418–1422

  • Happy SL, Dasgupta A, Patnaik P, Routray A (2013) Automated alertness and emotion detection for empathic feedback during e-learning. In: Proceedings of IEEE conference on technology for education, pp 47–50

  • He H, Garcia EA (2009) Learning from Imbalanced Data. IEEE Trans Knowl Data Eng 21:1263–1284

    Article  Google Scholar 

  • He X, Yan S, Hu Y, Niyogi P, Zhang H-J (2005) Face recognition using Laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27:328–340

    Article  Google Scholar 

  • Hsu C-W, Lin C-J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13:415–425

    Article  Google Scholar 

  • Hsu C-W, Chang C-C, Lin C-J (2003) A practical guide to support vector classication. https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf. 2016

  • Huang D, Shan C, Ardabilian M, Wang Y, Chen L (2011) Local binary patterns and its application to facial image analysis: a survey. IEEE Trans Syst Man Cybern Part C 41:765–781

    Article  Google Scholar 

  • Jangra A (2017) Verification and identification approach to maintain MVCC in cloud computing. Int J Comput Sci Appl 7:41–59

    Google Scholar 

  • Jia Z, Wang C, Yang G, Ma W (2012) A research of face feature extraction based on MB-LBP. In: Proceedings of IEEE conference on power engineering and automation, pp 69–72

  • Jia J, Xu Y, Zhang S, Xue X (2016) The facial expression recognition method of random forest based on improved PCA extracting feature. In: Proceedings of IEEE conference on communications and computing, pp 1–5

  • Jing WP, Jiang T, Zhang XG, Zhu LK (2019) The optimisation of speech recognition based on convolutional neural network. IJHPCN 13(2):222–231

    Article  Google Scholar 

  • Kumar P, Happy SL, Routray A (2017) A real-time robust facial expression recognition system using HOG features. In: Proceedings of IEEE conference on computing, analytics and security trends, pp 289–293

  • Kumari J, Rajesh R, Kumar A (2016) Fusion of features for the effective facial expression recognition. In: Proceedings of IEEE conference on communication and signal processing, pp 0457–0461

  • Li H, Sun J, Xu Z, Chen L (2017a) Multimodal 2D + 3D facial expression recognition with deep fusion convolutional neural network. IEEE Trans Multimed 19:2816–2831

    Article  Google Scholar 

  • Li Z, Nie F, Chang X, Yang Y (2017b) Beyond trace ratio: weighted harmonic mean of trace ratios for multiclass discriminant analysis. IEEE Trans Knowl Data Eng 29(10):2100–2110

    Article  Google Scholar 

  • Liao S, Zhu X, Lei Z, Zhang L, Li SZ (2007) Learning multi-scale block local binary patterns for face recognition. In: Proceedings of IEEE conference on advanced biometrics, pp 828–837

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 94–101

  • Luo Y, Wu C-M, Zhang Y (2013) Facial expression feature extraction using hybrid PCA and LBP. J China Univ Posts Telecommun 20:120–124

    Article  Google Scholar 

  • Luo M, Chang X, Nie L, Yang Y, Hauptmann AG, Zheng Q (2017) An adaptive semisupervised feature analysis for video semantic recognition. IEEE Trans Cybern 48:648–660

    Article  Google Scholar 

  • Maaten LVD, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  • Mao Q, Rao Q, Yu Y, Dong M (2017) Hierarchical Bayesian theme models for multipose facial expression recognition. IEEE Trans Multimed 19:861–873

    Article  Google Scholar 

  • Nakamura T, Maejima A, Morishima S (2014) Driver drowsiness estimation from facial expression features computer vision feature investigation using a CG model. In: Proceedings of IEEE conference on computer vision theory and applications, pp 207–214

  • Ngo T (2011) Data mining: practical machine learning tools and technique, third edition by Ian H. Witten, Eibe Frank, Mark A. Hell. ACM, New York

    Google Scholar 

  • Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24:971–987

    Article  Google Scholar 

  • Oliveira ASR, Almeida J, Ferreira J (2017) Fail silence mechanism for dependable vehicular communications. Int J High Perform Comput Netw 10:534

    Article  Google Scholar 

  • Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern B 36:433–449

    Article  Google Scholar 

  • Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern B 34:1449–1461

    Article  Google Scholar 

  • Ren F, Huang Z (2015) Facial expression recognition based on AAM–SIFT and adaptive regional weighting. IEEJ Trans Electr Electron Eng 10:713–722

    Article  Google Scholar 

  • Revina IM, Emmanuel WRS (2018) A survey on human face expression recognition techniques. J King Saud Univ Comput Inform Sci. https://doi.org/10.1016/j.jksuci.2018.09.002

    Article  Google Scholar 

  • Sadeghi H, Raie AA, Mohammadi MR (2013) Facial expression recognition using geometric normalization and appearance representation. In: Proceedings of IEEE conference on machine vision and image processing, pp 159–163

  • Sariyanidi E, Gunes H, Cavallaro A (2017) Learning bases of activity for facial expression recognition. IEEE Trans Image Process 26:1965–1978

    Article  MathSciNet  Google Scholar 

  • Sharieh A, Albdour L (2019) A heuristic approach for service allocation in cloud computing. Int J Cloud Appl Comput 7:60–74. https://doi.org/10.4018/IJCAC.2017100104

    Article  Google Scholar 

  • Shi J, Zhu H (2018) Scene image classification algorithm of fusing multi-feature. Comput Syst Appl 28(5):171–175

    Google Scholar 

  • Shojaeilangari S, Yun YW, Khwang TE (2012) Person independent facial expression analysis using Gabor features and genetic algorithm. In: Proceedings of IEEE conference on communications and signal processing, pp 1–5

  • Siddiqi MH, Ali R, Khan AM, Park YT, Lee S (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24:1386–1398

    Article  MathSciNet  Google Scholar 

  • Sun Y, Yu J (2016) Facial expression recognition by fusing Gabor and local binary pattern features. In: Proceedings of IEEE conference on multimedia modeling, pp 209–220

  • Tariq U, Huang TS (2012) Features and fusion for expression recognition—a comparative analysis. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 146–152

  • Turan C, Lam KM (2014) Region-based feature fusion for facial-expression recognition. In: Proceedings of IEEE conference on image processing, pp 5966–5970

  • Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3:71–86

    Article  Google Scholar 

  • Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp I-511–I-518

  • Wang X, Jin C, Liu W, Hu M, Xu L, Ren F (2013) Feature fusion of HOG and WLD for facial expression recognition. In: Proceedings of IEEE conference on system integration, pp 227–232

  • Wang SH, Yang W, Dong Z, Phillips P, Zhang YD (2017) Facial emotion recognition via discrete wavelet transform, principal component analysis, and cat swarm optimization. In: Proceedings of international conference on intelligent science and big data engineering, pp 203–214

  • Wu J, Song G, Huang H, Liu W, Yong X (2018) Information and communications technologies for sustainable development goals: state-of-the-art, needs and perspectives. IEEE Commun Surv Tutor 20(3):2389–2406

    Article  Google Scholar 

  • Xu X, Quan C, Ren F (2015) Facial expression recognition based on Gabor Wavelet transform and histogram of oriented gradients. In: Proceedings of IEEE conference on mechatronics and automation, pp 2117–2122

  • Xue M, Liu W, Li L (2013) Person-independent facial expression recognition via hierarchical classification. In: Proceedings of IEEE conference on intelligent sensors, sensor networks and information processing, pp 449–454

  • Yan S, Xu D, Zhang B, Zhang HJ, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29:40–51

    Article  Google Scholar 

  • Yang HY, Liu GD (2012) Face recognition based on MB-LBP operator and multi-linear PCA algorithm. Appl Res Comput 29:4728–4733

    Google Scholar 

  • Yang J, Yang J-Y (2003) Why can LDA be performed in PCA transformed space? Pattern Recognit 36:563–566

    Article  Google Scholar 

  • Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 211–216

  • Zamzami G, Ruiz G, Goldgof D, Kasturi R, Yu S, Ashmeade T (2015) Pain assessment in infants: towards spotting pain expression based on infants’ facial strain. In: Proceedings of IEEE conference on automatic face and gesture recognition, pp 1–5

  • Zhang YD, Yang ZJ, Lu HM, Zhou XX, Phillips P, Liu QM, Wang SH (2017) Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation. IEEE Access 4:8375–8385

    Article  Google Scholar 

  • Zhao X, Zhang S (2012) Facial expression recognition using local binary patterns and discriminant kernel locally linear embedding. EURASIP J Adv Signal Process 2012:20

    Article  Google Scholar 

  • Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) Learning active facial patches for expression analysis. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 2562–2569

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61866025, 61772255, 61866026, in part by the Key Research and Development Plan of Jiangxi Province under Grant 20161BBE50080, in part by the Advantage Subject Team Project of Jiangxi Province under Grant Nos. 20165BCB19007, 20152BCB24004, and the Science Technique Project of Jiangxi Province under Grant Nos. GJJ170608, GJJ170572.

Author information

Authors and Affiliations

  1. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, China

    Yan Wang & Ming Li

  2. Key Laboratory of Jiangxi Province for Image Processing and Pattern Recognition, Nanchang Hangkong University, Nanchang, 330063, China

    Yan Wang, Ming Li, Congxuan Zhang, Hao Chen & Yuming Lu

Authors
  1. Yan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Congxuan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuming Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ming Li.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by B. B. Gupta.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Li, M., Zhang, C. et al. Weighted-fusion feature of MB-LBPUH and HOG for facial expression recognition. Soft Comput 24, 5859–5875 (2020). https://doi.org/10.1007/s00500-019-04380-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-019-04380-x

Keywords

Search

Navigation