Abstract
Facial expression recognition is a challenging problem in computer vision. Due to the limited feature extraction capability of a single feature descriptor, this paper proposes a facial expression recognition method that iteratively fuses classifiers based on multi-orientation gradient calculated HOG (MO-HOG) features and deep-learned features. Diagonal orientation gradient calculated HOG (D-HOG) is a complementary part to the histogram of oriented gradient (HOG), which is proposed to obtain the diagonal gradient information and combines HOG to form a novel feature descriptor MO-HOG. Our method extracts MO-HOG features from whole facial images and expression-rich local facial images. Meanwhile, deep-learned features are not reliable enough on small databases but contain high-level semantic information, so the deep network is designed to extract effective deep-learned features. In addition, a classifier fusion method based on an optimization algorithm is proposed, and the best-fused classifier is obtained through iteration. The experiments are evaluated on the public databases (CK+ and JAFFE). The proposed method shows the effectiveness of facial expression recognition and outperforms the state-of-the-art methods. The recognition accuracy is 97.70% on the CK+ database and 97.64% on the JAFFE database.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Lei Y, Yuan W, Wang H, Wenhu Y, Bo W (2017) A skin segmentation algorithm based on stacked autoencoders. IEEE Trans Multimed 19:740–749
Zhang L, Tjondronegoro D, Chandran V (2014) Random Gabor based templates for facial expression recognition in images with facial occlusion. Neurocomputing 145:451–464
Chen J, Takiguchi T, Ariki Y (2017) Rotation-reversal invariant HOG cascade for facial expression recognition. Signal Image Video Process 11:1485–1492
Mlakar U, Potočnik B (2015) Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal Image Video Process 9:245–253
Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 1805–1812
Liu M, Li S, Shan S, Chen X (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing 159:126–136
Liu Y, Zeng J, Shan S, Zheng Z (2018) Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), pp 458–465
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): A complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101
Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition, pp 200–205
Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Effect Comput 6:1–12
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2005, pp 886–893
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp 1097–1105
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv Preprint arXiv:1409.1556
Yuan X, Xie L, Abouelenien M (2018) A regularized ensemble framework of deep learning for cancer detection from multi-class, imbalanced training data. Pattern Recognit 77:160–172
Yuan X, Abouelenien M (2006) A multi-class boosting method for learning from imbalanced data. Int J Granul Comput 4:13–29
Kennedy J (2011) Particle swarm optimization. In: Encyclopedia of machine learning. Springer, pp 760–766
Jia Y, Shelhamer E, Donahue J, Karayev S, Long J (2014) Caffe: convolutional architecture for fast feature embedding, pp 675–678
Carrier P-L, Courville A, Goodfellow IJ, Mirza M, Bengio Y (2013) FER-2013 face database. Univ. Montral
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628
Sun Y, Wen G (2017) Cognitive facial expression recognition with constrained dimensionality reduction. Neurocomputing 230:397–408
Owusu E, Zhan Y, Mao QR (2014) A neural-AdaBoost based facial expression recognition system. Expert Syst Appl 41:3383–3390
Liu Y, Xie Z, Yuan X, Chen J, Song W (2017) Multi-level structured hybrid forest for joint head detection and pose estimation. Neurocomputing 266:206–215
Acknowledgements
This work is supported by the Natural Science Foundation of Anhui Province (1708085MF146), Science and Technology Support Project of Sichuan Province (2016GZ0389), Project of Innovation Team of Ministry of Education of China (IRT17R32), and the Fundamental Research Funds for the Central Universities (No. PA2018GDQT0011).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Wang, H., Wei, S. & Fang, B. Facial expression recognition using iterative fusion of MO-HOG and deep features. J Supercomput 76, 3211–3221 (2020). https://doi.org/10.1007/s11227-018-2554-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-018-2554-8