Abstract
In facial expression recognition applications, the classification accuracy decreases because of the blur, illumination and localization problems in images. Therefore, a robust emotion recognition technique is needed. In this work, a Multi-scale and Rotation-Invariant Phase Pattern (MRIPP) is proposed. The MRIPP extracts the features from facial images, and the extracted patterns are blur-insensitive, rotation-invariant and robust. The performance of classification algorithms like Fisher faces, Support Vector Machine (SVM), Extreme Learning Machine (ELM), Convolutional Neural Network (CNN) and Deep Neural Network (DNN) are analyzed. In order to reduce the time for classification, an OPTICS-based pre-processing of the features is proposed that creates a non-redundant and compressed training set to classify the test set. Ten-fold cross validation is used in experimental analysis and the performance metric classification accuracy is used. The proposed approach has been evaluated with six datasets Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK +), Multi- media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 98.2%, 97.5%, 95.6%, 35.5%, 87.7% and 82.4% for seven class emotion detection using a stack of Restricted Boltzmann Machines(RBM), which is high when compared to other latest methods.
Similar content being viewed by others
References
Agarwal S, Santra B, Mukherjee DP (2016) Anubhav: recognizing emotions through facial expression. Vis Comput. https://doi.org/10.1007/s00371-016-1323-z
Ahmed F, Kabir MH (2012) Directional ternary pattern (DTP) for facial expression recognition. In: IEEE international conference on consumer electronics, pp 265–266
Aifanti, N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In: Proc. 11th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Desenzano, Italy, April 12–14
Alphonse AS, Dharma D (2017a) A novel Monogenic Directional Pattern (MDP) and pseudo-Voigt kernel for facilitating the identification of facial emotions. J Vis Commun Image Represent 49:459–470
Alphonse AS, Dharma D (2017b) Enhanced Gabor (E-Gabor), Hypersphere-based normalization and Pearson General Kernel-based discriminant analysis for dimension reduction and classification of facial emotions. Expert Syst Appl 90:127–145
Alphonse AS, Dharma D (2018) Novel directional patterns and a generalized supervised dimension reduction system (GSDRS) for facial emotion recognition. Multimedia Tools Appl 77(8):9455–9488
Anisetti M, Bellandi V (2009) Emotional state inference using face related features. New directions in intelligent interactive multimedia systems and services-2. Springer, Berlin Heidelberg, pp 401–411
Ankerst M, Breunig MM, Kriegel HP, Sander J (1999) OPTICS: ordering points to identify the clustering structure. ACM Sigmod record 28(2):49–60
Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1859–1866.
Athisayamani S, Singh AR, Athithan T (2020) Recognition of ancient Tamil palm leaf vowel characters in historical documents using B-spline curve recognition. Proc Comput Sci 171:2302–2309
Berretti S, Amor BB, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected key points. Vis Comput 27(11):1021–1036
Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221
Burkert P, Trier F, Afzal MZ, Dengel A, Liwicki M (2015) Dexpression: deep convolutional neural network for expression recognitionarXiv preprint arXiv:1509.05371. https://arxiv.org/abs/1509.05371
Chan CH (2008) Multi-scale Local Binary Pattern Histogram for Face Recognition, PhD thesis, UNIS.
Chan CH, Kittler J, Poh N, Ahonen T, Pietikäinen M (2009) (Multiscale) local phase quantisation histogram discriminant analysis with score normalisation for robust face recognition. In: Computer vision workshops (ICCV Workshops), IEEE 12th international conference on 2009 Sep 27 pp.633–640.
Chang CC, Lin CJ (2011) LIBSVM : a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27
Dalal N, Triggs (2005) Histograms of oriented gradients for human detection, In: Computer vision and pattern recognition, CVPR 2005, IEEE Computer Society Conference pp. 1886–893.
Dhall A, Goecke R, Joshi J, Sikka K, Gedeon T (2014) Emotion recognition in the wild challenge 2014:baseline, data and protocol, ACM ICMI 2014
Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial expression databases from movies. IEEE Multimedia 19:34–41
Eddy SR (1998) Profile hidden Markov models. Bioinformatics (Oxford, England) 14(9):755–763
Ghimire D, Lee J, Li ZN, Jeong S (2016) Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools Appl 15:1–26
Haghighat M, Zonouz S, Abdel-Mottaleb M (2015) CloudID: trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916
Hao XL, Tian M (2017) Deep Belief Network based on double weber local descriptor in micro-expression recognition, In: Advanced multimedia and ubiquitous engineering. Pp 419–425.
Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feed forward neural networks. Neural networks 2:985–990
Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. Part B: IEEE Tran Syst Man, Cybernet 42(2):513–529
Iosifidis A, Tefas A, Pitas I (2017) Approximate kernel extreme learning machine for large scale data classification. Neurocomputing 219:210–220
Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI journal 32(5):784–794
Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of fourth IEEE international conference in automatic face and gesture recognition, pp. 46–53.
Keyvanrad MA, Homayounpour MM (2014) A brief survey on deep belief networks and introducing a new object oriented toolbox (DeeBNet) arXiv: 1408.3264 [cs], Aug. 2014.
Khan SA, Hussain A, Usman (2017) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features, Multimedia Tools and Applications, pp. 1–33.
Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv: 1703.07140.
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks, In Advances in neural information processing systems pp. 1097–1105
LeCun Y, Bengio Y, Hinton G, Deep learning (2015).
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101.
Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Third IEEE international conference on automatic face and gesture recognition, pp 200–205
Nguyen HT, Caplier A (2014) Patch based local phase quantization of monogenic components for face recognition, In: 2014 IEEE International conference on image processing (ICIP) pp. 229–233
Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987
Ojansivu V, Rahtu E, Heikkila J (2008) Rotation invariant local phase quantization for blur insensitive texture analysis, In 2008 19th International Conference on Pattern Recognition pp.1–4.
Ojansivu V, Heikkilä J(2008) Blur insensitive texture classification using local phase quantization, In International conference on image and signal processing pp. 236–243.
Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Multimedia and Expo, IEEE International Conference, pp. 5.
Rahtu E, Heikkilä J, Ojansivu J, Ahonen T (2012) Local phase quantization for blur-insensitive image analysis. Image Vis Comput 30(8):501–512
Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752
Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100
Shinohara Y, Otsuf N (2004) Facial expression recognition using fisher weight maps. Sixth IEEE Int Conf Auto Face Gesture Recognition Proceedings 17:499–504
Shufu X, Shiguang S, Xilin C, Jie C (2010) Fusing local patterns of gabor magnitude and phase for face recognition. Image Processing IEEE Trans 19(5):1349–1361
Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713
Song I, Kim HJ, Jeon PB (2014) Deep learning for real-time robust facial expression recognition on a smartphone, In: 2014 international conference on consumer electronics (ICCE), pp 564–567
Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300
Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650
Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd intern. workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, pp 65.
Venkateswara H, Eusebio J, Chakraborty S, Panchanathan S (2017) Deep hashing network for unsupervised domain adaptation, In: Proceedings of the IEEE conference on computer vision and pattern recognition pp.5018–5027.
Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154
Vipparthi SK, Nagar SK (2014) Expert image retrieval system using directional local motif XoR patterns. Expert Syst Appl 41(17):8016–8026
Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition, Cognitive Computation, pp.1–4.
Zhang B, Shan S, Chen X, Gao W (2006) Histogram of gabor phase patterns (HGPP): a Novel object representation approach for face recognition. IEEE TIP 16(1):57–68
Zhang K, Huang Y, Du Y (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. In: IEEE Transactions on Image Processing, pp 4193–4203
Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and and recognition, In: Proc. ICCV, 786–791
Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vision Comput 29(9):607–619
Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multi-orientation fusion histogram. Math Prob Eng 2017
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Alphonse, A.S., Shankar, K., Jeyasheela Rakkini, M.J. et al. A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification. J Ambient Intell Human Comput 12, 3447–3463 (2021). https://doi.org/10.1007/s12652-020-02517-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-020-02517-7