A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification

Abstract

In facial expression recognition applications, the classification accuracy decreases because of the blur, illumination and localization problems in images. Therefore, a robust emotion recognition technique is needed. In this work, a Multi-scale and Rotation-Invariant Phase Pattern (MRIPP) is proposed. The MRIPP extracts the features from facial images, and the extracted patterns are blur-insensitive, rotation-invariant and robust. The performance of classification algorithms like Fisher faces, Support Vector Machine (SVM), Extreme Learning Machine (ELM), Convolutional Neural Network (CNN) and Deep Neural Network (DNN) are analyzed. In order to reduce the time for classification, an OPTICS-based pre-processing of the features is proposed that creates a non-redundant and compressed training set to classify the test set. Ten-fold cross validation is used in experimental analysis and the performance metric classification accuracy is used. The proposed approach has been evaluated with six datasets Japanese Female Facial Expression (JAFFE), Cohn Kanade (CK +), Multi- media Understanding Group (MUG), Static Facial Expressions in the Wild (SFEW), Oulu-Chinese Academy of Science, Institute of Automation (Oulu-CASIA) and Man–Machine Interaction (MMI) datasets to meet a classification accuracy of 98.2%, 97.5%, 95.6%, 35.5%, 87.7% and 82.4% for seven class emotion detection using a stack of Restricted Boltzmann Machines(RBM), which is high when compared to other latest methods.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

References

  1. Agarwal S, Santra B, Mukherjee DP (2016) Anubhav: recognizing emotions through facial expression. Vis Comput. https://doi.org/10.1007/s00371-016-1323-z

    Article  Google Scholar 

  2. Ahmed F, Kabir MH (2012) Directional ternary pattern (DTP) for facial expression recognition. In: IEEE international conference on consumer electronics, pp 265–266

  3. Aifanti, N, Papachristou C, Delopoulos A (2010) The MUG facial expression database. In: Proc. 11th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Desenzano, Italy, April 12–14

  4. Alphonse AS, Dharma D (2017a) A novel Monogenic Directional Pattern (MDP) and pseudo-Voigt kernel for facilitating the identification of facial emotions. J Vis Commun Image Represent 49:459–470

    Article  Google Scholar 

  5. Alphonse AS, Dharma D (2017b) Enhanced Gabor (E-Gabor), Hypersphere-based normalization and Pearson General Kernel-based discriminant analysis for dimension reduction and classification of facial emotions. Expert Syst Appl 90:127–145

    Article  Google Scholar 

  6. Alphonse AS, Dharma D (2018) Novel directional patterns and a generalized supervised dimension reduction system (GSDRS) for facial emotion recognition. Multimedia Tools Appl 77(8):9455–9488

    Article  Google Scholar 

  7. Anisetti M, Bellandi V (2009) Emotional state inference using face related features. New directions in intelligent interactive multimedia systems and services-2. Springer, Berlin Heidelberg, pp 401–411

    Chapter  Google Scholar 

  8. Ankerst M, Breunig MM, Kriegel HP, Sander J (1999) OPTICS: ordering points to identify the clustering structure. ACM Sigmod record 28(2):49–60

    Article  Google Scholar 

  9. Asthana A, Zafeiriou S, Cheng S, Pantic M (2014) Incremental face alignment in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1859–1866.

  10. Athisayamani S, Singh AR, Athithan T (2020) Recognition of ancient Tamil palm leaf vowel characters in historical documents using B-spline curve recognition. Proc Comput Sci 171:2302–2309

  11. Berretti S, Amor BB, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected key points. Vis Comput 27(11):1021–1036

    Article  Google Scholar 

  12. Bousmalis K, Mehu M, Pantic M (2013) Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: a survey of related cues, databases, and tools. Image Vis Comput 31(2):203–221

    Article  Google Scholar 

  13. Burkert P, Trier F, Afzal MZ, Dengel A, Liwicki M (2015) Dexpression: deep convolutional neural network for expression recognitionarXiv preprint arXiv:1509.05371. https://arxiv.org/abs/1509.05371

  14. Chan CH (2008) Multi-scale Local Binary Pattern Histogram for Face Recognition, PhD thesis, UNIS.

  15. Chan CH, Kittler J, Poh N, Ahonen T, Pietikäinen M (2009) (Multiscale) local phase quantisation histogram discriminant analysis with score normalisation for robust face recognition. In: Computer vision workshops (ICCV Workshops), IEEE 12th international conference on 2009 Sep 27 pp.633–640.

  16. Chang CC, Lin CJ (2011) LIBSVM : a library for support vector machines. ACM Trans Intell Syst Technol 2(3):27

    Article  Google Scholar 

  17. Dalal N, Triggs (2005) Histograms of oriented gradients for human detection, In: Computer vision and pattern recognition, CVPR 2005, IEEE Computer Society Conference pp. 1886–893.

  18. Dhall A, Goecke R, Joshi J, Sikka K, Gedeon T (2014) Emotion recognition in the wild challenge 2014:baseline, data and protocol, ACM ICMI 2014

  19. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial expression databases from movies. IEEE Multimedia 19:34–41

    Article  Google Scholar 

  20. Eddy SR (1998) Profile hidden Markov models. Bioinformatics (Oxford, England) 14(9):755–763

    Article  Google Scholar 

  21. Ghimire D, Lee J, Li ZN, Jeong S (2016) Recognition of facial expressions based on salient geometric features and support vector machines. Multimedia Tools Appl 15:1–26

    Google Scholar 

  22. Haghighat M, Zonouz S, Abdel-Mottaleb M (2015) CloudID: trustworthy cloud-based and cross-enterprise biometric identification. Expert Syst Appl 42(21):7905–7916

    Article  Google Scholar 

  23. Hao XL, Tian M (2017) Deep Belief Network based on double weber local descriptor in micro-expression recognition, In: Advanced multimedia and ubiquitous engineering. Pp 419–425.

  24. Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feed forward neural networks. Neural networks 2:985–990

    Google Scholar 

  25. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. Part B: IEEE Tran Syst Man, Cybernet 42(2):513–529

    Google Scholar 

  26. Iosifidis A, Tefas A, Pitas I (2017) Approximate kernel extreme learning machine for large scale data classification. Neurocomputing 219:210–220

    Article  Google Scholar 

  27. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI journal 32(5):784–794

    Article  Google Scholar 

  28. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings of fourth IEEE international conference in automatic face and gesture recognition, pp. 46–53.

  29. Keyvanrad MA, Homayounpour MM (2014) A brief survey on deep belief networks and introducing a new object oriented toolbox (DeeBNet) arXiv: 1408.3264 [cs], Aug. 2014.

  30. Khan SA, Hussain A, Usman (2017) Reliable facial expression recognition for multi-scale images using weber local binary image based cosine transform features, Multimedia Tools and Applications, pp. 1–33.

  31. Kim Y, Yoo B, Kwak Y, Choi C, Kim J (2017) Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv: 1703.07140.

  32. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks, In Advances in neural information processing systems pp. 1097–1105

  33. LeCun Y, Bengio Y, Hinton G, Deep learning (2015).

  34. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 94–101.

  35. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Third IEEE international conference on automatic face and gesture recognition, pp 200–205

  36. Nguyen HT, Caplier A (2014) Patch based local phase quantization of monogenic components for face recognition, In: 2014 IEEE International conference on image processing (ICIP) pp. 229–233

  37. Ojala T, Pietikainen M, Maenpaa T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987

    Article  Google Scholar 

  38. Ojansivu V, Rahtu E, Heikkila J (2008) Rotation invariant local phase quantization for blur insensitive texture analysis, In 2008 19th International Conference on Pattern Recognition pp.1–4.

  39. Ojansivu V, Heikkilä J(2008) Blur insensitive texture classification using local phase quantization, In International conference on image and signal processing pp. 236–243.

  40. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In Multimedia and Expo, IEEE International Conference, pp. 5.

  41. Rahtu E, Heikkilä J, Ojansivu J, Ahonen T (2012) Local phase quantization for blur-insensitive image analysis. Image Vis Comput 30(8):501–512

    Article  Google Scholar 

  42. Ramirez Rivera A, Rojas Castillo J, Chae O (2013) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    MathSciNet  Article  Google Scholar 

  43. Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100

    Article  Google Scholar 

  44. Shinohara Y, Otsuf N (2004) Facial expression recognition using fisher weight maps. Sixth IEEE Int Conf Auto Face Gesture Recognition Proceedings 17:499–504

    Article  Google Scholar 

  45. Shufu X, Shiguang S, Xilin C, Jie C (2010) Fusing local patterns of gabor magnitude and phase for face recognition. Image Processing IEEE Trans 19(5):1349–1361

    MathSciNet  Article  Google Scholar 

  46. Siddiqi MH, Lee S, Lee YK, Khan AM, Truc P (2013) Hierarchical recognition scheme for human facial expression recognition systems. Sensors 13:16682–16713

    Article  Google Scholar 

  47. Song I, Kim HJ, Jeon PB (2014) Deep learning for real-time robust facial expression recognition on a smartphone, In: 2014 international conference on consumer electronics (ICCE), pp 564–567

  48. Suykens JA, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300

    Article  Google Scholar 

  49. Tan X, Triggs B (2010) Enhanced local texture feature sets for face recognition under difficult lighting conditions. IEEE Trans Image Process 19(6):1635–1650

    MathSciNet  Article  Google Scholar 

  50. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the MMI facial expression database. In: Proc. 3rd intern. workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect, pp 65.

  51. Venkateswara H, Eusebio J, Chakraborty S, Panchanathan S (2017) Deep hashing network for unsupervised domain adaptation, In: Proceedings of the IEEE conference on computer vision and pattern recognition pp.5018–5027.

  52. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vision 57(2):137–154

    Article  Google Scholar 

  53. Vipparthi SK, Nagar SK (2014) Expert image retrieval system using directional local motif XoR patterns. Expert Syst Appl 41(17):8016–8026

    Article  Google Scholar 

  54. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition, Cognitive Computation, pp.1–4.

  55. Zhang B, Shan S, Chen X, Gao W (2006) Histogram of gabor phase patterns (HGPP): a Novel object representation approach for face recognition. IEEE TIP 16(1):57–68

    MathSciNet  Google Scholar 

  56. Zhang K, Huang Y, Du Y (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. In: IEEE Transactions on Image Processing, pp 4193–4203

  57. Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local gabor binary pattern histogram sequence (LGBPHS): a novel non-statistical model for face representation and and recognition, In: Proc. ICCV, 786–791

  58. Zhao G, Huang X, Taini M, Li SZ, Pietikäinen M (2011) Facial expression recognition from near-infrared videos. Image Vision Comput 29(9):607–619

    Article  Google Scholar 

  59. Zhao L, Wang Z, Zhang G (2017) Facial expression recognition from video sequences based on spatial-temporal motion local binary pattern and gabor multi-orientation fusion histogram. Math Prob Eng 2017

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to A. Robert Singh.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Alphonse, A.S., Shankar, K., Jeyasheela Rakkini, M.J. et al. A multi-scale and rotation-invariant phase pattern (MRIPP) and a stack of restricted Boltzmann machine (RBM) with preprocessing for facial expression classification. J Ambient Intell Human Comput 12, 3447–3463 (2021). https://doi.org/10.1007/s12652-020-02517-7

Download citation

Keyword

  • Emotion
  • Pattern
  • Classification
  • Feature
  • Texton