Skip to main content
SpringerLink
Log in

Texture based feature extraction using symbol patterns for facial expression recognition

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

Facial expressions can convey the internal emotions of a person within a certain scenario and play a major role in the social interaction of human beings. In automatic Facial Expression Recognition (FER) systems, the method applied for feature extraction plays a major role in determining the performance of a system. In this regard, by drawing inspiration from the Swastik symbol, three texture based feature descriptors named Symbol Patterns (SP1, SP2 and SP3) have been proposed for facial feature extraction. SP1 generates one pattern value by comparing eight pixels within a 3\(\times\)3 neighborhood, whereas, SP2 and SP3 generates two pattern values each by comparing twelve and sixteen pixels within a 5\(\times\)5 neighborhood respectively. In this work, the proposed Symbol Patterns (SP) have been evaluated with natural, fibonacci, odd, prime, squares and binary weights for determining the optimal recognition accuracy. The proposed SP methods have been tested on MUG, TFEID, CK+, KDEF, FER2013 and FERG datasets and the results from the experimental analysis demonstrated an improvement in the recognition accuracy when compared to the existing FER methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of Data and Materials

The datasets used in this work are available in the below links. MUG: https://mug.ee.auth.gr/fed/ TFEID: https://bml.ym.edu.tw/tfeid/modules/wfdownloads/ CK+: https://www.pitt.edu/~emotion/ck-spread.htm KDEF: https://www.kdef.se/download-2/register.html FER2013: https://www.kaggle.com/msambare/fer2013 FERG:http://grail.cs.washington.edu/projects/deepexpr/ferg-2d-db.html

References

  • Aghamaleki JA, Ashkani Chenarlogh V (2019) Multi-stream cnn for facial expression recognition in limited training data. Multimed Tools Appl 78(16):22861–22882

    Article  Google Scholar 

  • Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pages 1–4. IEEE

  • Aneja D, Colburn A, Faigin G, Shapiro L, Mones B (2016) Modeling stylized character expressions via deep learning. In: Asian conference on computer vision, pages 136–153. Springer

  • Arshid S, Hussain A, Munir A, Nawaz A, Aziz S (2018) Multi-stage binary patterns for facial expression recognition in real world. Clust Comput 21(1):323–331

    Article  Google Scholar 

  • Arya R, Vimina ER (2021) Local triangular coded pattern: A texture descriptor for image classification. IETE J Res, pp 1–12

  • Ashir Abubakar M, Alaa E (2017) Facial expression recognition based on image pyramid and single-branch decision tree. Signal Image Video Process 11(6):1017–1024

    Article  Google Scholar 

  • Barman A, Dutta P (2019) Influence of shape and texture features on facial expression recognition. IET Image Proc 13(8):1349–1363

    Article  Google Scholar 

  • Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: Recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  • Carrier PL, Courville A, Goodfellow IJ, Mirza M, Bengio Y (2013) Fer-2013 face database. Universit de Montral

  • Cen S, Yu Y, Yan G, Yu M, Guo Y (2022) Multi-task facial activity patterns learning for micro-expression recognition using joint temporal local cube binary pattern. Signal Process Image Commun, p 116616

  • Chen L-F, Yen Y-S (2007) Taiwanese facial expression image database. Institute of Brain Science, National Yang-Ming University, Taipei, Taiwan, Brain Mapping Laboratory

  • Chirra VRR, Uyyala SR, Kolli VKK (2021) Virtual facial expression recognition using deep cnn with ensemble learning. J Ambient Intell Hum Comput, pp 1–19

  • Dharanya V, Raj ANJ, Gopi Varun P (2021) Facial expression recognition through person-wise regeneration of expressions using auxiliary classifier generative adversarial network (ac-gan) based model. J Vis Commun Image Represent 77:103110

    Article  Google Scholar 

  • Fan X, Jiang M, Shahid AR, Yan H (2022) Hierarchical scale convolutional neural network for facial expression recognition. Cognit Neurodyn, pp 1–12

  • Farkhod Makhmudkhujaev Md, Iqbal TB, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turkish J Electr Eng Comput Sci 27(1):516–531

    Article  Google Scholar 

  • Farkhod Makhmudkhujaev M, Tauhid A-A-WM, Iqbal B, Ryu B, Chae O (2019) Facial expression recognition with local prominent directional pattern. Signal Process Image Commun 74:1–12

    Article  Google Scholar 

  • Fussell SR (2002) The verbal communication of emotion: Introduction and overview. In: The verbal communication of emotions, pp 9–24. Psychology Press

  • Goeleven E, De Raedt R, Leyman L, Verschuere B (2008) The karolinska directed emotional faces: a validation study. Cogn Emot 22(6):1094–1118

    Article  Google Scholar 

  • Iqbal MTB, Abdullah-Al-Wadud M, Ryu B, Makhmudkhujaev F, Chae O (2018) Facial expression recognition with neighborhood-aware edge directional pattern (nedp). IEEE Trans Affect Comput 11(1):125–137

    Article  Google Scholar 

  • Kalsum T, Mehmood Z, Kulsoom F, Chaudhry HN, Khan AR, Rashid M, Saba T (2021) Localization and classification of human facial emotions using local intensity order pattern and shape-based texture features. J Intell Fuzzy Syst, (Preprint):1–21

  • Karnati M, Seal A, Yazidi A, Krejcar O (2021) Fer-net: Facial expression recognition using deep neural net. Neural Comput Appl 33:9125–9136

    Article  Google Scholar 

  • Kartheek MN, Prasad MV, Bhukya R (2020) Local optimal oriented pattern for person independent facial expression recognition. In: Twelfth International Conference on Machine Vision (ICMV 2019), volume 11433, pages 114330R1–8. International Society for Optics and Photonics

  • Kartheek MN, Prasad MV, Bhukya R (2021) Radial mesh pattern: a handcrafted feature descriptor for facial expression recognition. J Ambient Intell Human Comput, pp 1–13

  • Kola DGR, Samayamantula SK (2021) A novel approach for facial expression recognition using local binary pattern with adaptive window. Multimed Tools Appl 80(2):2243–2262

    Article  Google Scholar 

  • Kola DGR, Samayamantula SK (2021a) Facial expression recognition using singular values and wavelet-based lgc-hd operator. IET Biometrics

  • Koley S, Roy H, Bhattacharjee D (2021) Gammadion binary pattern of shearlet coefficients (gbpsc): an illumination-invariant heterogeneous face descriptor. Pattern Recogn Lett 145:30–36

    Article  Google Scholar 

  • Kung H-W, Yi-Han T, Hsu C-T (2015) Dual subspace nonnegative graph embedding for identity-independent expression recognition. IEEE Trans Inf Forensics Secur 10(3):626–639

    Article  Google Scholar 

  • Lai C-C, Ko C-H (2014) Facial expression recognition based on two-stage features extraction. Optik-Int J Light Electron Opt 125(22):6678–6680

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 Ieee computer society conference on computer vision and pattern recognition-workshops, pages 94–101. IEEE

  • Majumder A, Behera L, Subramanian VK (2016) Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans Cybern 48(1):103–114

    Article  PubMed  Google Scholar 

  • Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kumar DK (2019) Regional adaptive affinitive patterns (radap) with logical operators for facial expression recognition. IET Image Proc 13(5):850–861

    Article  Google Scholar 

  • Michael RI, Sam EWR (2019) Mdtp: a novel multi-directional triangles pattern for face expression recognition. Multimed Tools Appl 78(18):26223–26238

    Article  Google Scholar 

  • Min H, Zheng Y, Yang C, Wang X, He L, Ren F (2019) Facial expression recognition using fusion features based on center-symmetric local octonary pattern. IEEE Access 7:29882–29890

    Article  Google Scholar 

  • Muhamed AA, Hanqi Z, Ibrahim Ali K (2017) An approach for facial expression classification. Int J Biomet 9(2):96–112

    Article  Google Scholar 

  • Reddy AH, Kolli K, Kiran YL(2021) Deep cross feature adaptive network for facial emotion classification. Signal Image Video Process, pp 1–8

  • Rivera AR, Castillo JR, Chae O (2015) Local directional texture pattern image descriptor. Pattern Recogn Lett 51:94–100

    Article  Google Scholar 

  • Rivera AR, Castillo JR, Oksam CO (2012) Local directional number pattern for face analysis: face and expression recognition. IEEE Trans Image Process 22(5):1740–1752

    Article  PubMed  Google Scholar 

  • Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018

    Article  Google Scholar 

  • SL Happy and Aurobinda Routray (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Google Scholar 

  • Sen D, Datta S, Balasubramanian R (2019) Facial emotion classification using concatenated geometric and textural features. Multimed Tools Appl 78(8):10287–10323

    Article  Google Scholar 

  • Shabat AMM, Tapamo J-R (2018) Angled local directional pattern for texture analysis with an application to facial expression recognition. IET Comput Vision 12(5):603–608

    Article  Google Scholar 

  • Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  • Shanthi P, Nickolas S (2021) An efficient automatic facial expression recognition using local neighborhood feature fusion. Multimed Tools Appl 80(7):10187–10212

    Article  Google Scholar 

  • Sun Z, Chiong R, Zheng-ping H (2020) Self-adaptive feature learning based on a priori knowledge for facial expression recognition. Knowl-Based Syst 204:106124

    Article  Google Scholar 

  • Sun Z, Zheng-ping H, Wang M, Zhao S (2017) Individual-free representation-based classification for facial expression recognition. SIViP 11(4):597–604

    Article  Google Scholar 

  • Sun Z, Zheng-ping H, Wang M, Zhao S (2019) Dictionary learning feature space via sparse representation classification for facial expression recognition. Artif Intell Rev 51(1):1–18

    Article  CAS  Google Scholar 

  • Swapna A, Bikash S, Prasad MD (2018) Anubhav: recognizing emotions through facial expression. Vis Comput 34(2):177–191

    Article  Google Scholar 

  • Taskeed Jabid Md, Kabir H, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Article  Google Scholar 

  • Tong Y, Chen R (2019) Local dominant directional symmetrical coding patterns for facial expression recognition. Comput Intell Neurosci 1–13:2019

    Google Scholar 

  • Tuncer T, Dogan S, Abdar M, Ehsan Basiri M, Plawiak P (2020) A novel facial image recognition method based on perceptual hash using quintet triple binary pattern. Multimed Tools Appl, pages 1–21

  • Turk M, Pentland A (1991) Face recognition using eigenfaces. In: Proceedings. 1991 IEEE computer society conference on computer vision and pattern recognition, pp 586–587

  • Uma Maheswari V, Varaprasad G, Viswanadha RS (2021) Local directional maximum edge patterns for facial expression recognition. J Ambient Intell Humaniz Comput 12(5):4775–4783

    Article  Google Scholar 

  • Verma M, Vipparthi SK, Singh G (2019) Hinet: Hybrid inherited feature learning network for facial expression recognition. IEEE Lett Comput Soc 2(4):36–39

    Article  Google Scholar 

  • Verma M, Sexena P, Vipparthi S, Singh G (2018) Quest: Quadriletral senary bit pattern for facial expression recognition. In: 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 1498–1503. IEEE

  • Verma M, Saxena P, Vipparthi SK, Singh G (2022) Cross-centroid ripple pattern for facial expression recognition. arXiv preprint arXiv:2201.05958

  • Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  • Wang Y, Li M, Wan X, Zhang C, Wang Y (2020) Multiparameter space decision voting and fusion features for facial expression recognition. Comput Intell Neurosci

  • Xie S, Haifeng H, Yongbo W (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recogn 92:177–191

    Article  Google Scholar 

  • Yang J, Wang X, Han S, Wang J, Park DS, Wang Y (2019) Improved real-time facial expression recognition based on a novel balanced and symmetric local gradient coding. Sensors 19(8):1899

    Article  PubMed  PubMed Central  Google Scholar 

  • Yang S, Bhanu B (2011) Facial expression recognition using emotion avatar image. In: Face and Gesture 2011, pp 866–871. IEEE

Download references

Author information

Authors and Affiliations

  1. Institute for Development and Research in Banking Technology, Hyderabad, India

    Mukku Nisanth Kartheek & Munaga V. N. K. Prasad

  2. Department of Computer Science and Engineering, National Institute of Technology, Warangal, India

    Mukku Nisanth Kartheek & Raju Bhukya

Authors
  1. Mukku Nisanth Kartheek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Munaga V. N. K. Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raju Bhukya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mukku Nisanth Kartheek.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kartheek, M.N., Prasad, M.V.N.K. & Bhukya, R. Texture based feature extraction using symbol patterns for facial expression recognition. Cogn Neurodyn 18, 317–335 (2024). https://doi.org/10.1007/s11571-022-09824-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11571-022-09824-z

Keywords

Search

Navigation