Skip to main content

Advertisement

SpringerLink
Log in

Human facial emotion recognition using improved black hole based extreme learning machine

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial Emotion Recognition (FER) plays an essential role in human-to-human communication and human-to-machine interaction. Based on the analysis of the facial expressions, the machine can understand the emotional status of the human and take suitable actions. A huge amount of works was done by researchers for decades to build FER systems that are able to discriminate facial emotion features and identify their categories. In this paper, a novel FER framework is suggested to overcome the drawbacks of the previous systems. The Extreme Learning Machine (ELM) universal approximation characteristic along with the Improved Black Hole algorithm global search ability are combined and used to classify the facial images. The Linear Discriminant Analysis (LDA) and Principal Component Analysis (PCA) are utilized to reduce the dimensions of the face images and keep the most discriminative features before feeding them into our system. The proposed system is evaluated over Japanese female facial expression (JAFFE), Karolinska directed emotional faces (KDEF), and extended Cohn-Kanade datasets (CK+), and succeeded to achieve an accuracy of more than 90% over all the datasets. The experiments are extended by testing the proposed system over our own designed facial dataset where the acquired accuracy of the LDA-BH-ELM approach reached 77%, 80% over CK+, KDEF datasets respectively. The comparison of results with the previous methods proved the efficacy and effectiveness of the proposed system, and its ability to achieve outstanding performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Chen CH (2015) Handbook of pattern recognition and computer vision. World Scientific

  2. Chen C-H, Lee I-J, Lin L-Y (2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res Dev Disabil 36:396–403

    Article  Google Scholar 

  3. Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18:32–80

    Article  Google Scholar 

  4. Deeb H, Sarangi A, Mishra D, Sarangi SK (2020) Improved black hole optimization algorithm for data clustering. J King Saud Univ Inf Sci

  5. Deng W-Y, Zheng Q-H, Lian S, Chen L, Wang X (2010) Ordinal extreme learning machine. Neurocomputing 74:447–456

    Article  Google Scholar 

  6. Friesen E, Ekman P (1978) Facial action coding system: a technique for the measurement of facial movement. Palo Alto 3:5

    Google Scholar 

  7. Fu K-S (2019) Applications of pattern recognition. CRC press

    Google Scholar 

  8. Ghojogh B, Crowley M (2019) Linear and quadratic discriminant analysis: tutorial. arXiv Prepr arXiv190602590

  9. Hatamlou A (2013) Black hole: a new heuristic optimization approach for data clustering. Inf Sci (Ny) 222:175–184. https://doi.org/10.1016/j.ins.2012.08.023

  10. Hickson S, Dufour N, Sud A, et al (2019) Eyemotion: classifying facial expressions in VR using eye-tracking cameras. In: 2019 IEEE winter conference on applications of computer vision (WACV). Pp 1626–1635

  11. Huang G-B, Chen L (2007) Convex incremental extreme learning machine. Neurocomputing 70:3056–3062

    Article  Google Scholar 

  12. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks (IEEE cat. No. 04CH37541). Pp 985–990

  13. Huang G-B, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man, Cybern Part B 42:513–529

    Article  Google Scholar 

  14. Huang X, Wu L, Ye Y (2019) A review on dimensionality reduction techniques. Int J Pattern Recognit Artif Intell 33:1950017

    Article  Google Scholar 

  15. Islam B, Mahmud F, Hossain A, et al (2018) Human facial expression recognition system using artificial neural network classification of Gabor feature based facial expression information. In: 2018 4th international conference on electrical engineering and Information & Communication Technology (iCEEiCT). Pp 364–368

  16. Islam B, Mahmud F, Hossain A (2018) Facial region segmentation based emotion recognition using extreme learning machine. In: 2018 international conference on advancement in electrical and electronic engineering (ICAEEE). Pp 1–4

  17. Izard CE (1971) The face of emotion.

  18. Jain AK, Duin RPW, Mao J (2000) Statistical pattern recognition: a review. IEEE Trans Pattern Anal Mach Intell 22:4–37

    Article  Google Scholar 

  19. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106

    Article  Google Scholar 

  20. Jasiewicz J, Stepinski TF (2013) Geomorphons—a pattern recognition approach to classification and mapping of landforms. Geomorphology 182:147–156

    Article  Google Scholar 

  21. Jolliffe IT, Cadima J (2016) Principal component analysis: a review and recent developments. Philos Trans R Soc A Math Phys Eng Sci 374:20150202

    Article  MathSciNet  Google Scholar 

  22. Kalantarian H, Jedoui K, Washington P, Tariq Q, Dunlap K, Schwartz J, Wall DP (2019) Labeling images with facial emotion and the potential for pediatric healthcare. Artif Intell Med 98:77–86

    Article  Google Scholar 

  23. Khan S, Hussain M, Aboalsamh H, Bebis G (2017) A comparison of different Gabor feature extraction approaches for mass classification in mammography. Multimed Tools Appl 76:33–57

    Article  Google Scholar 

  24. Kim J-H, Kim B-G, Roy PP, Jeong D-M (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Article  Google Scholar 

  25. Lan Y, Soh YC, Huang G-B (2010) Two-stage extreme learning machine for regression. Neurocomputing 73:3028–3038

    Article  Google Scholar 

  26. Liang N-Y, Huang G-B, Saratchandran P, Sundararajan N (2006) A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Trans Neural Netw 17:1411–1423

    Article  Google Scholar 

  27. Liu Z-T, Sui G-T, Li D-Y, Tan G-Z (2015) A novel facial expression recognition method based on extreme learning machine. In: 2015 34th Chinese control conference (CCC). Pp 3852–3857

  28. Liu Z-T, Li S-H, Cao W-H et al (2019) Combining 2D gabor and local binary pattern for facial expression recognition using extreme learning machine. J Adv Comput Intell Intell Inform 23:444–455

    Article  Google Scholar 

  29. Lucey P, Cohn JF, Kanade T, et al (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 ieee computer society conference on computer vision and pattern recognition-workshops. Pp 94–101

  30. Lundqvist D, Flykt A, Ohman A (1998) The Karolinska directed emotional faces (KDEF). CD ROM from dep. Clin. Neurosci. Psychol. Sect. Karolinska Institutet 91–630

  31. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings Third IEEE international conference on automatic face and gesture recognition. pp. 200–205

  32. Martinez AM, Kak AC (2001) Pca versus lda. IEEE Trans Pattern Anal Mach Intell 23:228–233

    Article  Google Scholar 

  33. Mehrabian A (2008) Communication without words. Commun Theory 6:193–200

    Google Scholar 

  34. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV). Pp 1–10

  35. Moore S, Bowden R (2011) Local binary patterns for multi-view facial expression recognition. Comput Vis Image Underst 115:541–558

    Article  Google Scholar 

  36. Paolanti M, Frontoni E (2020) Multidisciplinary pattern recognition applications: a review. Comput Sci Rev 37:100276

    Article  MathSciNet  Google Scholar 

  37. Prince SJD, Elder JH (2007) Probabilistic linear discriminant analysis for inferences about identity. In: 2007 IEEE 11th international conference on computer vision. Pp 1–8

  38. Rahulamathavan Y, Phan RC-W, Chambers JA, Parish DJ (2012) Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans Affect Comput 4:83–92

    Article  Google Scholar 

  39. Rao Q, Qu X, Mao Q, Zhan Y (2015) Multi-pose facial expression recognition based on SURF boosting. In: 2015 international conference on affective computing and intelligent interaction (ACII). Pp 630–635

  40. Rujirakul K, So-In C (2018) Histogram equalized deep PCA with ELM classification for expressive face recognition. In: 2018 international workshop on advanced image technology (IWAIT). Pp 1–4

  41. Shah JH, Sharif M, Yasmin M, Fernandes SL (2017) Facial expressions classification and false label reduction using LDA and threefold SVM. Pattern Recogn Lett

  42. Siddiqi MH, Ali R, Khan AM, Young-Tack Park, Sungyoung Lee (2015) Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans Image Process 24:1386–1398

    Article  MathSciNet  Google Scholar 

  43. Tsai H-H, Chang Y-C (2018) Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Comput 22:4389–4405

    Article  Google Scholar 

  44. Tzeng D-Y, Berns RS (2005) A review of principal component analysis and its applications to color technology. Color res Appl endorsed by inter-society color Counc colour gr (Great Britain), can Soc color color Sci Assoc Japan. Dutch Soc Study Color Swedish Colour Cent Found Colour Soc 30:84–98

    Google Scholar 

  45. Vikram K, Padmavathi S (2017) Facial parts detection using Viola Jones algorithm. In: 2017 4th international conference on advanced computing and communication systems (ICACCS). Pp 1–4

  46. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001. Pp I--I

  47. Wang X, Tang X (2004) Dual-space linear discriminant analysis for face recognition. In: proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. Pp II--II

  48. Xanthopoulos P, Pardalos PM, Trafalis TB (2013) Linear discriminant analysis. In: Robust data mining. Springer, pp. 27–33

  49. Xie S, Hu H (2018) Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans Multimed 21:211–220

    Article  Google Scholar 

  50. Yang B, Chen S (2013) A comparative study on local binary pattern (LBP) based face recognition: LBP histogram versus LBP image. Neurocomputing 120:365–379

    Article  Google Scholar 

  51. Ye J (2005) Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J Mach Learn Res 6:483–502

    MathSciNet  MATH  Google Scholar 

  52. Yu H, Yang J (2001) A direct LDA algorithm for high-dimensional data—with application to face recognition. Pattern Recogn 34:2067–2070

    Article  Google Scholar 

  53. Zhai J, Zang L, Zhou Z (2018) Ensemble dropout extreme learning machine via fuzzy integral for data classification. Neurocomputing 275:1043–1052

    Article  Google Scholar 

  54. Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Comput Games Technol 2008:1–7

    Article  Google Scholar 

  55. Zhang R, Lan Y, Huang G, Xu Z-B (2012) Universal approximation of extreme learning machine with adaptive growth of hidden nodes. IEEE Trans Neural Networks Learn Syst 23:365–371

    Article  Google Scholar 

  56. Zhao X, Zhang S (2012) Facial expression recognition based on local binary patterns and least squares support vector machines. Lect Notes Electr Eng 140 LNEE:707–712. https://doi.org/10.1007/978-3-642-27296-7_106

  57. Zhao Y, Chen D (2020) A Facial Expression Recognition Method Using Improved Capsule Network Model Sci Program 2020

  58. Zhou Y, Shi BE (2017) Action unit selective feature maps in deep networks for facial expression recognition. In: 2017 international joint conference on neural networks (IJCNN). Pp 2031–2038

  59. Zhu W, Miao J, Qing L, Huang G-B (2015) Hierarchical extreme learning machine for unsupervised representation learning. In: 2015 international joint conference on neural networks (ijcnn). Pp 1–8

Download references

Acknowledgments

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Availability of data and material

The data that support the findings of this study are available from the corresponding author, upon reasonable request.

Code availability

Not applicable.

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Siksha ‘O’ Anusandhan Deemed to be University, Bhubaneswar, Odisha, India

    Hasan Deeb, Archana Sarangi & Debahuti Mishra

  2. Department of Electronics & Communication Engineering, Siksha ‘O’ Anusandhan Deemed to be University, Bhubaneswar, Odisha, India

    Shubhendu Kumar Sarangi

Authors
  1. Hasan Deeb
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Archana Sarangi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Debahuti Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shubhendu Kumar Sarangi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Archana Sarangi.

Ethics declarations

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Deeb, H., Sarangi, A., Mishra, D. et al. Human facial emotion recognition using improved black hole based extreme learning machine. Multimed Tools Appl 81, 24529–24552 (2022). https://doi.org/10.1007/s11042-022-12498-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12498-8

Keywords

Search

Navigation