Skip to main content
SpringerLink
Log in

Facial emotion recognition and encoding application for the visually impaired

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Visual impairment refers to any kind of vision loss ranging from complete blindness to the partial loss of vision. Studies have shown that social understanding can be severely affected due to visual impairment. This paper tries to address this gap by presenting a socially assistive application for the visually impaired. The study develops the application using deep learning and app development technologies. A transfer learning Facial Expression Recognition (FER) model is embedded in a mobile application to recognize facial expressions. Guided by haptic feedback, the application developed in this study helps users perceive expressions of the person(s) they are interacting with. The practical value of this work lies in assisting and enhancing the social understanding of visually impaired individuals. On the other hand, the research value of the current study lies in the development of a novel application which is faster, lighter, implementable on low-end devices, and achieves better accuracy, on par with state-of-the-art models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. McAlpine LM, Moore CL (1995) The development of social understanding in children with visual impairments. J Visual Impair Blind 89:349–358

    Article  Google Scholar 

  2. Sheaffer BL, Golden JA, Averett P (2009) Facial expression recognition deficits and faulty learning: implications for theoretical models and clinical applications. Int J Behav Consult Therapy 5(1):31

    Article  Google Scholar 

  3. MA M (2020) MMA facial expression. In: Kaggle. https://www.kaggle.com/mahmoudima/mma-facial-expression. Accessed 15 Nov 2021

  4. Sambare M (2020) FER-2013. In: Kaggle. https://www.kaggle.com/msambare/fer2013. Accessed 15 Nov 2021

  5. Li S, Deng W (2020) Deep facial expression recognition: a survey. IEEE Trans Affect Comput 13(3):1195–1215

    Article  Google Scholar 

  6. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV). https://doi.org/10.1109/WACV.2016.7477450

  7. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face and gesture recognition (FG 2017). https://doi.org/10.1109/FG.2017.140

  8. Wang K, Peng X, Yang J, Meng D, Qiao Y (2020) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069

    Article  MATH  Google Scholar 

  9. Savchenko AV (2021) Facial expression and attributes recognition based on multi-task learning of lightweight neural networks. In: 2021 IEEE 19th international symposium on intelligent systems and informatics (SISY). https://doi.org/10.1109/SISY52375.2021.9582508

  10. Knyazev B, Shvetsov R, Efremova N, Kuharenko A (2017) Convolutional neural networks pretrained on large face recognition datasets for emotion classification from video. In: arXiv.org. https://arxiv.org/abs/1711.04598. Accessed 15 Nov 2021

  11. Liu C, Tang T, Lv K, Wang M (2018) Multi-feature based emotion recognition for video clips. In: Proceedings of the 20th ACM international conference on multimodal interaction. https://doi.org/10.1145/3242969.3264989

  12. Deng J, Dong W, Socher R, Li LJ, Kai L, Fei-Fei L (2009). ImageNet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognit. https://doi.org/10.1109/cvpr.2009.5206848

  13. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) MobileNetV2: Inverted residuals and linear bottlenecks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition. https://doi.org/10.1109/CVPR.2018.00474

  14. Saurav S, Saini R, Singh S (2021) EmNet: a deep integrated convolutional neural network for facial emotion recognition in the wild. Appl Intell 51:5543–5570. https://doi.org/10.1007/s10489-020-02125-0

    Article  Google Scholar 

  15. Knyazev B, Shvetsov R, Efremova N and Kuharenko A (2018) Leveraging large face recognition data for emotion classification. In: 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp 692–696. https://doi.org/10.1109/FG.2018.00109

  16. Dandıl E, Özdemir R (2019) Real-time facial emotion classification using deep learning. Data Sci Appl 2(1):13–17

    Google Scholar 

  17. Kusuma GP, Jonathan AP, Lim AP (2020) Emotion recognition on fer-2013 face images using fine-tuned vgg-16. Adv Sci Technol Eng Syst J 5(6):315–322

    Article  Google Scholar 

  18. Theckedath D, Sedamkar RR (2020) Detecting affect states using VGG16, ResNet50 and SE-ResNet50 networks. SN COMPUT SCI 1:79. https://doi.org/10.1007/s42979-020-0114-9

    Article  Google Scholar 

  19. Dubey AK, Jain V (2020) Automatic facial recognition using VGG16 based transfer learning model. J Inf Optim Sci 41(7):1589–1596

    Google Scholar 

  20. Mehendale N (2020) Facial emotion recognition using convolutional neural networks (FERC). SN Appl Sci 2(3):1–8. https://doi.org/10.1007/s42452-020-2234-1

    Article  Google Scholar 

  21. Riaz MN, Shen Y, Sohail M, Guo M (2020) eXnet: An efficient approach for emotion recognition in the wild. Sensors 20(4):1087

    Article  Google Scholar 

  22. Anand R, Shanthi T, Nithish MS, Lakshman S (2019) Face recognition and classification using googlenet architecture. Adv Intell Syst Comput 1084:261–269. https://doi.org/10.1007/978-981-15-0035-0_20

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Science and Engineering, Ramaiah Institute of Technology, Bangalore, Karnataka, India

    M. N Pushpalatha, Harshubh Meherishi, Avani Vaishnav, R. Anurag Pillai & Aman Gupta

Authors
  1. M. N Pushpalatha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harshubh Meherishi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Avani Vaishnav
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. R. Anurag Pillai
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aman Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. N Pushpalatha.

Ethics declarations

Conflict of interest

The authors declare that there is no competing interest or conflict of interest in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pushpalatha, M.N., Meherishi, H., Vaishnav, A. et al. Facial emotion recognition and encoding application for the visually impaired. Neural Comput & Applic 35, 749–755 (2023). https://doi.org/10.1007/s00521-022-07807-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07807-z

Keywords

Search

Navigation