Skip to main content
SpringerLink
Log in

Deep learning inspired intelligent embedded system for haptic rendering of facial emotions to the blind

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In our day-to-day social interactions, non-verbal cues such as facial emotions play a vital role. These cues assist people in understanding and inferring the hidden emotional state of the individuals. However, blind and visually impaired persons (VIPs) sadly lack access to such cues, which results in impaired interpersonal communication. To alleviate the issue, in this research, we present a proof-of-concept (POC) implementation of a deep learning-inspired vision-based low-cost intelligent embedded system for the haptic rendering of facial emotions to the VIPs. To this end, a novel lightweight shallow convolutional neural network (CNN) has been designed, optimized, and implemented on a resource-constrained embedded platform for the real-time analysis of facial emotions in static images. We evaluated the model on five benchmark FER datasets, namely CK+, RaFD, SFEW, FER2013, and RAF. Also, for real-time performance, the trained CNN is optimized using TensorRT SDK and deployed on the Nvidia Jetson TX2 embedded platform. Comparative analysis results with state-of-the-art FER techniques confirm the efficacy of the designed CNN that achieves competitive recognition accuracy and runs in real-time at a frame processing speed of 40 fps on the Jetson TX2 embedded device. Finally, the embedded FER platform is integrated with a low-cost and user-friendly haptic device to render emotions to the VIPs in the form of vibration cues. A working demo of the developed FER system is available at https://youtu.be/c73Ledn27dQ.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. http://connecttech.com/product/orbitty-carrier-for-nvidia-jetson-tx2-tx1/.

References

  1. Li THS, Kuo PH, Tsai TN, Luan PC (2019) Cnn and lstm based facial expression analysis model for a humanoid robot. IEEE Access 7:93998–94011

    Google Scholar 

  2. Cao NT, Ton-That AH, Choi HI (2016) An effective facial expression recognition approach for intelligent game systems. Int J Comput Vis Robot 6(3):223–234

    Google Scholar 

  3. Jeong M, Ko BC (2018) Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12):4270

    Google Scholar 

  4. Lee KW, Yoon HS, Song JM, Park KR (2018) Convolutional neural network-based classification of driver’s emotion during aggressive and smooth driving using multi-modal camera sensors. Sensors 18(4):957

    Google Scholar 

  5. Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Hum Comput 8(4):567–578

    Google Scholar 

  6. Zheng K, Yang D, Liu J, Cui J (2020) Recognition of teachers’ facial expression intensity based on convolutional neural network and attention mechanism. IEEE Access 8:226437–226444

    Google Scholar 

  7. Bahreini K, van der Vegt W, Westera W (2019) A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimed Tools Appl 78(14):18943–18966

    Google Scholar 

  8. Bu Y, Jia J, Tang Y, Zang X, Gao T (2018) Lookine: let the blind hear a smile. In: thirty-second AAAI conference on artificial intelligence

  9. Fei Z, Yang E, Li DDU, Butler S, Ijomah W, Li X, Zhou H (2020) Deep convolution network based emotion analysis towards mental health care. Neurocomputing 388:212–227

    Google Scholar 

  10. Sivasangari A, Ajitha P, Rajkumar I, Poonguzhali S (2019) Emotion recognition system for autism disordered people. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-019-01492-y

    Article  Google Scholar 

  11. Vergura DT, Luceri B (2018) Product packaging and consumers’ emotional response. Does spatial representation influence product evaluation and choice? J Consum Market 35(2):10

    Google Scholar 

  12. Bartkiene E, Steibliene V, Adomaitiene V, Juodeikiene G, Cernauskas D, Lele V, Klupsaite D, Zadeike D, Jarutiene L, Guiné RP (2019) Factors affecting consumer food preferences: food taste and depression-based evoked emotional expressions with the use of face reading technology. BioMed Res Int. https://doi.org/10.1155/2019/2097415

    Article  Google Scholar 

  13. Meshach WT, Hemajothi S, Anita EM (2020) Real-time facial expression recognition for affect identification using multi-dimensional svm. J Ambient Intell Hum Comput 12(6):6355–6365

    Google Scholar 

  14. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Google Scholar 

  15. Saurav S, Singh S, Saini R (2019) Facial expression recognition using histogram of oriented gradients with svm-rfe selected features. In: international conference on hybrid intelligent systems, Springer, pp 339–349

  16. Saurav S, Singh S, Saini R, Yadav M (2020) Facial expression recognition using improved adaptive local ternary pattern. In: proceedings of 3rd international conference on computer vision and image processing, Springer, pp 39–52

  17. Ashir AM, Eleyan A, Akdemir B (2020) Facial expression recognition with dynamic cascaded classifier. Neural Comput Appl 32(10):6295–6309

    Google Scholar 

  18. Caroppo A, Leone A, Siciliano P (2020) Comparison between deep learning models and traditional machine learning approaches for facial expression recognition in ageing adults. J Comput Sci Technol 35(5):1127–1146

    Google Scholar 

  19. Happy S, Dantcheva A, Bremond F (2021) Expression recognition with deep features extracted from holistic and part-based models. Image Vis Comput 105:104038

    Google Scholar 

  20. Valente D, Theurel A, Gentaz E (2018) The role of visual experience in the production of emotional facial expressions by blind people: a review. Psychon Bull Rev 25(2):483–497

    Google Scholar 

  21. Saeed A, Al-Hamadi A, Niese R, Elzobi M (2014) Frame-based facial expression recognition using geometrical features. Adv Hum-Comput Interact. https://doi.org/10.1155/2014/408953

    Article  Google Scholar 

  22. Saurav S, Singh S, Yadav M, Saini R (2020) Image-based facial expression recognition using local neighborhood difference binary pattern. In: proceedings of 3rd international conference on computer vision and image processing, Springer, pp 457–470

  23. Ahmed F, Hossain E (2013) Automated facial expression recognition using gradient-based ternary texture patterns. Chin J Eng. https://doi.org/10.1155/2013/831747

    Article  Google Scholar 

  24. Holder RP, Tapamo JR (2017) Improved gradient local ternary patterns for facial expression recognition. EURASIP J Image Video Process 1:42

    Google Scholar 

  25. Carcagnì P, Del Coco M, Leo M, Distante C (2015) Facial expression recognition and histograms of oriented gradients: a comprehensive study. SpringerPlus 4(1):645

    Google Scholar 

  26. Lekdioui K, Messoussi R, Ruichek Y, Chaabi Y, Touahni R (2017) Facial decomposition for expression recognition using texture/shape descriptors and svm classifier. Signal Process Image Commun 58:300–312

    Google Scholar 

  27. Liu M, Li S, Shan S, Chen X (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing 159:126–136

    Google Scholar 

  28. Zhao X, Shi X, Zhang S (2015) Facial expression recognition via deep learning. IETE Tech Rev 32(5):347–355

    Google Scholar 

  29. Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: proceedings of the 2015 ACM on international conference on multimodal interaction, pp 503–510

  30. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 1–10

  31. Sun B, Li L, Zhou G, He J (2016) Facial expression recognition in the wild based on multimodal texture features. J Electron Imaging 25(6):061407

    Google Scholar 

  32. Majumder A, Behera L, Subramanian VK (2016) Automatic facial expression recognition system using deep network-based data fusion. IEEE Trans Cybern 48(1):103–114

    Google Scholar 

  33. Yang B, Cao J, Ni R, Zhang Y (2017) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Google Scholar 

  34. Sang DV, Van Dat N, et al. (2017) Facial expression recognition using deep convolutional neural networks. In: 2017 9th international conference on knowledge and systems engineering (KSE), IEEE, pp 130–135

  35. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Google Scholar 

  36. Sun N, Li Q, Huan R, Liu J, Han G (2019) Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recognit Lett 119:49–61

    Google Scholar 

  37. Kong F (2019) Facial expression recognition method based on deep convolutional neural network combined with improved lbp features. Pers Ubiquitous Comput 23(3–4):531–539

    Google Scholar 

  38. Aghamaleki JA, Chenarlogh VA (2019) Multi-stream cnn for facial expression recognition in limited training data. Multimed Tools Appl 78(16):22861–22882

    Google Scholar 

  39. Li Y, Zeng J, Shan S, Chen X (2018) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    MathSciNet  Google Scholar 

  40. Sun W, Zhao H, Jin Z (2018) A visual attention based roi detection method for facial expression recognition. Neurocomputing 296:12–22

    Google Scholar 

  41. Nguyen HD, Yeom S, Oh IS, Kim KM, Kim SH (2018) Facial expression recognition using a multi-level convolutional neural network. In: international conference on pattern recognition and artificial intelligence, pp. 217–221

  42. Xie S, Hu H (2018) Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans Multimed 21(1):211–220

    Google Scholar 

  43. Wu BF, Lin CH (2018) Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE Access 6:12451–12461

    Google Scholar 

  44. Chen J, Xu R, Liu L (2018) Deep peak-neutral difference feature for facial expression recognition. Multimed Tools Appl 77(22):29871–29887

    Google Scholar 

  45. Li M, Xu H, Huang X, Song Z, Liu X, Li X (2018) Facial expression recognition with identity and emotion joint learning. IEEE Trans Affect Comput 12(2):544–550

    Google Scholar 

  46. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Google Scholar 

  47. Li K, Jin Y, Akram MW, Han R, Chen J (2020) Facial expression recognition with convolutional neural networks via a new face cropping and rotation strategy. Vis Comput 36(2):391–404

    Google Scholar 

  48. González-Lozoya SM, de la Calleja J, Pellegrin L, Escalante HJ, Medina MA, Benitez-Ruiz A (2020) Recognition of facial expressions based on cnn features. Multimed Tools Appl 79(19):13987–14007

    Google Scholar 

  49. Liu X, Zhou F (2019) Improved curriculum learning using ssm for facial expression recognition. Vis Comput 36(8):1635–1649

    Google Scholar 

  50. Wu M, Su W, Chen L, Liu Z, Cao W, Hirota K (2019) Weight-adapted convolution neural network for facial expression recognition in human-robot interaction. IEEE Trans Syst Man Cybern Syst 51(3):1473–1484

    Google Scholar 

  51. Kim JH, Kim BG, Roy PP, Jeong DM (2019) Efficient facial expression recognition algorithm based on hierarchical deep neural network structure. IEEE Access 7:41273–41285

    Google Scholar 

  52. Xie S, Hu H, Wu Y (2019) Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition. Pattern Recognit 92:177–191

    Google Scholar 

  53. Riaz MN, Shen Y, Sohail M, Guo M (2020) Exnet: an efficient approach for emotion recognition in the wild. Sensors 20(4):1087

    Google Scholar 

  54. Agrawal A, Mittal N (2020) Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis Comput 36(2):405–412

    Google Scholar 

  55. Zhao G, Yang H, Yu M (2020) Expression recognition method based on a lightweight convolutional neural network. IEEE Access 8:38528–38537

    Google Scholar 

  56. Sikkandar H, Thiyagarajan R (2020) Deep learning based facial expression recognition using improved cat swarm optimization. J Ambient Intell Hum Comput 12(2):3037–3053

    Google Scholar 

  57. Gogić I, Manhart M, Pandžić IS, Ahlberg J (2020) Fast facial expression recognition using local binary features and shallow neural networks. Vis Comput 36(1):97–112

    Google Scholar 

  58. Miao S, Xu H, Han Z, Zhu Y (2019) Recognizing facial expressions using a shallow convolutional neural network. IEEE Access 7:78000–78011

    Google Scholar 

  59. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, Berg AC (2016) Ssd: Single shot multibox detector. In: European conference on computer vision, Springer, pp 21–37

  60. Yang S, Luo P, Loy CC, Tang X (2016) Wider face: A face detection benchmark. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 5525–5533

  61. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, et al. (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 7310–7311

  62. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  63. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: proceedings of the IEEE international conference on computer vision, pp 1026–1034

  64. Chollet F, et al. (2018) Keras: The python deep learning library. Astrophysics Source Code Library pp ascl–1806

  65. Carrier PL, Courville A, Goodfellow IJ, Mirza M, Bengio Y (2013) Fer-2013 face database. Universit de Montral

  66. Li S, Deng W (2018) Reliable crowdsourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370

    MathSciNet  MATH  Google Scholar 

  67. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops, IEEE, pp 94–101

  68. Langner O, Dotsch R, Bijlstra G, Wigboldus DH, Hawk ST, Van Knippenberg A (2010) Presentation and validation of the radboud faces database. Cognit Emot 24(8):1377–1388

    Google Scholar 

  69. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In: 2011 IEEE international conference on computer vision workshops (ICCV Workshops), IEEE, pp 2106–2112

  70. Wen G, Chang T, Li H, Jiang L (2020) Dynamic objectives learning for facial expression recognition. IEEE Trans Multimed 22(11):2914–2925

    Google Scholar 

  71. Hasani B, Negi PS, Mahoor M (2020) Breg-next: facial affect computing using adaptive residual networks with bounded gradient. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2986440

    Article  Google Scholar 

  72. Jiang P, Liu G, Wang Q, Wu J (2020a) Accurate and reliable facial expression recognition using advanced softmax loss with fixed weights. IEEE Signal Process Lett 27:725–729

    Google Scholar 

  73. Jiang P, Wan B, Wang Q, Wu J (2020b) Fast and efficient facial expression recognition using a gabor convolutional network. IEEE Signal Process Lett 27:1954–1958

    Google Scholar 

  74. Wang K, Peng X, Yang J, Meng D, Qiao Y (2020a) Region attention networks for pose and occlusion robust facial expression recognition. IEEE Trans Image Process 29:4057–4069

    Google Scholar 

  75. Wang K, Peng X, Yang J, Lu S, Qiao Y (2020b) Suppressing uncertainties for large-scale facial expression recognition. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6897–6906

  76. Li H, Wen G (2019) Sample awareness-based personalized facial expression recognition. Appl Intell 49(8):2956–2969

    Google Scholar 

  77. Wang Z, Zeng F, Liu S, Zeng B (2021) Oaenet: oriented attention ensemble for accurate facial expression recognition. Pattern Recognit 112:107694

    Google Scholar 

  78. Vo TH, Lee GS, Yang HJ, Kim SH (2020) Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8:131988–132001

    Google Scholar 

  79. Li Y, Lu G, Li J, Zhang Z, Zhang D (2020) Facial expression recognition in the wild using multi-level features and attention mechanisms. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.3031602

    Article  Google Scholar 

  80. Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74

    Google Scholar 

  81. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: proceedings of the 2015 ACM on international conference on multimodal interaction, pp 435–442

  82. Dinelli G, Meoni G, Rapuano E, Benelli G, Fanucci L (2019) An fpga-based hardware accelerator for cnns using on-chip memories only: Design and benchmarking with intel movidius neural compute stick. Int J Reconfig Comput. https://doi.org/10.1155/2019/7218758

    Article  Google Scholar 

  83. Choudhary T, Mishra V, Goswami A, Sarangapani J (2020) A comprehensive survey on model compression and acceleration. Artif Intell Rev 57(3):5113–5155

    Google Scholar 

  84. Hinton G, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:150302531

  85. Gordon A, Eban E, Nachum O, Chen B, Wu H, Yang TJ, Choi E (2018) Morphnet: Fast & simple resource-constrained structure learning of deep networks. In: proceedings of the IEEE conference on computer vision and pattern recognition, pp 1586–1595

  86. Migacz S (2017) 8-bit inference with tensorrt. In: GPU technology conference, 4, p 5

Download references

Acknowledgements

The authors would like to thank the director, CSIR-CEERI, Pilani, for supporting and encouraging research activities at CSIR-CEERI, Pilani. Constant motivation by the group head, cognitive computing group, CSIR-CEERI is also acknowledged.

Author information

Authors and Affiliations

  1. Academy of Scientific & Innovative Research, Ghaziabad, Uttar Pradesh, 201002, India

    Sumeet Saurav, Ravi Saini & Sanjay Singh

  2. CSIR-Central Electronics Engineering Research Institute, Pilani, Rajasthan, 333031, India

    Sumeet Saurav, Anil Kumar Saini, Ravi Saini & Sanjay Singh

Authors
  1. Sumeet Saurav
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anil Kumar Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ravi Saini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sanjay Singh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sumeet Saurav.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saurav, S., Saini, A.K., Saini, R. et al. Deep learning inspired intelligent embedded system for haptic rendering of facial emotions to the blind. Neural Comput & Applic 34, 4595–4623 (2022). https://doi.org/10.1007/s00521-021-06613-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06613-3

Keywords

Search

Navigation