Skip to main content
SpringerLink
Log in

A vision system to assist visually challenged people for face recognition using multi-task cascaded convolutional neural network (MTCNN) and local binary pattern (LBP)

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Visually impaired people are socially disconnected in situations like face-to-face communication and recognize known individuals. Engaging freely with their sighted counterparts is still challenging and adequate attention is not given to non-verbal communication. This work proposed a compact wearable solution to recognize faces to aid the visually impaired in better social interaction. To address this, we develop a portable embedded device with face recognition capabilities, which facilitates a visually impaired person to recognize faces through the audio feedback system. In preprocessing a hybrid method is proposed for enhancing the visual quality of the face. This is based on LAB color space and Contrast Limited Adaptive Histogram Equalization (CLAHE) with gamma enhancement, accurately recognizing the faces irrespective of various illumination conditions. The efficiency of the proposed methodology is evaluated in a real-time scenario with the following parameters: Process CPU usage, process memory usage, Frame per Second (FPS), Model load analysis, and average CPU load analysis. Experimental results show The MTCNN based LPB uses optimal CPU utilization and improve the accuracy of real-time face recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig.4
Fig.5
Fig.6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig.13
Fig.14
Fig.15
Fig.16
Fig. 17
Fig. 18
Fig.19

Similar content being viewed by others

References

  • Aakash Krishna GS, Pon VN, Rai S, Baskar A (2020) Vision system with 3D audio feedback to assist navigation for visually impaired. Proc Comput Sci 167:235–243

    Article  Google Scholar 

  • Asmare MH, Asirvadam VS, Iznita L (2009) Color space selection for color image enhancement applications. International conference on signal acquisition and processing. IEEE, pp 208–212

    Google Scholar 

  • Baskar A, Gireesh Kumar T (2018) Facial expression classification using machine learning approach: a review. Data Eng Intell Comput 542:337–345

    Article  Google Scholar 

  • Bhattacharya J, Marsi S, Carrato S, Frey H, Ramponi G (2017) Feeding a DNN for face verification in video data acquired by a visually impaired user. 40th international convention on information and communication technology. Electronics and Microelectronics (MIPRO), pp 1084–1089

    Google Scholar 

  • Bourne RR, Flaxman SR, Braithwaite T, Cicinelli MV, Das A, Jonas JB, Keeffe J, Kempen JH, Leasher J, Limburg H, Naidoo K (2017) Vision loss expert group magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Glob Health-Elsevier 5(9):e888–e897. https://doi.org/10.1016/S2214-109X(17)30293-0

    Article  Google Scholar 

  • Bourne R, Steinmetz JD, Flaxman S, Briant PS, Taylor HR, Resnikoff S, Casson RJ, Abdoli A, Abu-Gharbieh E, Afshin A, Ahmadieh H (2021) Trends in prevalence of blindness and distance and near vision impairment over 30 years: an analysis for the global burden of disease study. Lancet Glob Health 9(2):e130–e143. https://doi.org/10.1016/S2214-109X(20)30425-3

    Article  Google Scholar 

  • Chang X, Nie F, Wang S, Yang Y, Zhou X, Zhang C (2015) Compound rank-k projections for bilinear analysis. IEEE Trans Neural Netw Learn Syst 27(7):1502–1513

    Article  MathSciNet  Google Scholar 

  • Chen K, Yao L, Zhang D, Wang X, Chang X, Nie F (2020) A semisupervised recurrent convolutional attention model for human activity recognition. IEEE Trans Neural Netw Learn Syst 31(5):1747–1756

    Article  Google Scholar 

  • Ding C, Tao D (2016) A comprehensive survey on pose-invariant face recognition. ACM Trans Intell Syst Technol 7(3):1–40. https://doi.org/10.1145/2845089

    Article  Google Scholar 

  • Han H, Shan S, Chen X, Gao W (2013) A comparative study on illumination preprocessing in face recognition. Pattern Recogn 46(6):1691–1699. https://doi.org/10.1016/j.patcog.2012.11.022

    Article  Google Scholar 

  • Li Z, Nie F, Chang X, Nie L, Zhang H, Yang Y (2018a) Rank-constrained spectral clustering with flexible embedding. IEEE Trans Neural Netw Learn Syst 29(12):6073–6082

    Article  MathSciNet  Google Scholar 

  • Li Z, Nie F, Chang X, Yang Y, Zhang C, Sebe N (2018b) Dynamic affinity graph construction for spectral clustering using multiple features. IEEE Trans Neural Netw Learn Syst 29(12):6323–6332

    Article  MathSciNet  Google Scholar 

  • Li Z, Yao L, Chang X, Zhan K, Sun J, Zhang H (2019) Zero-shot event detection via event-adaptive concept relevance mining. Pattern Recogn 88:595–603

    Article  Google Scholar 

  • Luo M, Chang X, Nie L, Yang Y, Hauptmann AG, Zheng Q (2018) An adaptive semi supervised feature analysis for video semantic recognition. IEEE Trans Cybern 48(2):648–660

    Article  Google Scholar 

  • Neto LB, Grijalva F, Maike VR, Martini LC, Florencio D, Baranauskas MC, Rocha A, Goldenstein S (2016) A kinect-based wearable face recognition system to aid visually impaired users. IEEE Trans Human-Mach Syst 47(1):52–64

    Google Scholar 

  • Rabia J, Ali SA, Arabnia HR (2013) Face recognition for the visually impaired. In: Proceedings of the international conference on information and knowledge engineering (IKE). The steering committee of the world congress in computer science, Computer engineering and applied computing (WorldComp). IEEE, pp 1–7

    Google Scholar 

  • Rahim MA, Azam MS, Hossain N, Islam MR (2013) Face recognition using local binary patterns (LBP). Glob J Comp Sci Technol 13(4):1–8

    Google Scholar 

  • Sanath K, Meenakshi K, Rajan M, Balamurugan V, Harikumar ME (2021) RFID and face recognition based smart attendance system. 5th international conference on computing methodologies and communication (ICCMC). ICCMC, pp 492–499

    Google Scholar 

  • Schroff F, Kalenichenko D, Philbin J (2015) Facenet: a unified embedding for face recognition and clustering. Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 815–823

    Google Scholar 

  • Sun X, Wu P, Hoi SC (2018) Face detection using deep learning: an improved faster RCNN approach. Neurocomputing 299:42–50

    Article  Google Scholar 

  • Tapu R, Mocanu B, Zaharia T (2020) Wearable assistive devices for visually impaired: a state of the art survey. Pattern Recogn Lett 137:37–52

    Article  Google Scholar 

  • Vamsi M, Soman KP, Guruvayurappan K (2020) Automatic seat adjustment using face recognition. International conference on inventive computation technologies (ICICT). ICICT, pp 449–453

    Google Scholar 

  • Yan C, Chang X, Luo M, Zheng Q, Zhang X, Li Z, Nie F (2020) Self-weighted robust LDA for multiclass classification with edge classes. ACM Trans Intell Syst Technol (TIST) 12(1):1–19. https://doi.org/10.1145/3418284

    Article  Google Scholar 

  • Yang M-H, Kriegman DJ, Ahuja N (2002) Detecting faces in images: a survey. IEEE Trans Pattern Anal Mach Intell 24(1):34–58

    Article  Google Scholar 

  • Yu E, Ma J, Sun J, Chang X, Zhang H, Hauptmann AG (2022) Deep discrete cross-modal hashing with multiple supervision. Neurocomputing 486:215–224

    Article  Google Scholar 

  • Yuan D, Chang X, Li Z, He Z (2022) Learning adaptive spatial-temporal context-aware correlation filters for UAV tracking. ACM Trans Multimedia Comput Commun Appl (TOMM) 18(3):1–18. https://doi.org/10.1145/3486678

    Article  Google Scholar 

  • Zhang K, Zhang Z, Li Z, Qiao Y (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  • Zhang D, Yao L, Chen K, Wang S, Chang X, Liu Y (2020) Making sense of spatio-temporal preserving representations for EEG-based human intention recognition. IEEE Trans Cybern 50(7):3033–3044

    Article  Google Scholar 

  • Zhou R, Chang X, Shi L, Shen YD, Yang Y, Nie F (2019) Person reidentification via multi-feature fusion with adaptive graph learning. IEEE Trans Neural Netw Learn Syst 31(5):1592–1601

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Amrita School of Computing, Amrita Vishwa Vidyapeetham, Coimbatore, India

    A. Baskar & T. Gireesh Kumar

  2. Geosystems Research Institute, Mississippi State University, Starkville, MS, 39762, USA

    Sathishkumar Samiappan

Authors
  1. A. Baskar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. Gireesh Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sathishkumar Samiappan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. Baskar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baskar, A., Kumar, T.G. & Samiappan, S. A vision system to assist visually challenged people for face recognition using multi-task cascaded convolutional neural network (MTCNN) and local binary pattern (LBP). J Ambient Intell Human Comput 14, 4329–4341 (2023). https://doi.org/10.1007/s12652-023-04542-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-023-04542-8

Keywords

Search

Navigation