Skip to main content
SpringerLink
Log in

Masked face recognition using frontal and profile faces with multiple fusion levels

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The main goal of this work is to construct a face recognition system that can overcome the limitation of masked face by exploiting the unmasked parts of frontal and profile faces. This study mainly depends on the upper part of frontal face that contains the eyes and forehead beside of the unmasked parts of profile face such as ear. Many experiments are conducted to measure the efficiency and reliability of face recognition system in case of pandemic where the people have to cover most of their face by mask which makes the process of face recognition harder. Binarized statistical image features approaches is used as texture-based descriptors for feature extraction process beside of convolutional neural networks. Additionally, four fusion approaches are implemented, namely sensor-level, feature-level, score-level and decision-level fusion. Images of frontal and profile face under different conditions from ND-TWINS-2009-2010 and CFPW datasets are used in the experiments. The experimental results show that the proposed method is more reliable and accurate in the case of masked face than using profile or frontal face images as a standalone system. The maximum performance of the proposed method for recognizing masked face as recognition rate is 99.83%. Moreover, the proposed system is better than the state-of-the-art CNN-based and texture-based systems using both frontal and profile faces with and without mask on the faces.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Toygar, Ö., Alqaralleh, E., Afaneh, A.: Person identification using multimodal biometrics under different challenges. Hum.-Robot Interact. Theory Appl. 66, 81–96 (2018)

    Google Scholar 

  2. Jain, A.K., Ross, A., Prabhakar, S.: An introduction to biometric recognition. IEEE Trans. Circuits Syst. Video Technol. 14(1), 4–20 (2004)

    Article  Google Scholar 

  3. Annapurani, K., Sadiq, M., Malathy, C.: Fusion of shape of the ear and tragus-a unique feature extraction method for ear authentication system. Expert Syst. Appl. 42(1), 649–656 (2015)

    Article  Google Scholar 

  4. Alqaralleh, E., Toygar, Ö.: Ear recognition based on fusion of ear and tragus under different challenges. Int. J. Pattern Recognit. Artif. Intell. 32(09), 1856009 (2018)

    Article  MathSciNet  Google Scholar 

  5. Akbari, A., Awais, M., Bashar, M., Kittler, J.: How does loss function affect generalization performance of deep learning? Application to human age estimation. In: International Conference on Machine Learning (PMLR), pp. 141–151 (2021)

  6. Akbari, A., Awais, M., Bashar, M., Kittler, J.: A theoretical insight into the effect of loss function for deep semantic-preserving learning. IEEE Trans. Neural Netw. Learn. Syst. 6, 66 (2021)

    Google Scholar 

  7. Wang, Z., Wang, G., Huang, B., Xiong, Z., Hong, Q., Wu, H., Yi, P., Jiang, K., Wang, N., Pei, Y., et al.: Masked face recognition dataset and application, arXiv preprint arXiv:2003.09093 (2020)

  8. Mundial, I.Q., Hassan, M.S.U., Tiwana, M.I., Qureshi, W.S., Alanazi, E.: Towards facial recognition problem in Covid-19 pandemic. In: 2020 4rd International Conference on Electrical, Telecommunication and Computer Engineering (ELTICOM), pp. 210–214. IEEE (2020)

  9. Damer, N., Grebe, J.H., Chen, C., Boutros, F., Kirchbuchner, F., Kuijper, A.: The effect of wearing a mask on face recognition performance: an exploratory study. In: 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), pp. 1–6. IEEE (2020)

  10. Loey, M., Manogaran, G., Taha, M.H.N., Khalifa, N.E.M.: A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the covid-19 pandemic. Measurement 167, 108288 (2020)

    Article  Google Scholar 

  11. He, E.J., Fernandez, J.A., Kumar, B.V., Alkanhal, M.: Masked correlation filters for partially occluded face recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1293–1297. IEEE (2016)

  12. Alyuz, N., Gokberk, B., Akarun, L.: 3-d face recognition under occlusion using masked projection. IEEE Trans. Inf. Forens. Secur. 8(5), 789–802 (2013)

    Article  Google Scholar 

  13. Sengupta, S., Chen, J.-C., Castillo, C., Patel, V.M., Chellappa, R., Jacobs, D.W.: Frontal to profile face verification in the wild. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9. IEEE (2016)

  14. Afaneh, A., Noroozi, F., Toygar, Ö.: Recognition of identical twins using fusion of various facial feature extractors. EURASIP J. Image Video Process. 2017(81), 1–14 (2017)

    Google Scholar 

  15. Moon, H.-M., Seo, C.H., Pan, S.B.: A face recognition system based on convolution neural network using multiple distance face. Soft Comput. 21(17), 4995–5002 (2017)

    Article  Google Scholar 

  16. Toygar, Ö., Alqaralleh, E., Afaneh, A.: Symmetric ear and profile face fusion for identical twins and non-twins recognition. Signal Image Video Process. 12(6), 1157–1164 (2018)

    Article  Google Scholar 

  17. Rathore, R., Prakash, S., Gupta, P.: Efficient human recognition system using ear and profile face. In: 2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS), pp. 1–6. IEEE (2013)

  18. Kannala, J., Rahtu, E.: Bsif: binarized statistical image features. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp. 1363–1366. IEEE (2012)

  19. Yuan, Z.-W., Zhang, J.: Feature extraction and image retrieval based on alexnet. In: Eighth International Conference on Digital Image Processing (ICDIP 2016), vol. 10033, p. 100330E. International Society for Optics and Photonics (2016)

  20. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

    Article  Google Scholar 

  21. Ojansivu, V., Heikkilä, J.: Blur insensitive texture classification using local phase quantization. In: International Conference on Image and Signal Processing, pp. 236–243. Springer (2008)

  22. Druzhkov, P., Kustikova, V.: A survey of deep learning methods and software tools for image classification and object detection. Pattern Recognit. Image Anal. 26(1), 9–15 (2016)

    Article  Google Scholar 

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

  24. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

  25. Soltanpour, S., Wu, Q.J.: Multimodal 2d–3d face recognition using local descriptors: pyramidal shape map and structural context. IET Biometr. 6(1), 27–35 (2016)

    Article  Google Scholar 

  26. Dataset: cvrl data sets [online]. https://sites.google.com/a/nd.edu/public-cvrl/data-sets. Accessed 15 Nov 2016 (2013)

  27. Phillips, P.J., Flynn, P.J., Bowyer, K.W., Bruegge, R.W.V., Grother, P.J., Quinn, G.W., Pruitt, M.: Distinguishing identical twins by face recognition. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 185–192. IEEE (2011)

  28. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001 (CVPR 2001), vol. 1, p. I. IEEE (2001)

  29. Akbari, A., Awais, M., Feng, Z., Farooq, A., Kittler, J.: Distribution cognisant loss for cross-database facial age estimation with sensitivity analysis. IEEE Trans. Pattern Anal. Mach. Intell. 6, 66 (2020)

    Google Scholar 

  30. Toygar, Ö., Alqaralleh, E., Afaneh, A.: On the use of ear and profile faces for distinguishing identical twins and nontwins. Expert Syst. 37(1), e12389 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Department, Khaybar Community College, Taibah University, Almadinah Almunawarah, Saudi Arabia

    Esraa Alqaralleh

  2. Computer Engineering Department, Faculty of Engineering, Eastern Mediterranean University, Via Mersin 10, Famagusta, North Cyprus

    Ayman Afaneh & Önsen Toygar

Authors
  1. Esraa Alqaralleh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayman Afaneh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Önsen Toygar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Önsen Toygar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alqaralleh, E., Afaneh, A. & Toygar, Ö. Masked face recognition using frontal and profile faces with multiple fusion levels. SIViP 17, 1375–1382 (2023). https://doi.org/10.1007/s11760-022-02345-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02345-6

Keywords

Search

Navigation