Skip to main content
SpringerLink
Log in

Performance Assessment of Face Analysis Algorithms with Occluded Faces

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12662))

Included in the following conference series:

Abstract

In retail environments, it is important to acquire information about customers entering in a selling area, by counting them, but also by understanding stable traits (such as gender, age, or ethnicity) and temporary feelings (such as the emotion). Anyway, in the last year, due to the COVID-19 pandemic, it is becoming mandatory to wear a mask, covering at least half of the face, thus making the above mentioned face analysis tasks definitely more challenging. In this paper, we evaluate the drop in the performance of these analytics when the face is partially covered by a mask, in order to evaluate how existing face analysis applications can perform with occluded faces. According to our knowledge, this is the first time a similar analysis has been performed. Furthermore, we also propose two new datasets, designed as extensions with masked faces of the widely adopted VGG-Face and RAF-DB datasets, that we make publicly available for benchmarking purposes. The analysis we conducted demonstrates that, except for gender and ethnicity recognition whose accuracy drop is quite limited (less than \(10\%\)), further investigations are necessary for increasing the performance of methods for age estimation (MAE drop between 4 and 10 years) and emotion recognition (accuracy decrease between \(45\%\) and \(55\%\)).

This research was partially supported by the Italian MIUR within PRIN 2017 grants, Projects Grant20172BH297 002CUP D44I17000200005 I-MALL, and by A.I. Tech - www.aitech.vision.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://mivia.unisa.it/.

References

  1. Google facenet scores almost 100% recognition. Biometric Technology Today 2015(4), 2–3 (2015). https://doi.org/10.1016/S0969-4765(15)30052-7

  2. Acampora, G., Foggia, P., Saggese, A., Vento, M.: A hierarchical neuro-fuzzy architecture for human behavior analysis. Information Sciences 310, 130 – 148 (2015). https://doi.org/10.1016/j.ins.2015.03.021, http://www.sciencedirect.com/science/article/pii/S0020025515001863

  3. Azzopardi, G., Greco, A., Saggese, A., Vento, M.: Fast gender recognition in videos using a novel descriptor based on the gradient magnitudes of facial landmarks. In: 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 1–6. IEEE (2017)

    Google Scholar 

  4. Azzopardi, G., Greco, A., Saggese, A., Vento, M.: Fusion of domain-specific and trainable features for gender recognition from face images. IEEE Access 6, 24171–24183 (2018)

    Article  Google Scholar 

  5. Bekhouche, S.E., Dornaika, F., Benlamoudi, A., Ouafi, A., Taleb-Ahmed, A.: A comparative study of human facial age estimation: handcrafted features vs. deep features. Multimedia Tools Appl. 79(35), 26605–26622 (2020). https://doi.org/10.1007/s11042-020-09278-7

    Article  Google Scholar 

  6. Brun, L., Saggese, A., Vento, M.: Dynamic scene understanding for behavior analysis based on string kernels. IEEE Trans. Circ. Syst. Video Technol. 24(10), 1669–1681 (2014). https://doi.org/10.1109/TCSVT.2014.2302521

    Article  Google Scholar 

  7. Bu, W., Xiao, J., Zhou, C., Yang, M., Peng, C.: A cascade framework for masked face detection. In: 2017 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), pp. 458–462 (2017)

    Google Scholar 

  8. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: a dataset for recognising faces across pose and age. In: IEEE Conference on FGR, pp. 67–74 (2018)

    Google Scholar 

  9. Carletti, V., Greco, A., Percannella, G., Vento, M.: Age from faces in the deep learning revolution. IEEE Trans. Pattern Anal. Mach. Intell. 42(9), 2113–2132 (2020)

    Article  Google Scholar 

  10. Carletti, V., Greco, A., Saggese, A., Vento, M.: An effective real time gender recognition system for smart cameras. J. Ambient Intell. Humanized Comput. 11(6), 2407–2419 (2019). https://doi.org/10.1007/s12652-019-01267-5

    Article  Google Scholar 

  11. Corneanu, C.A., Simón, M.O., Cohn, J.F., Guerrero, S.E.: Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans. Pattern Anal. Mach. Intell. 38(8), 1548–1568 (2016)

    Article  Google Scholar 

  12. Ejaz, M.S., Islam, M.R.: Masked face recognition using convolutional neural network. In: 2019 International Conference on Sustainable Technologies for Industry 4.0 (STI), pp. 1–6 (2019)

    Google Scholar 

  13. Ejaz, M.S., Islam, M.R., Sifatullah, M., Sarker, A.: Implementation of principal component analysis on masked and non-masked face recognition. In: 2019 1st International Conference on Advances in Science, Engineering and Robotics Technology (ICASERT), pp. 1–5 (2019)

    Google Scholar 

  14. Foggia, P., Greco, A., Percannella, G., Vento, M., Vigilante, V.: A system for gender recognition on mobile robots. In: Proceedings of the 2nd International Conference on Applications of Intelligent Systems, pp. 1–6 (2019)

    Google Scholar 

  15. Fu, S., He, H., Hou, Z.: Learning race from face: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2483–2509 (2014)

    Article  Google Scholar 

  16. Fu, Y., Guo, G., Huang, T.S.: Age synthesis and estimation via faces: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 32(11), 1955–1976 (2010)

    Article  Google Scholar 

  17. Greco, A., Saggese, A., Vento, M.: Digital signage by real-time gender recognition from face images. In: 2020 IEEE International Workshop on Metrology for Industry 4.0 IoT, pp. 309–313 (2020)

    Google Scholar 

  18. Greco, A., Percannella, G., Vento, M., Vigilante, V.: Benchmarking deep network architectures for ethnicity recognition using a new large face dataset. Mach. Vis. Appl. 31(7), 1–13 (2020)

    Google Scholar 

  19. Greco, A., Roberto, A., Saggese, A., Vento, M., Vigilante, V.: Emotion analysis from faces for social robotics. In: 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pp. 358–364. IEEE (2019)

    Google Scholar 

  20. Greco, A., Saggese, A., Vento, M., Vigilante, V.: A convolutional neural network for gender recognition optimizing the accuracy/speed tradeoff. IEEE Access 8, 130771–130781 (2020)

    Article  Google Scholar 

  21. Guo, G., Zhang, N.: A survey on deep learning based face recognition. Comput. Vis. Image Underst. 189, 102805 (2019)

    Article  Google Scholar 

  22. Han, H., Jain, A.K., Shan, S., Chen, X.: Heterogeneous face attribute estimation: a deep multi-task learning approach. IEEE Trans. Pattern Anal. Mach. Intell. 99, 2597–2609 (2017). https://doi.org/10.1109/TPAMI.2017.2738004

    Article  Google Scholar 

  23. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  24. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

    Google Scholar 

  25. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  26. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Workshop on faces in ‘Real-Life’ Images: Detection, Alignment and Recognition (2008)

    Google Scholar 

  27. Kärkkäinen, K., Joo, J.: Fairface: Face attribute dataset for balanced race, gender, and age. arXiv preprint arXiv:1908.04913 (2019)

  28. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

    Google Scholar 

  29. Kollias, D., et al.: Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond. Int. J. Comput. Vis. 127(6–7), 907–929 (2019)

    Article  Google Scholar 

  30. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020)

    Google Scholar 

  31. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2584–2593. IEEE (2017)

    Google Scholar 

  32. Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46448-0_2

    Chapter  Google Scholar 

  33. Ng, C.-B., Tay, Y.-H., Goi, B.-M.: A review of facial gender recognition. Pattern Anal. Appl. 18(4), 739–755 (2015). https://doi.org/10.1007/s10044-015-0499-6

    Article  MathSciNet  Google Scholar 

  34. Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: British Machine Vision Conference (BMVC) (2015)

    Google Scholar 

  35. Rhodes, G., Hickford, C., Jeffery, L.: Sex-typicality and attractiveness: are supermale and superfemale faces super-attractive? Br. J. Psychol. 91(1), 125–140 (2000)

    Article  Google Scholar 

  36. Saggese, A., Vento, M., Vigilante, V.: MIVIABot: a cognitive robot for smart museum. In: Vento, M., Percannella, G. (eds.) CAIP 2019. LNCS, vol. 11678, pp. 15–25. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-29888-3_2

    Chapter  Google Scholar 

  37. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. arXiv (2018)

    Google Scholar 

  38. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  39. Wan, W., Chen, J.: Occlusion robust face recognition based on mask learning. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3795–3799 (2017)

    Google Scholar 

  40. Wang, Z., et al.: Masked face recognition dataset and application. arXiv preprint arXiv:2003.09093 (2020)

  41. Wen, C.Y., Chiu, S.H., Tseng, Y.R., Lu, C.P.: The mask detection technology for occluded face analysis in the surveillance system. J. Forensic Sci. 50, 593–601 (2005). https://doi.org/10.1520/JFS2004409

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Engineering, Electrical Engineering and Applied Mathematics, University of Salerno, Salerno, Italy

    Antonio Greco, Alessia Saggese, Mario Vento & Vincenzo Vigilante

Authors
  1. Antonio Greco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessia Saggese
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mario Vento
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vincenzo Vigilante
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessia Saggese .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Alberto Del Bimbo

  2. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Rita Cucchiara

  3. Department of Computer Science, Boston University, Boston, MA, USA

    Stan Sclaroff

  4. Dipartimento di Matematica e Informatica, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Cloud & AI, JD.COM, Beijing, China

    Tao Mei

  6. Dipartimento di Ingegneria dell’Informazione, Universita di Firenze, Firenze, Italy

    Marco Bertini

  7. Computational Sciences Department, National Institute of Astrophysics, Optics and Electronics (INAOE), Tonantzintla, Puebla, Mexico

    Hugo Jair Escalante

  8. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Roberto Vezzani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Greco, A., Saggese, A., Vento, M., Vigilante, V. (2021). Performance Assessment of Face Analysis Algorithms with Occluded Faces. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12662. Springer, Cham. https://doi.org/10.1007/978-3-030-68790-8_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68790-8_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68789-2

  • Online ISBN: 978-3-030-68790-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation