Skip to main content
SpringerLink
Log in

A CNN-based multi-level face alignment approach for mitigating demographic bias in clinical populations

  • Original Paper
  • Published:
Computational Statistics Aims and scope Submit manuscript

Abstract

The investigation of demographic bias in facial analysis applications is a topic of growing interest with achievements in face recognition and gender classification. State-of-the-art convolutional neural networks (CNN) and traditional machine learning models for locating facial landmarks have reached overall performance levels close to human annotation. However, recent studies demonstrated that these models presented performance gaps when applied to different populations, characterizing bias led by demographic features. Nevertheless, few studies have addressed this problem in face alignment and facial landmarks localization methods. In this work, we propose a multi-level face alignment approach settled on CNN models to reduce performance gaps among different populations. We created facial subunit CNN models tied to a facial subunit detector at a higher level. The proposal seeks to improve bad results caused by facial impairment, guided by the following assumptions: facial unit landmarks localization does not require global texture, and combining different facial unit models can improve the final model’s variability. We applied the models in a balanced dataset mixing healthy controls and individuals with neurological disorders: amyotrophic lateral sclerosis and post-stroke. With fewer samples for training, our approach significantly reduced face alignment performance differences among those groups as compared to a state-of-the-art CNN model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Bandini A, Rezaei S, Guarín DL et al. (2021) A new dataset for facial motion analysis in individuals with neurological disorders. IEEE J Biomed Health Inform 25(4):1111–1119. https://doi.org/10.1109/JBHI.2020.3019242

    Article  Google Scholar 

  • Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: 2017 IEEE international conference on computer vision (ICCV), pp 1021–1030, https://doi.org/10.1109/ICCV.2017.116

  • Buolamwini J, Gebru T (2018) Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Friedler SA, Wilson C (eds) Proceedings of the 1st conference on fairness, accountability and transparency, proceedings of machine learning research, vol 81 PMLR, New York, USA, pp 77–91, http://proceedings.mlr.press/v81/buolamwini18a.html

  • Cao D, Chen C, Piccirilli M, et al. (2011) Can facial metrology predict gender? In: 2011 International Joint Conference on Biometrics (IJCB), pp 1–8, https://doi.org/10.1109/IJCB.2011.6117471

  • Citron D, Pasquale F (2014) The scored society: due process for automated predictions. Washington Law Rev 89:1–33

    Google Scholar 

  • Cootes T, Taylor C, Cooper D et al. (1995) Active shape models-their training and application. Comput Vis Image Underst 61(1):38–59. https://doi.org/10.1006/cviu.1995.1004

    Article  Google Scholar 

  • Drozdowski P, Rathgeb C, Dantcheva A et al. (2020) Demographic bias in biometrics: a survey on an emerging challenge. IEEE Trans Technol Soc 1(2):89–103. https://doi.org/10.1109/TTS.2020.2992344

    Article  Google Scholar 

  • Du C, Wu Q, Yang J, et al. (2008) Svm based asm for facial landmarks location. In: 8th IEEE international conference on computer and information technology (CIT ‘08), pp 321–326, https://doi.org/10.1109/CIT.2008.4594695

  • Dunn OJ (1961) Multiple comparisons among means. J Am Stat Assoc 56(293):52–64

    Article  MathSciNet  MATH  Google Scholar 

  • Garcia RV, Wandzik L, Grabner L, et al. (2019) The harms of demographic bias in deep face recognition research. In: 2019 international conference on biometrics (ICB), pp 1–6, https://doi.org/10.1109/ICB45273.2019.8987334

  • Georgopoulos M, Panagakis Y, Pantic M (2020) Investigating bias in deep face analysis: the kanface dataset and empirical study. Image Vis Comput 102(103):954

    Google Scholar 

  • Georgopoulos M, Oldfield J, Nicolaou MA et al. (2021) Mitigating demographic bias in facial datasets with style-based multi-attribute transfer. Int J Comput Vision 129(7):2288–2307

    Article  Google Scholar 

  • Gross R, Matthews I, Cohn J et al. (2010) Multi-pie. Image Vision Comput 28(5):807–813. https://doi.org/10.1016/j.imavis.2009.08.002

    Article  Google Scholar 

  • Hsu CF, Lin CC, Hung TY, et al. (2020) A detailed look at cnn-based approaches in facial landmark detection. arXiv:2005.08649

  • Hsu RL, Abdel-Mottaleb M, Jain AK (2002) Face detection in color images. IEEE Trans Pattern Anal Mach Intell 24(5):696–706. https://doi.org/10.1109/34.1000242

    Article  Google Scholar 

  • Hupont I, Fernández C (2019) Demogpairs: Quantifying the impact of demographic imbalance in deep face recognition. In: 2019 14th IEEE international conference on automatic face gesture recognition (FG 2019), pp 1–7, https://doi.org/10.1109/FG.2019.8756625

  • Johnston B, Chazal P (2018) A review of image-based automatic facial landmark identification techniques. EURASIP J Image Video Process 2018:86. https://doi.org/10.1186/s13640-018-0324-4

    Article  Google Scholar 

  • Karras T, Laine S, Aittala M, et al. (2020) Analyzing and improving the image quality of stylegan. In: 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 8107–8116, https://doi.org/10.1109/CVPR42600.2020.00813

  • Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621. https://doi.org/10.1080/01621459.1952.10483441

    Article  MATH  Google Scholar 

  • Le V, Brandt J, Lin Z et al. (2012) Interactive facial feature localization. In: Fitzgibbon A, Lazebnik S, Perona P et al. (eds) Computer vision - ECCV 2012. Springer, Berlin Heidelberg, Berlin, Heidelberg, pp 679–692

    Chapter  Google Scholar 

  • Milborrow S (2007) Locating facial features with active shape models. Master’s thesis, Faculty of Engineering, University of Cape Town, Cape Town, South Africa

  • Milborrow S (2016) Multiview Active Shape Models with SIFT Descriptors. Doctoral thesis, Department of Image Processing, University of Cape Town, Cape Town, South Africa

  • Milborrow S, Nicolls F (2014) Active shape models with sift descriptors and mars. In: 2014 international conference on computer vision theory and applications (VISAPP), pp 380–387

  • Moghaddam B, Yang MH (2000) Gender classification with support vector machines. In: proceedings fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580), pp 306–311, https://doi.org/10.1109/AFGR.2000.840651

  • Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445. https://doi.org/10.1109/34.895976

    Article  Google Scholar 

  • Sagonas C, Antonakos E, Tzimiropoulos G et al. (2016) 300 faces in-the-wild challenge: database and results. Image Vis Comput 47:3–18. https://doi.org/10.1016/j.imavis.2016.01.002

    Article  Google Scholar 

  • Seshadri K, Savvides M (2009) Robust modified active shape model for automatic facial landmark annotation of frontal faces. In: IEEE 3rd international conference on biometrics: theory, applications, and systems (BTAS ‘09), pp 1–8, https://doi.org/10.1109/BTAS.2009.5339057

  • Sun Y, Wang X, Tang X (2013) Deep convolutional network cascade for facial point detection. In: 2013 IEEE conference on computer vision and pattern recognition, pp 3476–3483, https://doi.org/10.1109/CVPR.2013.446

  • Taati B, Zhao S, Ashraf AB, et al. (2019) Algorithmic bias in clinical populations-evaluating and improving facial analysis technology in older adults with dementia. IEEE Access 7:25,527–25,534. https://doi.org/10.1109/ACCESS.2019.2900022

  • Turk M, Pentland A (1991) Face recognition using eigenfaces. In: IEEE conference on computer vision and pattern recognition (CVPR ‘91). IEEE, pp 586–591

  • Wang C (2019) The development and challenges of face alignment algorithms. J Phys Conf Ser 1335(1):12. https://doi.org/10.1088/1742-6596/1335/1/012009

    Article  MathSciNet  Google Scholar 

  • Wu W, Protopapas P, Yang Z, et al. (2020) Gender classification and bias mitigation in facial images. In: 12th ACM conference on web science. Association for computing machinery, New York, USA, WebSci ’20, pp 106–114

  • Xu Z, Li B, Geng M et al. (2021) Anchorface: An anchor-based facial landmark detector across large poses. Proceedings of the AAAI conference on artificial intelligence 35:3092–3100. https://doi.org/10.1609/aaai.v35i4.16418

  • Yang J, Liu Q, Zhang K (2017) Stacked hourglass network for robust facial landmark localisation. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 2025–2033, https://doi.org/10.1109/CVPRW.2017.253

Download references

Author information

Author notes

  1. Kelson R. T. Aires, Anselmo C. de Paiva, Rodrigo de M. S. Veras, and Pedro L. M. Soares have contributed equally to this work.

Authors and Affiliations

  1. Departamento de Computação, Universidade Federal do Piauí, Ininga, Teresina, PI, Brasil

    Ricardo T. Freitas, Kelson R. T. Aires, Rodrigo de M. S. Veras & Pedro L. M. Soares

  2. Departamento de Informática, Universidade Federal do Maranhão, Bacanga, São Luís, MA, Brasil

    Anselmo C. de Paiva

Authors
  1. Ricardo T. Freitas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kelson R. T. Aires
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anselmo C. de Paiva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rodrigo de M. S. Veras
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pedro L. M. Soares
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ricardo T. Freitas.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Freitas, R.T., Aires, K.R.T., de Paiva, A.C. et al. A CNN-based multi-level face alignment approach for mitigating demographic bias in clinical populations. Comput Stat (2023). https://doi.org/10.1007/s00180-023-01395-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00180-023-01395-9

Keywords

Search

Navigation