Skip to main content
View expanded cover

European Conference on Computer Vision

ECCV 2020: Computer Vision – ECCV 2020 pp 330–347Cite as

Jointly De-Biasing Face Recognition and Demographic Attribute Estimation

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12374)

Abstract

We address the problem of bias in automated face recognition and demographic attribute estimation algorithms, where errors are lower on certain cohorts belonging to specific demographic groups. We present a novel de-biasing adversarial network (DebFace) that learns to extract disentangled feature representations for both unbiased face recognition and demographics estimation. The proposed network consists of one identity classifier and three demographic classifiers (for gender, age, and race) that are trained to distinguish identity and demographic attributes, respectively. Adversarial learning is adopted to minimize correlation among feature factors so as to abate bias influence from other factors. We also design a new scheme to combine demographics with identity features to strengthen robustness of face representation in different demographic groups. The experimental results show that our approach is able to reduce bias in face recognition as well as demographics estimation while achieving state-of-the-art performance.

Keywords

  • Bias
  • Feature disentanglement
  • Face recognition
  • Fairness

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-58526-6_20
  • Chapter length: 18 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-58526-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   149.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Notes

  1. 1.

    This is different from the notion of machine learning bias, defined as “any basis for choosing one generalization [hypothesis] over another, other than strict consistency with the observed training instances”  [15].

  2. 2.

    In our case, \(K_G=2\), i.e., male and female.

  3. 3.

    To clarify, we consider two race groups, Black and White; and two ethnicity groups, East Asian and Indian. The word race denotes both race and ethnicity in this paper.

References

  1. https://yanweifu.github.io/FG_NET_data

  2. http://trillionpairs.deepglint.com/overview

  3. Abadi, M., et al.: Deep learning with differential privacy. In: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security, pp. 308–318 (2016)

    Google Scholar 

  4. Amini, A., Soleimany, A., Schwarting, W., Bhatia, S., Rus, D.: Uncovering and mitigating algorithmic bias through learned latent structure. In: AAAI/ACM Conference on AI, Ethics, and Society (2019)

    Google Scholar 

  5. Bolukbasi, T., Chang, K.W., Zou, J.Y., Saligrama, V., Kalai, A.T.: Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In: Advances in Neural Information Processing Systems, pp. 4349–4357 (2016)

    Google Scholar 

  6. Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribution-aware margin loss. arXiv preprint arXiv:1906.07413 (2019)

  7. Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A.: Vggface2: a dataset for recognising faces across pose and age. In: IEEE International Conference on Automatic Face & Gesture Recognition. IEEE (2018)

    Google Scholar 

  8. Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artif. Intell. Res. 16, 321–357 (2002)

    CrossRef  Google Scholar 

  9. Chen, B.C., Chen, C.S., Hsu, W.H.: Cross-age reference coding for age-invariant face recognition and retrieval. In: ECCV (2014)

    Google Scholar 

  10. Cheng, J., Li, Y., Wang, J., Yu, L., Wang, S.: Exploiting effective facial patches for robust gender recognition. Tsinghua Sci. Technol. 24(3), 333–345 (2019)

    CrossRef  Google Scholar 

  11. Cook, C.M., Howard, J.J., Sirotin, Y.B., Tipton, J.L., Vemury, A.R.: Demographic effects in facial recognition and their dependence on image acquisition: an evaluation of eleven commercial systems. IEEE Trans. Biometrics Behav. Identity Sci. 1, 32–41 (2019)

    CrossRef  Google Scholar 

  12. Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: CVPR (2019)

    Google Scholar 

  13. Deb, D., Best-Rowden, L., Jain, A.K.: Face recognition performance under aging. In: CVPRW (2017)

    Google Scholar 

  14. Deng, J., Guo, J., Xue, N., Zafeiriou, S.: ArcFace: additive angular margin loss for deep face recognition. In: CVPR (2019)

    Google Scholar 

  15. Dietterich, T.G., Kong, E.B.: Machine learning bias, statistical bias, and statistical variance of decision tree algorithms. Tech. rep. (1995)

    Google Scholar 

  16. Drummond, C., Holte, R.C., et al.: C4. 5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling. In: Workshop on Learning from Imbalanced Datasets II. Citeseer (2003)

    Google Scholar 

  17. Fredrikson, M., Jha, S., Ristenpart, T.: Model inversion attacks that exploit confidence information and basic countermeasures. In: The 22nd ACM SIGSAC (2015)

    Google Scholar 

  18. Goodfellow, I., et al.: Generative adversarial nets. In: NIPS (2014)

    Google Scholar 

  19. Guo, Y., Zhang, L., Hu, Y., He, X., Gao, J.: MS-Celeb-1M: a dataset and benchmark for large-scale face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 87–102. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_6

    CrossRef  Google Scholar 

  20. Han, H., A, K.J., Shan, S., Chen, X.: Heterogeneous face attribute estimation: a deep multi-task learning approach. IEEE Trans. Pattern Anal. Mach. Intelli. PP(99), 1–1 (2017)

    Google Scholar 

  21. Hayat, M., Khan, S., Zamir, W., Shen, J., Shao, L.: Max-margin class imbalanced learning with Gaussian affinity. arXiv preprint arXiv:1901.07711 (2019)

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)

    Google Scholar 

  23. Howard, J., Sirotin, Y., Vemury, A.: The effect of broad and specific demographic homogeneity on the imposter distributions and false match rates in face recognition algorithm performance. In: IEEE BTAS (2019)

    Google Scholar 

  24. Huang, C., Li, Y., Chen, C.L., Tang, X.: Deep imbalanced learning for face recognition and attribute prediction. IEEE Trans Pattern Anal. Mach. Intell. (2019)

    Google Scholar 

  25. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments (2008)

    Google Scholar 

  26. Jourabloo, A., Yin, X., Liu, X.: Attribute preserved face de-identification. In: ICB (2015)

    Google Scholar 

  27. Khan, S., Hayat, M., Zamir, S.W., Shen, J., Shao, L.: Striking the right balance with uncertainty. In: CVPR (2019)

    Google Scholar 

  28. Kim, H., Mnih, A.: Disentangling by factorising. arXiv preprint arXiv:1802.05983 (2018)

  29. Klare, B.F., Burge, M.J., Klontz, J.C., Bruegge, R.W.V., Jain, A.K.: Face recognition performance: Role of demographic information. IEEE Trans. Inform. Forensics Secur. 7(6), 1789–1801 (2012)

    CrossRef  Google Scholar 

  30. Klare, B.F., et al.: Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In: CVPR (2015)

    Google Scholar 

  31. Liu, F., Zeng, D., Zhao, Q., Liu, X.: Disentangling features in 3D face shapes for joint face reconstruction and recognition. In: CVPR (2018)

    Google Scholar 

  32. Liu, Y., Wang, Z., Jin, H., Wassell, I.: Multi-task adversarial network for disentangled feature learning. In: CVPR (2018)

    Google Scholar 

  33. Liu, Y.H., Chen, Y.T.: Face recognition using total margin-based adaptive fuzzy support vector machines. IEEE Trans. Neural Networks 18(1), 178–192 (2007)

    CrossRef  Google Scholar 

  34. Liu, Y., Wei, F., Shao, J., Sheng, L., Yan, J., Wang, X.: Exploring disentangled feature representation beyond face identification. In: CVPR (2018)

    Google Scholar 

  35. Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811.12359 (2018)

  36. Long, M., Cao, Z., Wang, J., Jordan, M.I.: Conditional adversarial domain adaptation. In: NIPS (2018)

    Google Scholar 

  37. Maze, B., et al.: Iarpa janus benchmark-c: face dataset and protocol. In: 2018 ICB (2018)

    Google Scholar 

  38. Moschoglou, S., Papaioannou, A., Sagonas, C., Deng, J., Kotsia, I., Zafeiriou, S.: AgeDB: the first manually collected, in-the-wild age database. In: CVPRW (2017)

    Google Scholar 

  39. Mullick, S.S., Datta, S., Das, S.: Generative adversarial minority oversampling. arXiv preprint arXiv:1903.09730 (2019)

  40. Narayanaswamy, S., et al.: Learning disentangled representations with semi-supervised deep generative models. In: NIPS (2017)

    Google Scholar 

  41. Niu, Z., Zhou, M., Wang, L., Gao, X., Hua, G.: Ordinal regression with multiple output CNN for age estimation. In: CVPR (2016)

    Google Scholar 

  42. Patrick Grother, M.N., Hanaoka, K.: Face recognition vendor test (FRVT) part 3: demographic effects. Tech. rep., National Institute of Standards and Technology (2019)

    Google Scholar 

  43. Rothe, R., Timofte, R., Van Gool, L.: Deep expectation of real and apparent age from a single image without facial landmarks. IJCV 126(2–4), 144–157 (2018)

    MathSciNet  CrossRef  Google Scholar 

  44. Schmidhuber, J.: Learning factorial codes by predictability minimization. Neural Comput. 4(6), 863–879 (1992)

    CrossRef  Google Scholar 

  45. Setty, S., et al.: Indian movie face database: a benchmark for face recognition under wide variations. In: NCVPRIPG (2013)

    Google Scholar 

  46. Shi, Y., Jain, A.K., Kalka, N.D.: Probabilistic face embeddings. arXiv preprint arXiv:1904.09658 (2019)

  47. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: Closing the gap to human-level performance in face verification. In: CVPR (2014)

    Google Scholar 

  48. Tao, C., Lv, F., Duan, L., Wu, M.: Minimax entropy network: learning category-invariant features for domain adaptation. arXiv preprint arXiv:1904.09601 (2019)

  49. Torralba, A., Efros, A.A., et al.: Unbiased look at dataset bias. In: CVPR (2011)

    Google Scholar 

  50. Tran, L., Yin, X., Liu, X.: Disentangled representation learning GAN for pose-invariant face recognition. In: CVPR (2017)

    Google Scholar 

  51. Tran, L., Yin, X., Liu, X.: Representation learning by rotating your faces. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 3007–3021 (2019)

    CrossRef  Google Scholar 

  52. Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: CVPR (2015)

    Google Scholar 

  53. Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. In: CVPR (2017)

    Google Scholar 

  54. Wang, F., Cheng, J., Liu, W., Liu, H.: Additive margin softmax for face verification. IEEE Signal Process. Lett. 25(7), 926–930 (2018)

    CrossRef  Google Scholar 

  55. Wang, H., et al.: CosFace: large margin cosine loss for deep face recognition. In: CVPR (2018)

    Google Scholar 

  56. Wang, M., Deng, W.: Mitigating bias in face recognition using skewness-aware reinforcement learning. In: CVPR (2020)

    Google Scholar 

  57. Wang, M., Deng, W., Hu, J., Tao, X., Huang, Y.: Racial faces in the wild: reducing racial bias by information maximization adaptation network. In: ICCV (2019)

    Google Scholar 

  58. Wang, P., Su, F., Zhao, Z., Guo, Y., Zhao, Y., Zhuang, B.: Deep class-skewed learning for face recognition. Neurocomputing 363, 35–45 (2019)

    CrossRef  Google Scholar 

  59. Xie, W., Zisserman, A.: Multicolumn networks for face recognition. arXiv preprint arXiv:1807.09192 (2018)

  60. Yin, B., Tran, L., Li, H., Shen, X., Liu, X.: Towards interpretable face recognition. In: ICCV (2019)

    Google Scholar 

  61. Yin, X., Liu, X.: Multi-task convolutional neural network for pose-invariant face recognition. IEEE Trans. Image Process. 27(2), 964–975 (2017)

    MathSciNet  CrossRef  Google Scholar 

  62. Yin, X., Yu, X., Sohn, K., Liu, X., Chandraker, M.: Towards large-pose face frontalization in the wild. In: ICCV (2017)

    Google Scholar 

  63. Yin, X., Yu, X., Sohn, K., Liu, X., Chandraker, M.: Feature transfer learning for face recognition with under-represented data. In: CVPR (2019)

    Google Scholar 

  64. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    CrossRef  Google Scholar 

  65. Zhang, X., Fang, Z., Wen, Y., Li, Z., Qiao, Y.: Range loss for deep face recognition with long-tailed training data. In: CVPR (2017)

    Google Scholar 

  66. Zhang, Y., Zhou, Z.H.: Cost-sensitive face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1758–1769 (2009)

    CrossRef  Google Scholar 

  67. Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. In: CVPR. IEEE (2017)

    Google Scholar 

  68. Zhang, Z., et al.: Gait recognition via disentangled representation learning. In: CVPR (2019)

    Google Scholar 

Download references

Acknowledgement

This work is supported by U.S. Department of Commerce (#60NANB19D154), National Institute of Standards and Technology. The authors thank reviewers, area chairs, Dr. John J. Howard, and Dr. Yevgeniy Sirotin for offering constructive comments.

Author information

Authors and Affiliations

  1. Michigan State University, East Lansing, USA

    Sixue Gong, Xiaoming Liu & Anil K. Jain

Authors
  1. Sixue Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anil K. Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sixue Gong .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Prof. Dr. Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 229 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Gong, S., Liu, X., Jain, A.K. (2020). Jointly De-Biasing Face Recognition and Demographic Attribute Estimation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12374. Springer, Cham. https://doi.org/10.1007/978-3-030-58526-6_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58526-6_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58525-9

  • Online ISBN: 978-3-030-58526-6

  • eBook Packages: Computer ScienceComputer Science (R0)