Skip to main content
SpringerLink
Log in

Investigating Bias and Fairness in Facial Expression Recognition

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 Workshops (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12540))

Included in the following conference series:

Abstract

Recognition of expressions of emotions and affect from facial images is a well-studied research problem in the fields of affective computing and computer vision with a large number of datasets available containing facial images and corresponding expression labels. However, virtually none of these datasets have been acquired with consideration of fair distribution across the human population. Therefore, in this work, we undertake a systematic investigation of bias and fairness in facial expression recognition by comparing three different approaches, namely a baseline, an attribute-aware and a disentangled approach, on two well-known datasets, RAF-DB and CelebA. Our results indicate that: (i) data augmentation improves the accuracy of the baseline model, but this alone is unable to mitigate the bias effect; (ii) both the attribute-aware and the disentangled approaches equipped with data augmentation perform better than the baseline approach in terms of accuracy and fairness; (iii) the disentangled approach is the best for mitigating demographic bias; and (iv) the bias mitigation strategies are more suitable in the existence of uneven attribute distribution or imbalanced number of subgroup data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alvi, M., Zisserman, A., Nellåker, C.: Turning a blind eye: explicit removal of biases and variation from deep neural network embeddings. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  2. Amini, A., Soleimany, A.P., Schwarting, W., Bhatia, S.N., Rus, D.: Uncovering and mitigating algorithmic bias through learned latent structure. In: AAAI/ACM Conference on AI, Ethics, and Society, AIES (2019)

    Google Scholar 

  3. Bellamy, R.K., et al.: Ai fairness 360: an extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018)

  4. Breslin, S., Wadhwa, B.: Gender and Human-Computer Interaction, chap. 4, pp. 71–87. Wiley (2017). https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118976005.ch4

  5. Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency, pp. 77–91 (2018)

    Google Scholar 

  6. Clapes, A., Bilici, O., Temirova, D., Avots, E., Anbarjafari, G., Escalera, S.: From apparent to real age: gender, age, ethnic, makeup, and expression bias analysis in real age estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2373–2382 (2018)

    Google Scholar 

  7. Commission, E.: White paper on artificial intelligence-a european approach to excellence and trust (2020)

    Google Scholar 

  8. Das, A., Dantcheva, A., Bremond, F.: Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  9. Denton, E.L., Hutchinson, B., Mitchell, M., Gebru, T.: Detecting bias with generative counterfactual face attribute augmentation. ArXiv abs/1906.06439 (2019)

    Google Scholar 

  10. Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., Busch, C.: Demographic bias in biometrics: A survey on an emerging challenge. IEEE Transactions on Technology and Society (2020)

    Google Scholar 

  11. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, pp. 214–226 (2012)

    Google Scholar 

  12. Georgopoulos, M., Panagakis, Y., Pantic, M.: Investigating bias in deep face analysis: the kanface dataset and empirical study. Image and Vision Computing, in press (2020)

    Google Scholar 

  13. Gong, S., Liu, X., Jain, A.K.: Mitigating face recognition bias via group adaptive classifier. arXiv preprint arXiv:2006.07576 (2020)

  14. Grother, P., Ngan, M., Hanaoka, K.: Ongoing face recognition vendor test (frvt) part 3: demographic effects. National Institute of Standards and Technology, Tech. Rep. NISTIR 8280 (2019)

    Google Scholar 

  15. Gunes, H., Schuller, B.: Categorical and dimensional affect analysis in continuous input: current trends and future directions. Image Vis. Comput. 31, 120–136 (2013)

    Article  Google Scholar 

  16. Guo, G., Mu, G.: Human age estimation: what is the influence across race and gender? In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 71–78. IEEE (2010)

    Google Scholar 

  17. Hardt, M., Price, E., Srebro, N.: Equality of opportunity in supervised learning. In: Advances in Neural Information Processing Systems, pp. 3315–3323 (2016)

    Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  19. Howard, A., Zhang, C., Horvitz, E.: Addressing bias in machine learning algorithms: a pilot study on emotion recognition for intelligent systems. In: 2017 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), pp. 1–7 (2017)

    Google Scholar 

  20. Iosifidis, V., Ntoutsi, E.: Dealing with bias via data augmentation in supervised learning scenarios. Jo Bates Paul D. Clough Robert Jäschke 24 (2018)

    Google Scholar 

  21. Kamiran, F., Calders, T.: Data preprocessing techniques for classification without discrimination. Knowl. Inf. Syst. 33(1), 1–33 (2012)

    Article  Google Scholar 

  22. Khiyari, H., Wechsler, H.: Face verification subject to varying (age, ethnicity, and gender) demographics using deep learning. Journal of Biometrics & Biostatistics 07 (2016). https://doi.org/10.4172/2155-6180.1000323

  23. Kilbride, J.E., Yarczower, M.: Ethnic bias in the recognition of facial expressions. J. Nonverbal Behav. 8(1), 27–41 (1983)

    Article  Google Scholar 

  24. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  25. Koenecke, A., et al.: Racial disparities in automated speech recognition. Proc. National Acad. Sci. 117(14), 7684–7689 (2020)

    Article  Google Scholar 

  26. Kuo, C.M., Lai, S.H., Sarkis, M.: A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2121–2129 (2018)

    Google Scholar 

  27. Li, S., Deng, W.: Deep facial expression recognition: a survey. In: IEEE Transactions on Affective Computing, pp. 1–1 (2020)

    Google Scholar 

  28. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

    Google Scholar 

  29. Liu, Y., Wei, F., Shao, J., Sheng, L., Yan, J., Wang, X.: Exploring disentangled feature representation beyond face identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2080–2089 (2018)

    Google Scholar 

  30. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV), December 2015

    Google Scholar 

  31. Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., Bachem, O.: On the fairness of disentangled representations. In: Advances in Neural Information Processing Systems, pp. 14611–14624 (2019)

    Google Scholar 

  32. Lu, K., Mardziel, P., Wu, F., Amancharla, P., Datta, A.: Gender bias in neural natural language processing. ArXiv abs/1807.11714 (2018)

    Google Scholar 

  33. Martinez, B., Valstar, M.F., et al.: Automatic analysis of facial actions: a survey. IEEE Trans. Affective Comput. 10(3), 325–347 (2019)

    Article  Google Scholar 

  34. Mayson, S.G.: Bias in, bias out. YAle lJ 128, 2218 (2018)

    Google Scholar 

  35. Morales, A., Fierrez, J., Vera-Rodriguez, R.: Sensitivenets: Learning agnostic representations with application to face recognition. arXiv preprint arXiv:1902.00334 (2019)

  36. Ngxande, M., Tapamo, J., Burke, M.: Bias remediation in driver drowsiness detection systems using generative adversarial networks. IEEE Access 8, 55592–55601 (2020)

    Article  Google Scholar 

  37. Paszke, A., et al.: Automatic differentiation in pytorch (2017)

    Google Scholar 

  38. Phillips, P.J., Grother, P., Micheals, R., Blackburn, D.M., Tabassi, E., Bone, M.: Face recognition vendor test 2002. In: 2003 IEEE International SOI Conference. Proceedings (Cat. No. 03CH37443), p. 44. IEEE (2003)

    Google Scholar 

  39. du Pin Calmon, F., Wei, D., Vinzamuri, B., Ramamurthy, K.N., Varshney, K.R.: Optimized pre-processing for discrimination prevention. In: Advances in Neural Information Processing Systems (NIPS) (2017)

    Google Scholar 

  40. Rhue, L.: Racial influence on automated perceptions of emotions. Available at SSRN. https://doi.org/10.2139/ssrn.3281765 (2018)

  41. Robinson, J.P., Livitz, G., Henon, Y., Qin, C., Fu, Y., Timoner, S.: Face recognition: too bias, or not too bias? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–1 (2020)

    Google Scholar 

  42. Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(6), 1113–1133 (2015). https://doi.org/10.1109/TPAMI.2014.2366127

    Article  Google Scholar 

  43. Shin, M., Seo, J.H., Kwon, D.S.: Face image-based age and gender estimation with consideration of ethnic difference. In: 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pp. 567–572. IEEE (2017)

    Google Scholar 

  44. Terhörst, P., Kolf, J.N., Damer, N., Kirchbuchner, F., Kuijper, A.: Face quality estimation and its correlation to demographic and non-demographic bias in face recognition. arXiv preprint arXiv:2004.01019 (2020)

  45. Verma, S., Rubin, J.: Fairness definitions explained. In: 2018 IEEE/ACM International Workshop on Software Fairness (FairWare), pp. 1–7. IEEE (2018)

    Google Scholar 

  46. Wang, M., Deng, W.: Mitigate bias in face recognition using skewness-aware reinforcement learning. CoRR abs/1911.10692 (2019), http://arxiv.org/abs/1911.10692

  47. Wang, M., Deng, W., Hu, J., Tao, X., Huang, Y.: Racial faces in the wild: reducing racial bias by information maximization adaptation network. In: 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. pp. 692–702. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00078

  48. Wang, T., Zhao, J., Yatskar, M., Chang, K., Ordonez, V.: Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5309–5318 (2019)

    Google Scholar 

  49. Wang, Z., et al.: Towards fairness in visual recognition: effective strategies for bias mitigation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8919–8928 (2020)

    Google Scholar 

  50. Wu, W., Michalatos, P., Protopapaps, P., Yang, Z.: Gender classification and bias mitigation in facial images. In: 12th ACM Conference on Web Science, pp. 106–114 (2020)

    Google Scholar 

  51. Zhang, B.H., Lemoine, B., Mitchell, M.: Mitigating unwanted biases with adversarial learning. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 335–340 (2018)

    Google Scholar 

Download references

Acknowledgments

The work of T. Xu and H. Gunes is funded by the European Union’s Horizon 2020 research and innovation programme, under grant agreement No. 826232. S. Kalkan is supported by Scientific and Technological Research Council of Turkey (TÜBİTAK) through BIDEB 2219 Program.

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, University of Cambridge, Cambridge, UK

    Tian Xu, Jennifer White & Hatice Gunes

  2. Department of Computer Engineering, Middle East Technical University, Ankara, Turkey

    Sinan Kalkan

Authors
  1. Tian Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jennifer White
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sinan Kalkan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hatice Gunes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tian Xu .

Editor information

Editors and Affiliations

  1. University of Clermont Auvergne, Clermont Ferrand, France

    Adrien Bartoli

  2. Università degli Studi di Udine, Udine, Italy

    Andrea Fusiello

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, T., White, J., Kalkan, S., Gunes, H. (2020). Investigating Bias and Fairness in Facial Expression Recognition. In: Bartoli, A., Fusiello, A. (eds) Computer Vision – ECCV 2020 Workshops. ECCV 2020. Lecture Notes in Computer Science(), vol 12540. Springer, Cham. https://doi.org/10.1007/978-3-030-65414-6_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-65414-6_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-65413-9

  • Online ISBN: 978-3-030-65414-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation