Skip to main content
SpringerLink
Log in

Ethical AI in facial expression analysis: racial bias

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Facial expression recognition using deep neural networks has become very popular due to their successful performances. However, the datasets used during the development and testing of these methods lack a balanced distribution of races among the sample images. This leaves a possibility of the methods being biased toward certain races. Therefore, a concern about fairness arises, and the lack of research aimed at investigating racial bias only increases the concern. On the other hand, such bias in the method would decrease the real-world performance due to the wrong generalization. For these reasons, in this study, we investigated the racial bias within popular state-of-the-art facial expression recognition methods such as Deep Emotion, Self-Cure Network, ResNet50, InceptionV3, and DenseNet121. We compiled an elaborated dataset with images of different races, cross-checked the bias for methods trained, and tested on images of people of other races. We observed that the methods are inclined towards the races included in the training data. Moreover, an increase in the performance increases the bias as well if the training dataset is imbalanced. Some methods can make up for the bias if enough variance is provided in the training set. However, this does not mitigate the bias completely. Our findings suggest that an unbiased performance can be obtained by adding the missing races into the training data equally.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Roychowdhury, S., Emmons, M.: A survey of the trends in facial and expression recognition databases and methods. arXiv:1511.02407 (2015)

  2. Li, S., Deng, W.: Deep facial expression recognition: a survey. In: IEEE Transactions on Affective Computing. IEEE, pp 1–20 (2020)

  3. Kamińska, D., Aktas, K., Rizhinashvili, D., Kuklyanov, D., Sham, A.H., Escalera, S., Nasrollahi, K., Moeslund, T.B., Anbarjafari, G.: Two-stage recognition and beyond for compound facial emotion recognition. Electronics 10(22), 2847 (2021)

    Article  Google Scholar 

  4. Sang, D.V., Van Dat, N., et al.: Facial expression recognition using deep convolutional neural networks. In: 2017 9th International Conference on Knowledge and Systems Engineering (KSE), pp. 130–135. IEEE (2017)

  5. Drozdowski, P., Rathgeb, C., Dantcheva, A., Damer, N., Busch, C.: Demographic bias in biometrics: a survey on an emerging challenge. IEEE Trans. Technol. Soc. 1(2), 89–103 (2020)

    Article  Google Scholar 

  6. Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J.R., Jurafsky, D., Goel, S.: Racial disparities in automated speech recognition. Proc. Natl. Acad. Sci. 117(14), 7684–7689 (2020)

    Article  Google Scholar 

  7. Xu, T., White, J., Kalkan, S., Gunes, H.: Investigating bias and fairness in facial expression recognition. In: European Conference on Computer Vision, pp. 506–523. Springer (2020)

  8. Yang, K., Qinami, K., Fei-Fei, L., Deng, J., Russakovsky, O.: Towards fairer datasets: filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 547–558 (2020)

  9. De Vries, T., Misra, I., Wang, C., Van der Maaten, L.: Does object recognition work for everyone? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 52–59 (2019)

  10. Kasapoglu, T., Masso, A.: Attaining security through algorithms: perspectives of refugees and data experts. In: Theorizing Criminality and Policing in the Digital Media Age. Emerald Publishing Limited (2021)

  11. Perkowitz, S.: The bias in the machine: Facial recognition technology and racial disparities. MIT Case Studies in Social and Ethical Responsibilities of Computing, no. Winter. https://mit-serc.pubpub.org/pub/bias-in-machine (2021)

  12. Robinson, J.P., Livitz, G., Henon, Y., Qin, C., Fu, Y., Timoner, S.: Face recognition: too bias, or not too bias?” In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0–1 (2020)

  13. Das, A., Dantcheva, A., Bremond, F.: Mitigating bias in gender, age and ethnicity classification: a multi-task convolution neural network approach. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

  14. Guo, G., Mu, G.: Human age estimation: What is the influence across race and gender? In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 71–78. IEEE (2010)

  15. Chen, Y., Joo, J.: Understanding and mitigating annotation bias in facial expression recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14980–14991 (2021)

  16. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

  17. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

  18. Fu, S., He, H., Hou, Z.-G.: Learning race from face: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(12), 2483–2509 (2014)

    Article  Google Scholar 

  19. Domnich, A., Anbarjafari, G.: Responsible ai: gender bias assessment in emotion recognition. arXiv:2103.11436 (2021)

  20. Conley, M.I., Dellarco, D.V., Rubien-Thomas, E., Cohen, A.O., Cervera, A., Tottenham, N., Casey, B.: The racially diverse affective expression (radiate) face stimulus set. Psychiatry Res. 270, 1059–1067 (2018)

    Article  Google Scholar 

  21. Dailey, M.N., Joyce, C., Lyons, M.J., Kamachi, M., Ishi, H., Gyoba, J., Cottrell, G.W.: Evidence and a computational explanation of cultural differences in facial expression recognition. Emotion 10(6), 874 (2010)

    Article  Google Scholar 

  22. Fischer, A.H., Rodriguez Mosquera, P.M., Van Vianen, A.E., Manstead, A.S.: Gender and culture differences in emotion. Emotion 4(1), 87 (2004)

    Article  Google Scholar 

  23. Laurence, S., Zhou, X., Mondloch, C.J.: The flip side of the other-race coin: they all look different to me. Br. J. Psychol. 107(2), 374–388 (2016)

    Article  Google Scholar 

  24. Prado, C., Mellor, D., Byrne, L.K., Wilson, C., Xu, X., Liu, H.: Facial emotion recognition: a cross-cultural comparison of Chinese, Chinese living in Australia, and Anglo-Australians. Motiv. Emot. 38(3), 420–428 (2014)

    Article  Google Scholar 

  25. Strohminger, N., Gray, K., Chituc, V., Heffner, J., Schein, C., Heagins, T.B.: The mr2: a multi-racial, mega-resolution database of facial stimuli. Behav. Res. Methods 48(3), 1197–1204 (2016)

    Article  Google Scholar 

  26. Shimoda, K., Argyle, M., Bitti, P.R.: The intercultural recognition of emotional expressions by three national racial groups: English, Italian and Japanese. Eur. J. Soc. Psychol. 8(2), 169–179 (1978)

    Article  Google Scholar 

  27. Ghallab, M.: Responsible ai: requirements and challenges. AI Perspect. 1(1), 1–7 (2019)

    Article  Google Scholar 

  28. Benjamins, R., Barbado, A., Sierra, D.: Responsible ai by design in practice. arXiv:1909.12838 (2019)

  29. Vetrò, A., Santangelo, A., Beretta, E., De Martin, J.C.: Ai: from rational agents to socially responsible agents. Digital Policy, Regulation and Governance (2019)

  30. Livingston, M.: Preventing racial bias in federal ai, JSPG, vol. 16 (2020)

  31. Benjamins, R.: A choices framework for the responsible use of ai. AI Ethics 1(1), 49–53 (2021)

    Article  Google Scholar 

  32. Shneiderman, B.: Responsible ai: bridging from ethics to practice. Commun. ACM 64(8), 32–35 (2021)

    Article  Google Scholar 

  33. Wang, W., He, F., Zhao, Q.: Facial ethnicity classification with deep convolutional neural networks. In: Chinese Conference on Biometric Recognition, pp. 176–185. Springer (2016)

  34. Lopes, A.T., De Aguiar, E., De Souza, A.F., Oliveira-Santos, T.: Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn. 61, 610–628 (2017)

    Article  Google Scholar 

  35. Benitez-Garcia, G., Nakamura, T., Kaneko, M.: Multicultural facial expression recognition based on differences of Western-Caucasian and East-Asian facial expressions of emotions. IEICE Trans. Inf. Syst. 101(5), 1317–1324 (2018)

    Article  Google Scholar 

  36. Sohail, M., Ali, G., Rashid, J., Ahmad, I., Almotiri, S.H., AlGhamdi, M.A., Nagra, A.A., Masood, K.: Racial identity-aware facial expression recognition using deep convolutional neural networks. Appl. Sci. 12(1), 88 (2022)

    Article  Google Scholar 

  37. Olteanu, A., Castillo, C., Diaz, F., Kıcıman, E.: Social data: biases, methodological pitfalls, and ethical boundaries. Front. Big Data 2, 13 (2019)

    Article  Google Scholar 

  38. Minaee, S., Minaei, M., Abdolrashidi, A.: Deep-emotion: facial expression recognition using attentional convolutional network. Sensors 21(9), 3046 (2021)

    Article  Google Scholar 

  39. Kai, W., Xiaojiang, P., Jianfei, Y., Shijian, L., Yu, Q: Suppressing uncertainties for large-scale facial expression recognition. arXiv:2002.10392 (2020)

  40. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015)

  41. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna,Z.: Rethinking the inception architecture for computer vision (2015)

  42. Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. CoRR, vol. abs/1608.06993. http://arxiv.org/abs/1608.06993 (2016)

  43. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzzin. Knowl. Based Syst. 6(02), 107–116 (1998)

    Article  MATH  Google Scholar 

Download references

Acknowledgements

Our thanks to Pexels API for granting us the rights for the data collection of the database. This work has been partially supported by the EU MobilitasPluss grant (MOBTT90), Enactive Virtuality Lab, Tallinn University (2017-2022).

Author information

Author notes

  1. Abdallah Hussein Sham and Kadir Aktas are both equally led this work.

Authors and Affiliations

  1. Enactive Virtuality Lab, Baltic Film, Media, and Arts School, Tallinn University, Narva mnt 25, 10120, Tallinn, Estonia

    Abdallah Hussein Sham

  2. iCV Lab, University of Tartu, Tartu, Estonia

    Kadir Aktas, Davit Rizhinashvili, Danila Kuklianov, Ikechukwu Ofodile, Cagri Ozcinar & Gholamreza Anbarjafari

  3. iVCV, Tartu, 51011, Estonia

    Kadir Aktas & Gholamreza Anbarjafari

  4. Faculty of Engineering, Hasan Kalyoncu University, Gaziantep, Turkey

    Fatih Alisinanoglu

  5. PwC Advisory, Helsinki, Finland

    Gholamreza Anbarjafari

  6. Yildiz Technical University, Istanbul, Turkey

    Gholamreza Anbarjafari

Authors
  1. Abdallah Hussein Sham
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kadir Aktas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Davit Rizhinashvili
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Danila Kuklianov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fatih Alisinanoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ikechukwu Ofodile
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Cagri Ozcinar
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdallah Hussein Sham.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Our thanks to Pexels API for granting us the rights for the data collection of the database.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sham, A.H., Aktas, K., Rizhinashvili, D. et al. Ethical AI in facial expression analysis: racial bias. SIViP 17, 399–406 (2023). https://doi.org/10.1007/s11760-022-02246-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-022-02246-8

Keywords

Search

Navigation