Skip to main content

Causal inference for social discrimination reasoning


The discovery of discriminatory bias in human or automated decision making is a task of increasing importance and difficulty, exacerbated by the pervasive use of machine learning and data mining. Currently, discrimination discovery largely relies upon correlation analysis of decisions records, disregarding the impact of confounding biases. We present a method for causal discrimination discovery based on propensity score analysis, a statistical tool for filtering out the effect of confounding variables. We introduce causal measures of discrimination which quantify the effect of group membership on the decisions, and highlight causal discrimination/favoritism patterns by learning regression trees over the novel measures. We validate our approach on two real world datasets. Our proposed framework for causal discrimination has the potential to enhance the transparency of machine learning with tools for detecting discriminatory bias both in the training data and in the learning algorithms.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5


  1. See e.g., the AI Now report on Discriminating Systems, April 2019 (

  2. Code at:



  • Agresti, A. (2002). Categorical data analysis. Wiley series in probability and statistics, 2 edn. Wiley-Interscience.

  • Austin, P.C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424.

    Article  Google Scholar 

  • Baeza-Yates, R.A. (2018). Bias on the web. Communications of the ACM, 61(6), 54–61.

    Article  Google Scholar 

  • Barocas, S., & Selbst, A.D. (2016). Big data’s disparate impact. California Law Review, 104.

  • Bendic, M. (2007). Situation testing for employment discrimination in the United States of America. Horizons Stratégiques, 3(5), 17–39.

    Google Scholar 

  • Berk, R., Heidari, H., Jabbari, S., Kearns, M., Roth, A. (2018). Fairness in criminal justice risk assessments: the state of the art. Sociological Methods & Research.

  • Bickel, P.J., Hammel, E.A., O’Connell, J.W. (1975). Sex bias in graduate admissions: data from Berkeley. Science, 187(4175), 398–404.

    Article  Google Scholar 

  • Bolukbasi, T., Chang, K., Zou, J.Y., Saligrama, V., Kalai, A.T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In NIPS (pp. 4349–4357).

  • Bonchi, F., Hajian, S., Mishra, B., Ramazzotti, D. (2017). Exposing the probabilistic causal structure of discrimination. I. Journal Data Science and Analytics, 3(1), 1–21.

    Article  Google Scholar 

  • Breiman, L., Friedman, J, Olshen, R., Stone, C. (1984). Classification and regression trees. Wadsworth Publishing Company.

  • Bryson, A., Dorsett, R., Purdon, S. (2002). The use of propensity score matching in the evaluation of active labour market policies. Crown.

  • Calders, T., Karim, A., Kamiran, F., Ali, W., Zhang, X. (2013). Controlling attribute effect in linear regression. In ICDM (pp. 71–80): IEEE.

  • Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of Economic Surveys, 22(1), 31–72.

    Article  Google Scholar 

  • Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1).

  • Fortin, N., Lemieux, T., Firpo, S. (2011). Decomposition methods in economics. In Handbook of labor economics, (Vol. 4 pp. 1–102): Elsevier.

  • Foster, S.R. (2004). Causation in antidiscrimination law: beyond intent versus impact. Houston Law Review, 41(5), 1469–1548.

    Google Scholar 

  • Grimes, D.A., & Schulz, K.F. (2002). Bias and causal associations in observational research. Lancet, 359, 248–252.

    Article  Google Scholar 

  • Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D. (2019). A survey of methods for explaining black box models. ACM Computing Survey, 51(5), 93:1–93:42.

    Google Scholar 

  • Guo, X.S., & Fraser, M.W. (2015). Propensity score analysis: statistical methods and applications, Sage Publications, Inc., 2.

  • Kilbertus, N., Ball, P.J., Kusner, M.J., Weller, A., Silva, R. (2019). The sensitivity of counterfactual fairness to unmeasured confounding. In UAI (p. 213): AUAI Press.

  • Kohavi, R., & Longbotham, R. (2017). Online controlled experiments and A/B testing. In Encyclopedia of machine learning and data mining (pp. 922–929): Springer.

  • Kohler-Hausmann, I. (2019). Eddie Murphy and the dangers of counterfactual causal thinking about detecting racial discrimination . Northwestern University Law Rev, 113, 1163–1227.

    Google Scholar 

  • Kulshrestha, J., Eslami, M., Messias, J., Zafar, M.B., Ghosh, S., Gummadi, K.P., Karahalios, K. (2019). Search bias quantification: investigating political bias in social media and web search. Information Retrieval Journal, 22(1–2), 188–227.

    Article  Google Scholar 

  • Kusner, M.J., Loftus, J.R., Russell, C., Silva, R. (2017). Counterfactual fairness. In NIPS (pp. 4069–4079).

  • Loftus, J.R., Russell, C., Kusner, M.J., Silva, R. (2018). Causal reasoning for algorithmic fairness. arXiv:abs/1805.05859.

  • Luong, B.T., Ruggieri, S., Turini, F. (2011). k-NN as an implementation of situation testing for discrimination discovery and prevention. In KDD (pp. 502–510): ACM.

  • Morgan, S.L., & Todd, J.L. (2008). A diagnostic routine for the detection of consequential heterogeneity of causal effects. Sociological Methodology, 38(1), 231–281.

    Article  Google Scholar 

  • Pearl, J. (2009). Causality: models, reasoning, and inference, 2nd edn. New York: Cambridge University Press.

    Book  Google Scholar 

  • Romei, A., & Ruggieri, S. (2014). A multidisciplinary survey on discrimination analysis. The Knowledge Engineering Review, 29(5), 582–638.

    Article  Google Scholar 

  • Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

    MathSciNet  Article  Google Scholar 

  • Shadish, W.R., Cook, T.D., Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton-Mifflin.

  • Verma, S., & Rubin, J. (2018). Fairness definitions explained. In FairWare@ICSE (pp. 1–7): ACM.

  • Wu, Y., Zhang, L., Wu, X. (2019). Counterfactual fairness: Unidentification, bound and algorithm. In IJCAI. (pp. 1438–1444).

  • Zhang, J., & Bareinboim, E. (2018). Fairness in decision-making - the causal explanation formula. In AAAI: AAAI Press.

  • Zhang, L., & Wu, X. (2017). Anti-discrimination learning: a causal modeling-based framework. I. Journal Data Science and Analytics, 4(1), 1–16.

    Article  Google Scholar 

  • Zhang, L., Wu, Y., Wu, X. (2016). Situation testing-based discrimination discovery: a causal inference approach. In IJCAI (pp. 2718–2724).

  • Zhang, L., Wu, Y., Wu, X. (2017). Achieving non-discrimination in data release. In KDD (pp. 1335–1344): ACM.

  • Zliobaite, I. (2017). Measuring discrimination in algorithmic decision making. Data Mining and Knowledge Discovery, 31(4), 1060–1089.

    MathSciNet  Article  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Bilal Qureshi.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Qureshi, B., Kamiran, F., Karim, A. et al. Causal inference for social discrimination reasoning. J Intell Inf Syst 54, 425–437 (2020).

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI:


  • Social discrimination
  • Fairness, accountability, and transparency
  • Propensity score
  • Causal analysis