Skip to main content
SpringerLink
Log in

Do the Ends Justify the Means? Variation in the Distributive and Procedural Fairness of Machine Learning Algorithms

  • Original Paper
  • Published:
Journal of Business Ethics Aims and scope Submit manuscript

Abstract

Recent advances in machine learning methods have created opportunities to eliminate unfairness from algorithmic decision making. Multiple computational techniques (i.e., algorithmic fairness criteria) have arisen out of this work. Yet, urgent questions remain about the perceived fairness of these criteria and in which situations organizations should use them. In this paper, we seek to gain insight into these questions by exploring fairness perceptions of five algorithmic criteria. We focus on two key dimensions of fairness evaluations: distributive fairness and procedural fairness. We shed light on variation in the potential for different algorithmic criteria to facilitate distributive fairness. Subsequently, we discuss procedural fairness and provide a framework for understanding how algorithmic criteria relate to essential aspects of this construct, which helps to identify when a specific criterion is suitable. From a practical standpoint, we encourage organizations to recognize that managing fairness in machine learning systems is complex, and that adopting a blind or one-size-fits-all mentality toward algorithmic criteria will surely damage people’s attitudes and trust in automated technology. Instead, firms should carefully consider the subtle yet significant differences between these technical solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Actual human operators label the outcomes in the case of supervised learning. This paper assumes that the developers train the ML model on a training set created using human input.

  2. Organizational justice researchers have also studied fairness as a single dimension (e.g., Ambrose & Schminke, 2009) and as a multidimensional construct comprising distributive, procedural, and interactional fairness components (Colquitt et al., 2013; Karriker & Williams, 2009).

  3. In line with organizational justice scholarship, we use the terms fairness and justice interchangeably. Although differences exist among the concepts, both are geared toward promoting equity and avoiding bias.

  4. Closely related to representativeness is the concept of voice, which allows individuals from different subgroups to express their concerns, opinions, and values to decision makers as part of the decision process (Thibaut & Walker, 1975). While we believe that voice is an influential factor to consider when examining perceived fairness, it falls outside Leventhal’s (1980) theory of procedural fairness criteria—our current focus—and is thus beyond the scope of this paper.

References

  • Adams, J. S. (1965). Inequity in social exchange. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 2, pp. 267–299). Academic Press.

    Google Scholar 

  • Ambrose, M. L., & Schminke, M. (2009). The role of overall justice judgments in organizational justice research: A test of mediation. Journal of Applied Psychology, 94(2), 491–500.

    Article  Google Scholar 

  • Apfelbaum, E. P., Pauker, K., Sommers, S. R., & Ambady, N. (2010). In blind pursuit of racial equality? Psychological Science, 21(11), 1587–1592.

    Article  Google Scholar 

  • Barrett-Howard, E., & Tyler, T. R. (1986). Procedural justice as a criterion in allocation decisions. Journal of Personality and Social Psychology, 50(2), 296–304.

    Article  Google Scholar 

  • Beckman, C. M., & Haunschild, P. R. (2002). Network learning: The effects of partners’ heterogeneity of experience on corporate acquisitions. Administrative Science Quarterly, 47(1), 92–124.

    Article  Google Scholar 

  • Bigman, Y., Gray, K., Waytz, A., Arnestad, M., & Wilson, D. (2020). Algorithmic discrimination causes less moral outrage than human discrimination. PsyArXiv. https://doi.org/10.31234/osf.io/m3nrp.

    Article  Google Scholar 

  • Bird, S., Kenthapadi, K., Kiciman, E., & Mitchell, M. (2019). Fairness-aware machine learning: Practical challenges and lessons learned. Proceedings of the ACM International Conference on Web Search and Data Mining. https://doi.org/10.1145/3308560.3320086.

  • Brennan, T., Dieterich, W., & Ehret, B. (2009). Evaluating the predictive validity of the COMPAS risk and needs assessment system. Criminal Justice and Behavior, 36(1), 21–40.

    Article  Google Scholar 

  • Chao, G. T., & Moon, H. (2005). The cultural mosaic: A metatheory for understanding the complexity of culture. Journal of Applied Psychology, 90(6), 1128–1140.

    Article  Google Scholar 

  • Chouldechova, A., & Roth, A. (2018). The frontiers of fairness in machine learning. arXiv. https://arxiv.org/abs/1810.08810.

  • Clarke, J. A. (2017). Protected class gatekeeping. NYU Law Review, 92, 101.

    Google Scholar 

  • Cohen, T. R., & Morse, L. (2014). Moral character: What it is and what it does. Research in Organizational Behavior, 34, 43–61.

    Article  Google Scholar 

  • Cohen, T. R., Panter, A. T., Turan, N., Morse, L., & Kim, Y. (2014). Moral character in the workplace. Journal of Personality and Social Psychology, 107(5), 943–963.

    Article  Google Scholar 

  • Colquitt, J. A. (2012). Organizational justice. In S. W. J. Kozlowski (Ed.), The Oxford handbook of organizational psychology (pp. 526–547). Oxford University Press.

    Chapter  Google Scholar 

  • Colquitt, J. A., & Rodell, J. B. (2015). Measuring justice and fairness. In R. S. Cropanzano & M. L. Ambrose (Eds.), Oxford library of psychology. The Oxford handbook of justice in the workplace (pp. 187–202). Oxford University Press.

    Google Scholar 

  • Colquitt, J. A., Conlon, D. E., Wesson, M. J., Porter, C. O., & Ng, K. Y. (2001). Justice at the millennium: A meta-analytic review of 25 years of organizational justice research. Journal of Applied Psychology, 86(3), 425–445.

    Article  Google Scholar 

  • Colquitt, J. A., Scott, B. A., Rodell, J. B., Long, D. M., Zapata, C. P., Conlon, D. E., & Wesson, M. J. (2013). Justice at the millennium, a decade later: A meta-analytic test of social exchange and affect-based perspectives. Journal of Applied Psychology, 98(2), 199–236.

    Article  Google Scholar 

  • Cook, I. (2021). How HR Can Tackle Diversity Using the Rooney Rule. Visier. Retrieved from https://www.visier.com/clarity/how-hr-can-tackle-diversity-using-the-rooney-rule/.

  • Cropanzano, R., & Stein, J. H. (2009). Organizational justice and behavioral ethics: Promises and prospects. Business Ethics Quarterly, 19, 193–233.

    Article  Google Scholar 

  • Dastin, J. (2018, October 10). Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women. Reuters Business News. Retrieved from https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G.

  • Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). Fairness through awareness. Proceedings of the Innovations in Theoretical Computer Science Conference. https://arxiv.org/abs/1104.3913.

  • Ely, R. J., & Thomas, D. A. (2001). Cultural diversity at work: The effects of diversity perspectives on work group processes and outcomes. Administrative Science Quarterly, 46(2), 229–273.

    Article  Google Scholar 

  • Farnadi, G., Babaki, B., & Getoor, L. (2018). Fairness in relational domains. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3278721.3278733.

  • Farrar, J., Massey, D. W., Osecki, E., & Thorne, L. (2020). Tax fairness: Conceptual foundations and empirical measurement. Journal of Business Ethics, 162, 487–503.

    Article  Google Scholar 

  • Ghassami, A. (2018). Fairness in supervised learning: An information theoretic approach. IEEE International Symposium on Information Theory. https://doi.org/10.1109/isit.2018.8437807.

    Article  Google Scholar 

  • Glikson, E., & Woolley, A. W. (2020). Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals, 14(2), 627–660.

    Article  Google Scholar 

  • Goldman, B., & Cropanzano, R. (2015). “Justice” and “fairness” are not the same thing. Journal of Organizational Behavior, 36(2), 313–318.

    Article  Google Scholar 

  • Greenberg, J. (2011). Organizational justice: The dynamics of fairness in the workplace. In S. Zedeck (Ed.), APA handbook of industrial and organizational psychology (pp. 271–327). American Psychological Association.

    Google Scholar 

  • Greenwood, B. N., Adjerid, I., Angst, C., & Meikle, N. (2020). How unbecoming of you: Online experiments uncovering gender biases in perceptions of ridesharing performance. Journal of Business Ethics, 18, 1–20.

    Google Scholar 

  • Grgić-Hlača, N., Zafar, M. B., Gummadi, K. P., & Weller, A. (2018). Beyond distributive fairness in algorithmic decision making: Feature selection for procedurally fair learning. Proceedings of the AAAI Conference on Artificial Intelligence. http://mlg.eng.cam.ac.uk/adrian/AAAI18-BeyondDistributiveFairness.pdf.

  • Hambrick, D. C., Cho, T. S., & Chen, M.-T. (1996). The influence of top management team heterogeneity on firms’ competitive moves. Administrative Science Quarterly, 41(4), 659–684.

    Article  Google Scholar 

  • Hardt, M., Price, E., & Srebro, N. (2016). Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems. https://arxiv.org/abs/1610.02413.

  • Harrison, D. A., Price, K. H., & Bell, M. P. (1998). Beyond relational demography: Time and the effects of surface-and deep-level diversity on work group cohesion. Academy of Management Journal, 41(1), 96–107.

    Article  Google Scholar 

  • Hu, L., & Chen, Y. (2018). A short-term intervention for long-term fairness in the labor market. Proceedings of the World Wide Web Conference. https://arxiv.org/abs/1712.00064.

  • James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. Springer.

    Book  Google Scholar 

  • Jehn, K. A., Northcraft, G. B., & Neale, M. A. (1999). Why differences make a difference: A field study of diversity, conflict and performance in workgroups. Administrative Science Quarterly, 44(4), 741–763.

    Article  Google Scholar 

  • Karriker, J. H., & Williams, M. L. (2009). Organizational justice and organizational citizenship behavior: A mediated multifoci model. Journal of Management, 35(1), 112–135.

    Article  Google Scholar 

  • Khan, K., Abbas, M., Gul, A., & Raja, U. (2015). Organizational justice and job outcomes: Moderating role of Islamic work ethic. Journal of Business Ethics, 126(2), 235–246.

    Article  Google Scholar 

  • Kim, T. W., & Scheller-Wolf, A. (2019). Technological unemployment, meaning in life, purpose of business, and the future of stakeholders. Journal of Business Ethics, 160(2), 319–337.

    Article  Google Scholar 

  • Knight, W. (2019, November 19). The Apple Credit Card Didn’t ‘See’ Gender—and That’s the Problem. Wired. Retrieved from https://www.wired.com/story/the-apple-card-didnt-see-genderand-thats-the-problem/.

  • Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017). Counterfactual Fairness. Advances in Neural Information Processing Systems. https://arxiv.org/abs/1703.06856.

  • Lazo, C. (2020). Toward Engineering AI Software for Fairness [MSc Thesis, Delft University of Technology]. TUDelft Respository.

  • Lee, M. K. (2018). Understanding perception of algorithmic decisions: Fairness, trust, and emotion in response to algorithmic management. Big Data & Society. 5(1), 1–16. https://doi.org/10.1177/2053951718756684.

    Article  Google Scholar 

  • Lee, M. K., Jain, A., Cha, H. J., Ojha, S., & Kusbit, D. (2019). Procedural justice in algorithmic fairness: Leveraging transparency and outcome control for fair algorithmic mediation. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3359284.

    Article  Google Scholar 

  • Leicht-Deobald, U., Busch, T., Schank, C., Weibel, A., Schafheitle, S., Wildhaber, I., & Kasper, G. (2019). The challenges of algorithm-based HR decision-making for personal integrity. Journal of Business Ethics, 160, 377–392.

    Article  Google Scholar 

  • Leventhal, G. S. (1980). What should be done with equity theory? In K. J. Gergen, M. S. Greenberg, & R. H. Willis (Eds.), Social exchange (pp. 27–55). Springer.

    Chapter  Google Scholar 

  • Martin, K. (2019a). Ethical implications and accountability of algorithms. Journal of Business Ethics, 160, 835–850.

    Article  Google Scholar 

  • Martin, K. (2019b). Designing ethical algorithms. MIS Quarterly Executive, 18(2), 129–142.

    Article  Google Scholar 

  • Martin, K., & Freeman, R. E. (2004). The separation of technology and ethics in business ethics. Journal of Business Ethics, 53(4), 353–364.

    Article  Google Scholar 

  • Mathur, P., & Sarin Jain, S. (2020). Not all that glitters is golden: The impact of procedural fairness perceptions on firm evaluations and customer satisfaction with favorable outcomes. Journal of Business Research, 117, 357–367.

    Article  Google Scholar 

  • McFarlin, D. B., & Sweeney, P. D. (1992). Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes. Academy of Management Journal, 35(3), 626–637.

    Article  Google Scholar 

  • Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019). A Survey on Bias and Fairness in Machine Learning. arXiv. https://arxiv.org/abs/1908.09635.

  • Miller, A. P. (2018, July 26). Want Less-Biased Decisions? Use Algorithms. Harvard Business Review. Retrieved from https://hbr.org/2018/07/want-less-biased-decisions-usealgorithms.

  • Newman, D. T., Fast, N. J., & Harmon, D. J. (2020). When eliminating bias isn’t fair: Algorithmic reductionism and procedural justice in human resource decisions. Organizational Behavior and Human Decision Processes, 160, 149–167.

    Article  Google Scholar 

  • North-Samardzic, A. (2019). Biometric technology and ethics: Beyond security applications. Journal of Business Ethics, 167, 433–450.

    Article  Google Scholar 

  • O’Neil, C. (2016). Weapons of math destruction: How big data increases inequality and threatens democracy. Crown.

    Google Scholar 

  • Patra, K. (2020, May 18). NFL Instituting Changes to the Rooney Rule. NFL. Retrieved from https://www.nfl.com/news/nfl-instituting-changes-to-rooney-rule.

  • Pezzo, M. V., & Beckstead, J. W. (2020). Algorithm aversion is too often presented as though it were non-compensatory: A reply to Longoni et al. (2020). Judgment and Decision Making, 15(3), 449–451.

    Google Scholar 

  • Pfeffer, J., & Langton, N. (1988). Wage inequality and the organization of work: The case of academic departments. Administrative Science Quarterly, 33(4), 588–606.

    Article  Google Scholar 

  • Podsiadlowski, A., Gröschke, D., Kogler, M., Springer, C., & Van Der Zee, K. (2013). Managing a culturally diverse workforce: Diversity perspectives in organizations. International Journal of Intercultural Relations, 37(2), 159–175.

    Article  Google Scholar 

  • Purdie-Vaughns, V., & Eibach, R. P. (2008). Intersectional invisibility: The distinctive advantages and disadvantages of multiple subordinate-group identities. Sex Roles, 59(5–6), 377–391.

    Article  Google Scholar 

  • Rafaeli, A., & Pratt, M. G. (1993). Tailored meanings: On the meaning and impact of organizational dress. Academy of Management Review, 18(1), 32–55.

    Article  Google Scholar 

  • Robert, L. P., Pierce, C., Marquis, L., Kim, S., & Alahmad, R. (2020). Designing fair AI for managing employees in organizations: A review, critique, and design agenda. Human-Computer Interaction, 35(5–6), 1–31.

    Google Scholar 

  • Saxena, N., Huang, K., DeFilippis, E., Radanovic, G., Parkes, G., & Liu, Y. (2019). How do fairness definitions fare? Examining public attitudes towards algorithmic definitions of fairness. arXiv. arxiv:1811.03654.

  • Schwartz, D. S. (2009). The case of the vanishing protected class: reflections on reverse discrimination, affirmative action, and racial balancing. Wisconsin Law Review, 2, 657.

  • Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019). Fairness and abstraction in sociotechnical systems. Proceedings of the Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3287560.3287598.

    Article  Google Scholar 

  • Silverman, R. E., & Waller, N. (2015, March 13). The algorithm that tells the boss who might quit. Wall Street Journal. Retrieved from https://www.wsj.com/articles/the-algorithm-that-tells-the-boss-who-might-quit-1426287935.

  • Solomon, B., Hall, M. E. K., & Muir, C. P. (2021). When and why bias suppression is difficult to sustain: The asymmetric effect of intermittent accountability. Academy of Management Journal. Advance online publication. https://doi.org/10.5465/amj.2020.0441.

    Article  Google Scholar 

  • Teodorescu, M. H. M. (2017). Machine Learning methods for strategy research. Harvard Business School Research Paper Series #18–011. https://www.hbs.edu/faculty/Pages/item.aspx?num=53076.

  • Teodorescu, M. H. M., & Yao, X. (2021). Machine Learning Fairness is Computationally Difficult and Algorithmically Unsatisfactorily Unsolved. Proceedings of IEEE High Performance Computing Conference.

  • Teodorescu, M. H. M., Morse, L., Awwad, Y., & Kane, G. C. (2021). Failures of fairness in automation require a deeper understanding of human–ML augmentation. MIS Quarterly, 45(3b), 1483–1499. 

    Article  Google Scholar 

  • Thibaut, J., & Walker, L. (1975). Procedural justice: A psychological analysis. Erlbaum.

    Google Scholar 

  • Tyler, T. R. (2003). Procedural justice, legitimacy, and the effective rule of law. Crime and Justice, 30, 283–357.

    Article  Google Scholar 

  • Valcke, B., Van Hiel, A., Onraet, E., & Dierckx, K. (2020). Procedural fairness enacted by societal actors increases social trust and social acceptance among ethnic minority members through the promotion of sense of societal belonging. Journal of Applied Social Psychology, 50, 573–587.

    Article  Google Scholar 

  • van den Bos, K., Wilke, H. A. M., & Lind, E. A. (1998). When do we need procedural fairness? The role of trust in authority. Journal of Personality and Social Psychology, 75(6), 1449–1458.

    Article  Google Scholar 

  • van den Bos, K., Lind, E. A., & Wilke, H. A. M. (2001). The psychology of procedural and distributive justice viewed from the perspective of fairness heuristic theory. In R. Cropanzano (Ed.), Series in applied psychology. Justice in the workplace: From theory to practice (pp. 49–66). Lawrence Erlbaum Associates Publishers.

    Google Scholar 

  • van der Toorn, J., Tyler, T. R., & Jost, J. T. (2011). More than fair: Outcome dependence, system justification, and the perceived legitimacy of authority figures. Journal of Experimental Social Psychology, 47(1), 127–138.

    Article  Google Scholar 

  • Verma, S., & Rubin, J. (2018). Fairness definitions explained. Proceedings of the International Workshop on Software Fairness. https://doi.org/10.1145/3194770.3194776.

    Article  Google Scholar 

  • Zhao, H., Coston, A., Adel, T., & Gordon, G. J. (2019). Conditional learning of fair representations. arXiv. https://arxiv.org/abs/1910.07162.

Download references

Acknowledgements

We are grateful for feedback and advice from Daniel Frey, Sam Ransbotham, Aubra Anthony, Shachee Doshi, Craig Jolley, Amy Paul, Maggie Linak, Rich Fletcher, Amit Gandhi, Lauren McKown, Kendra Leith, Nancy Adams, John Deighton, and the anonymous referees at the Society for Business Ethics Conference, Academy of Management Annual Meeting, Strategic Management Society Conference, and NYU AI Conference.

Funding

This research was partially supported by USAID Grant AID-OAA-A-12-00095.

Author information

Authors and Affiliations

  1. College of Business & Economics, West Virginia University, Morgantown, USA

    Lily Morse

  2. Carroll School of Management, Boston College, Chestnut Hill, USA

    Mike Horia M. Teodorescu & Gerald C. Kane

  3. D-Lab, Massachusetts Institute of Technology, Cambridge, USA

    Mike Horia M. Teodorescu & Yazeed Awwad

Authors
  1. Lily Morse
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mike Horia M. Teodorescu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yazeed Awwad
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerald C. Kane
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lily Morse.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies involving human subjects or animals performed by any of the authors.

Informed Consent

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Morse, L., Teodorescu, M.H.M., Awwad, Y. et al. Do the Ends Justify the Means? Variation in the Distributive and Procedural Fairness of Machine Learning Algorithms. J Bus Ethics 181, 1083–1095 (2022). https://doi.org/10.1007/s10551-021-04939-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10551-021-04939-5

Keywords

Search

Navigation