1 Introduction

Recent years have brought fast and impressive advances in artificial intelligence (AI), in particular within its subset machine learning (ML). This progress has been driven by a virtuous circle where new algorithms, more online data and cheaper computing power have reinforced each other (Jordan and Mitchell, 2015). Even though some of the enthusiasm has been premature (Hutson, 2020; Cross, 2020), it is clear that AI has, and will continue to have, profound effects on modern society.

However, concerns have been raised about the black box-like nature of many of these systems: while results are often impressive, systems often depend on millions of parameters determined by training on large data sets, making it difficult explain why particular decisions were made (Castelvecchi, 2016). Furthermore, many disconcerting discoveries have been made of cases where systems have turned out to be biased in various ways, disadvantaging, e.g., poorer people and those from minorities (Nature, 2016). As a result, there is much technically oriented research on how to “open” black boxes and offer more transparent and explainable AI (for recent reviews see, e.g., Guidotti et al., 2018; Du et al., 2019), and also a growing body of philosophical literature addressing questions of transparency, bias, and responsibility in AI (see, e.g., Floridi et al., 2018; de Laat 2018; Zerilli et al.2019).

However, it should be noted that while AI and particularly ML black boxes get much attention, concerns should not be limited to these particular technologies only. Fleischmann and Wallace (2005), well before the current deep learning–focused AI revolution, argue convincingly that the problems of opacity can be as pressing in other kinds of automated decision-making.

The purpose of this article is to make a contribution to the literature on transparency, bias, and responsibility in automated decision-making by drawing attention to and further elaborating a distinction first made by Nozick (1993, pp. 103–106) between first-level bias and second-level bias, as well as a second distinction between discrimination and arbitrariness. While these distinctions have not received much attention in the literature, the following sections aim to demonstrate how to apply them to problems of black box-like automated decision-making, allowing us to characterize algorithmic fairness initiatives in a new and illuminating way.

2 Bias: Two Distinctions

In The Nature of Rationality, Nozick (1993, pp. 103–106) distinguishes two kinds of bias: First-level bias “involves the uneven application of existing standards. Discrimination in the social arena, for instance, involves the uneven application of standards to different groups or individuals” (p. 103). This is the case if, for example, one standard is applied to members of one particular group (e.g. “ah, you belong to group X, then I will grant your loan immediately”), whereas another standard is applied to members of another group (e.g. “ah, you belong to group Y, then I will need to see your latest paycheck to process your loan”).

Second-level bias, by contrast, is the case when “the explanation of why these standards rather than others are chosen, or why these weights rather than others are given, in part involves the belief by some that these very standards and weights would work to the exclusion or detriment of particular groups and this motivated them to put forward these particular standards. These standards were chosen in order to exclude certain cases” (Nozick 1993, p. 103, emphasis in original). (Continuing the example above, the bigoted banker would explicitly exhibit second-level bias by saying “let us make all loan applicants produce their latest paycheck—that should get us rid of group Y.”)

This distinction between first- and second-level bias is the main distinction made by Nozick, and the only one made explicitly. However, there is also a second distinction implicitly made: between discrimination and arbitrariness. In the main text, Nozick uses discrimination (first- and second-level) rather than bias throughout the later part of his treatment (pp. 104–106), where second-level discrimination is described as the case when “the standards applied (or the weights they are given) are not a random sample from the set of possible relevant standards. Moreover, any criteria that purport to justify these particular standards (or weights) are not a random sample (or objectively justified subset) of the possibly relevant evaluative criteria. Here the sampling is intentionally biased” (Nozick 1993, p. 105). However, Nozick immediately proceeds to remark that “[i]n other cases the matter may be less clear”, and introduces the concept of second-level arbitrariness in a note, remarking that “it need not be discrimination” (Nozick 1993, p. 200, note 60). Thus, at least on the second level (choice of standards), there is a difference between discrimination and arbitrariness. Indeed, it seems reasonable to extend this to the first level (application of standards) as well, even though Nozick does not mention this case.

Trying to account for both distinctions, throughout the rest of this article, we use bias as a general term, corresponding to the heading of the section in Nozick’s treatment (p. 100), and to Nozick’s promise to say “something about the general notion of bias” (p. 103). Furthermore, heeding both the explicit and the implicit distinction, we partition bias into discrimination if intentional and done in order to exclude certain cases,Footnote 1 and arbitrariness otherwise (i.e., if it is unintentional and thus not done in order to accomplish anything in particular, or if it is intentional, but not done in order to exclude certain cases). Thus, we get a fourfold typology of bias; {first-level, second-level} × {discrimination, arbitrariness} as summarized in Table 1. Unbiased decisions (first- and second level) are not depicted—but could of course be imagined as an additional column.

Table 1 A fourfold typology of bias
Full size table

2.1 Remarks and Delimitations

It should immediately be remarked that arbitrariness, on this definition, is a matter of degree. It ranges from completely unintentional uneven applications and choices of standards to applications and choices of standards that may be thoroughly considered and fully intentional, but still biased in that they are uneven. On the first level, acts such as clemency and pardons are examples of highly intentional uneven application of standards, not to the exclusion or detriment of particular groups, but rather to the inclusion or benefit of particular individuals. (Continuing the banking example: “I really should ask you to produce your latest paycheck, but let’s forgo that—I will grant your loan immediately.”) On the second level, a standard may be chosen without regard to all the relevant alternatives: You may be a text-book Kantian, evenly applying Kantian standards to first-level matters, but if you have never read the other chapters in the text-book on normative theories, then you exhibit second level-arbitrariness in your choice of Kantianism as the standard to adhere to. Thus, arbitrariness is a matter of degree. If a “rational person will use some procedures to operate upon and correct other of her procedures, to correct biases in the sampling and evaluation of reasons and in the procedure of estimation based upon them” (Nozick 1993, p. 102, emphasis in original), then arbitrariness will be reduced, even if it is not completely removed. Reading the rest of the text-book may not make the text-book Kantian completely unbiased, but certainly much less biased.

Furthermore, the neat partitioning of actions into intentional and unintentional is clearly a simplification. In particular, there is the complication of indirect or oblique intent, i.e., the situation wherein you are aware that your act will or is highly likely to produce a particular outcome, even if you do not especially desire it. This is likely to be true of many instances of discrimination: you may not particularly desire to exclude a particular group, but accept this as a side-effect of an outcome you do desire. Nozick asks a similar question: “When the standard is initiated as second-level bias but other reasons also can be produced for it, and the standard is continued in part for these other reasons, the definitional and normative situation becomes more complicated.” (Nozick 1993, p. 103) We do not resolve this complicated situation here, though we revisit it in Sections 3 and 5.2.

It should also be remarked that even though it may be tempting to look askance at the left part of Table 1, the typology is not intended to delineate acts which are morally right from acts which are morally wrong under any particular normative theory. For example, it is not difficult to imagine fully legitimate second-level discrimination as defined in Table 1. Continuing the banking example, choosing standards in order to exclude those who cannot be expected to be able to pay interest and amortization seems mandatory for any prudent banker—perhaps foremost as an obligation to other customers, but it is not far-fetched to imagine a Socratic dialog convincing you that if you are denied the loan for this reason, this is the best even for yourself. More generally, looking at Table 1, it is clear that some normative theories, such as text-book act utilitarianism, find intentions completely irrelevant, whereas many deontological theories find intentions (mens rea) very important. Some theories, such as text-book rule utilitarianism, find the even application of exactly the right standards very important, whereas others, such as virtue ethics, find the continuous application of some standards at all much more important than theorizing about exactly what those standards should be or applying them exactly evenly each time. Yet other theories, such as contractarianism find important notions (such as real and hypothetical contracts and corresponding contracting conditions) simply absent in the table.

Of course, these descriptions of normative theories are only suggestive sketches. Questions of discrimination are more philosophically subtle than might be thought at first sight, and the typology is not intended as a vehicle for reasoning about the applied ethics of discrimination.Footnote 2 Our purpose here is make a contribution to the literature on rationality (rather than ethics), by applying the typology of Table 1 to the kind of automated decision-making that is becoming more common and more important in modern society by the day.

Finally, while the typology is intended to be a reasonable exegesis of Nozick, the definitions have first and foremost been chosen to serve as a fruitful stipulative terminology to facilitate the discussion in the following sections.

3 First-Level Bias in Automated Decision-Making

As mentioned above, first-level arbitrariness is never explicitly mentioned by Nozick. Its possibility is revealed only in a schema such as Table 1. However, we will now see that it is a very useful concept; succinctly explaining an important rationale for the concern about automated decision-making black boxes.

Consider the case of pulse oximetry, which has received some attention lately. Sjoding et al. (2020) recently showed that pulse oximeters overestimate the blood-oxygen saturation of black patients more often compared to white patients. Therefore, if pulse oximetry is used to triage patients, black patients may be subjected to an increased risk for hypoxemia. In our terminology, this is first-level arbitrariness: it is not that the oximeters are built to be biased, it is not that different oximeters are used for different groups of patients, but it is rather that applying the same oximeter, attempting to apply the same standard (blood-oxygen saturation, measured as a percentage), the machine nevertheless implements an uneven application of this standard.Footnote 3

Or consider some of the other cases of bias treated in the literature: systems for prediction of medical needs (Obermeyer et al., 2019), face recognition (Cavazos et al., 2020), court decisions (Dressel and Farid, 2018), speech recognition (Koenecke et al., 2020), and soap dispensers (Hankerson et al., 2016), which have all turned out to be biased to the disadvantage of some groups. It is not that different groups of people are intentionally sent to different systems. On the contrary, they are all sent to the same emergency rooms, cameras, courtrooms, microphones, and restrooms, and there is thus good prima facie reason to believe that the very same standards are being applied to all of them. Indeed, there is even some prima facie reason to believe that the algorithms underpinning these systems can apply standards more evenly than humans, since they are not subject to some well-known human weaknesses such as confirmation bias (see, e.g., Nickerson 1998), getting tired (see, e.g., Danziger et al., 2011; Timmons & Byrne 2019) being overconfident when in a position of power (see, e.g., Fast et al., 2012), being intimidated by social status (see, e.g., Foushee 1984), etc. But, as demonstrated in the literature, under the hood, these systems nevertheless implement uneven applications of seemingly identical standards. This is first-level arbitrariness, not first-level discrimination.Footnote 4

One of the most important concerns over the black boxes of automated decision-making—perhaps the most important—is that they may hide such first-level arbitrariness. Given cases such as those listed above, it is difficult to dispel the suspicion that similar biases may be found in other systems as well, though this has not not yet been discovered. An important reason why explainability,Footnote 5 while no panacea, is often held to be at least a prima facie moral requirement on automated decision-making systems is precisely the hope that explainability will expose such first-level arbitrariness. (Oftentimes, of course, it is more straightforward to check for first-level arbitrariness by just inspecting outcomes with respect to groups already known to be relevant, but an important reason for the appeal of explainability is its ability to identify problematic aspects not already known to be relevant, for instance Clever Hans-like use of confounding factors as discussed by Sturm 2013.)

The hoped-for role of explainability in exposing first-level arbitrariness is made clear when contrasting with the problem of first-level discrimination: if you are intentionally applying standards unevenly, then being told, by an explainability technique, that you are applying standards unevenly makes little difference. But if you are unintentionally applying standards unevenly, not even obliquely intending to do so, then being told so can be expected to make a big difference. At least as an approximation, we may say that the problem of first-level discrimination is one of desires, whereas the problem of first-level arbitrariness is one of beliefs. Thus, what explainability techniques (a kind of epistemic enhancers) can first and foremost hope to alleviate is not first-level bias in general, but rather its subset of first-level arbitrariness, in particular when this arbitrariness is genuinely inadvertent, and there is not even oblique intent.Footnote 6 Therefore, explicitly defining first-level arbitrariness is illuminating, explaining the black box concern and the quest for explainability more clearly.

4 Second-Level Bias in Automated Decision-Making

As described in the previous section, first-level bias is abundantly reported and discussed in the literature on automated decision-making. By contrast, second-level bias is more rarely mentioned. On the one hand, this is not surprising, since “[v]ery sophisticated observers sometimes can skillfully discuss the issue of first-level discrimination without noticing the attendant possibility of second-level discrimination” (Nozick 1993, p. 104). On the other hand, this is disconcerting, for precisely the same reason.

The problem is that whereas the existence of first-level bias can be investigated using straightforward statistical measures, second-level bias requires much more thorough and less straightforward investigation. Nozick gives the example of graduate admissions to the University of California, Berkeley, investigated statistically by Bickel et al. (1975).Footnote 7Bickel et al. found that even though a far lower percentage of female applicants were admitted overall, disaggregated data on departmental level suggested that admissions committees were quite fair: each department admitted approximately the same percentage of female applicants as of male applicants. There was no (first-level) discrimination. The explanation for the overall difference was rather that some departments admit a much lower percentage of (all) applicants than others, and women apply mostly to these departments. But what about second-level discrimination?

Women just happened to apply to departments with a smaller number of admission places per applicant. But did this “just happen”? Why aren’t the sizes of graduate departments proportional to the number of well-qualified applicants? […] Suppose, however, that some graduate departments are underfunded because they are more heavily populated by women graduate students. Suppose that for this reason certain subjects are not regarded as highly by university administrators or by the wider society as it allocates funding for graduate programs. […] Some departments are kept smaller than they otherwise would be because of their high percentage of women applicants; or other departments are kept larger because of their high percentage of men applicants. I am not claiming that this is so but simply describing a not completely implausible possibility under which the Berkeley statistics would fit a pattern of second-level discrimination. The statistics alone would not demonstrate this; that would require investigation of a structural question about the university’s organization, namely, why the various departments differ in the percentage of their total applicant pool that they are able to admit. Nozick (1993, pp. 104–105, emphasis in original)

The final point is crucial: statistics designed to detect first-level discrimination will not demonstrate second-level discrimination, or second-level bias more generally. This requires other methods (some of which may, of course, be statistical). Thus, we should not take the (relative) absence of reports about second-level bias in automated decision-making as convincing evidence that there is none. (That would be biased.) In fact, there are many reasons to believe that second-level arbitrariness (though not necessarily discrimination) is quite common in the design and use of automated decision-making:

Poor awareness :

First, the observation made above that second-level bias is not discussed in the literature on automated decision-making to the same extent as first-level bias means that practitioners are seldom aware of the problem, nor equipped to detect it.

Looking under the lamppost :

Data scientists developing new algorithms often need to try them out on large amounts of data. For example, it is no coincidence that image-recognition has taken off using databases of publicly available imagery from the Internet, such as ImageNet (Deng et al., 2009) even though such data sets may exhibit skewed distributions (see, e.g., Yang et al.2020). Thus, existing data sets are favored, and alternative ways of solving problems, involving data sets that do not yet exist or at least are more difficult to come by, are not considered.

Returns to scale :

Related to but distinct from the previous argument, machine-learning algorithms typically get better the more data they are trained on (see, e.g., Jordan and Mitchell 2015). Thus, in the choice between using a larger data set or a smaller data set to solve the same task, there will often be a tendency to select the larger one.

Path dependence :

Related to but distinct from the previous two arguments, there is path dependence in the development and use of technology (see, e.g., Rycroft & Kash 2002). Previous components and solutions get re-used (sometimes to the detriment of security, as discussed by Ji et al., 2018) whereas alternative ‘from scratch’ solutions do not get selected as often.

Judgment calls :

Automated decision-making systems are often used in situations where time will not tell whether any particular decision was indeed the right one, making it difficult to establish “objectively justified” standards. (It is this lack of appropriate feedback which is lamented by O’Neil (2016, p. 7) when she writes of her eponymous weapons of math destruction that “[t]hey define their own reality and use it to justify their results”.)

Lack of aggregate data :

Some parts of the world are plagued by poor statistics and lack of data, such as birth certificates and official proofs of identity, depriving poor people of many potential benefits (World Bank, 2021). Thus, data scientists applying algorithms to find solutions will not consider data sets that do not exist or at least are more difficult to come by. (This is a more applied version of the Looking under the lamppost argument, considering application rather than development of algorithms.)

Lack of detailed data :

Related to but distinct from the previous argument, the technologies that collect data (smart phones, internet-of-things sensors, etc.) are unevenly distributed between countries, groups, and individuals (see, e.g., Carcary et al., 2018; Berenguer et al., 2016). Thus, an algorithm suitable for solving a particular problem in one context may be unsuitable for solving the same problem in another context.

Procurement practice :

Whenever the user of a particular technology also pays for it upfront, market mechanisms provide a strong incentive to select standards that are “objectively justified” in the sense that they appeal to the market. For example, you will not yourself pay for a soap dispenser that fails to dispense soap onto a hand with the color of your own skin. However, if someone else procures the soap dispenser, e.g., for a bathroom in a public space, this market incentive gets lost. Unfortunately, in many cases such as medical equipment, technology is indeed procured by someone else (Economist, 2021).

As pointed out by an anonymous reviewer, many if not all of the examples listed above may be about habitual practices, rather than deliberated (or even deliberate) standards. Few would explicitly advocate a standard such as “existing data sets are best,” at least not without additional qualifications. However, the fact that standards are implicit and habitual does not mean that they are not standards. Nozick remarks that “[t]he Greeks knew that the dramas of Aeschylus and Sophocles were worthy of sustained attention before Aristotle wrote his Poetics to enunciate explicitly the standards that such dramas meet” Nozick (1993, p. 105). On this view, the existence and choice of a standard may predate its explicit articulation. Thus, even if standards are chosen only implicitly and out of habit—it “just happened” that this was the system developed, or this system worked well on the data that was abundant, or this was the system used last time, or this was a system the outcome of which cannot be objectively evaluated, or this system works well when appropriate data is available even though it is not always thus available, or this system was procured by someone who does not use it—then there is second-level arbitrariness (but not discrimination) in this choice of standards. Therefore, the examples listed belong to the right-hand column of Table 1.

However, the observation about habitual practices is valuable in that it shows that second-level arbitrariness can be had in two forms: as uneven implicit and habitual choices of standards or as uneven explicit choices of standards. In practice, as the list of examples above suggests, the former is probably common. The status of the latter is a more complicated problem, to which we will turn shortly, in Sections 5.2–5.4.

5 Alleviating Arbitrariness

Given the descriptions of first- and second-level bias in automated decision-making outlined in the previous two sections, it is natural to ask what can be done about it. The left-hand side of Table 1, the discrimination side, concerns intentional bias. Rectifying it—when it needs to be rectified—requires rectifying those intentions. In the rest of this section, we turn instead to the right-hand side of the table, the arbitrariness side.

5.1 Alleviating First-Level Arbitrariness

As mentioned in the introduction and exemplified in Section 3, first-level arbitrariness in automated decision-making—typically under the labels of bias or discrimination—has received much attention. Encouragingly, this has also led to the development of numerous technical tools to prevent and discover arbitrariness. Some examples from leading providers of cloud computing services include the Amazon SageMaker Clarify,Footnote 8 the AI Fairness 360 toolkit from IBM,Footnote 9 the Fairness Indicators library in Google’s TensorFlow,Footnote 10 and the Responsible ML toolkits in Microsoft Azure.Footnote 11 What such tools do—broadly speaking—is that they allow developers to automatically and regularly apply statistical tests of first-level bias to their systems, to inspect models in order to explain their decisions and inner workings, and offer suggestions on how to rectify any unwanted behavior detected. (Note how the fact that these tools are made for the developers rather than users of systems again reinforces the point made in Section 3 that such tools are built to alleviate first-level arbitrariness rather than first-level discrimination.)

There is no reason to downplay the importance and the promise of such techniques. Making sophisticated statistical tools automatically available to developers using major machine learning platforms makes it vastly easier to detect and prevent unwanted arbitrariness in the systems developed. This is indeed good news.

However, there is also no reason to overstate what can be accomplished by such tools. What these tools do is to check whether the system being tested conforms to existing standards, i.e., whether there is first-level bias. They cannot check whether those existing standards were chosen for good reasons, or whether they should be modified, amended, or discarded, i.e., whether there is second-level bias.Footnote 12 In thus delineating the limits of current automated tools,Footnote 13 we note again how illuminating the distinction between first- and second-level bias is. This leads us to the problem of alleviating second-level arbitrariness.

5.2 Alleviating Second-Level Arbitrariness

The problem of choosing standards is at the core of the research area most often referred to as fairness in AI, ML, or algorithms. Encouragingly, this is an area which has grown rapidly in the past few years and contributed much needed discussions about which formal evaluation metrics should be used and about diversity in the design process. This is not the place to review this subject extensively—a recent review is given by Chouldechova and Roth (2020).

However, in the context of second-level bias, the following observation is very relevant: there are several statistical notions of fairness (most often involving parity of some statistical measure over protected demographic groups), and not only do they not necessarily entail each other—they can actually be at odds. More precisely, a fundamental impossibility result (reminiscent of the famous impossibility theorem given by Arrow 1951) has been proven: (i) false positive rates, (ii) false negative rates, and (iii) positive predictive value can only be equalized across protected groups in trivial settings (Chouldechova, 2017; Kleinberg et al., 2017). In practice they are at odds with each other.

The issue of conflicting standards also sheds some additional light on the question of oblique intent, first raised in Section 2.1. Say that you chose a standard in which you equalize false positive rates and false negative rates across protected groups in a particular case. Then, by the impossibility result, you will also fail to equalize positive predictive value across those same groups, and by the informal definition given in Section 2.1, you thus obliquely intend to have non-equal positive predictive values for the groups. A first observandum is that this is not a far-fetched situation. Since there are many candidates for statistical measures to equalize and many candidates for protected groups, some version of this situation is almost always the case. A second observandum is that a naïve statistical test may flag this as bias; first-level if seen as an uneven application of an (impossible) standard requiring all three measures to be equalized across the groups,Footnote 14 second-level if seen as an uneven choice of which measures to equalize across groups. (A more sophisticated statistical investigation would instead check for equality over several parallel statistical measures, illuminating their interrelationships and possible trade-offs in the spirit of Kleinberg et al.) A third observandum is that despite oblique intent, this is second-level discrimination only if your forced choice is “deviously” made in order to exclude certain cases. Otherwise, it is second-level arbitrariness, reflecting the fundamental constraint that you have to choose unevenly, failing to equalize (at least) one of the three measures across the groups.

However, the problem of choosing standards does not stop there. Someone might reject statistical definitions of fairness in favor of individual definitions of fairness, for example claiming that statistical fairness amounts to some people being treated merely as a means to achieving an end at the group level, not their own ends, inspired by Kant (1948 [1785], 4:433). Or someone might accept (several notions of) fairness as prima facie goods to be traded-off against each other and against other goods such a public safety (e.g., as discussed in Corbett-Davies et al., 2017), inspired by consequentialist theories. Then again, someone might argue that it is not permissible to make trade-offs with (some of) these particular goods; that they should instead be lexically ordered (Rawls 1999, pp. 36–40) or treated as side-constraints (Nozick 1974, pp. 30–33).Footnote 15 Or someone might, inspired by Rawls (1999, pp. 118–123), offer a conception of algorithmic fairness as a deliberative result best produced behind the famous “veil of ignorance” (a concrete such proposal is given by Heidari et al.2018 and a more general discussion about the applicability of Rawls to algorithmic fairness is given by Franke 2021). The list of alternatives goes on. And of course, each of these alternatives is vulnerable to all the criticisms raised against the corresponding normative and political theories over the years.

This observation about contrasting and sometimes conflicting standards is not new—Binns (2018b) notes that “attempts to formalise “fairness” in machine learning contain echoes of these old philosophical debates,” and then goes on to identify relevant lessons for fairness in automated decision-making from political philosophy. Such lessons can fruitfully inform suggestions about appropriate normative theories of fairness in automated decision-making (see, e.g., Binns 2018a; Wong 2019, for some recent examples). In this context, we can again observe that the distinction between first- and second-level bias is illuminating. For example, Wong (2019) argues that researchers have taken algorithmic bias seriously but primarily conceptualized it as a technical task, while it should rather, first and foremost, be conceptualized as a political question. Now, in light of the distinction between first- and second-level bias, it seems illuminating to understand Wong’s claim as follows: addressing first-level bias is indeed mostly a technical task (as discussed above, even application of existing standards can be automatically checked using statistical tests incorporated into coding environments), whereas second-level bias is mostly a political question (as discussed above, choosing which standards to apply is a normative question, one that reasonably should be the subject of public deliberation in some form).

However, it is not our purpose here to make a contribution to this normative theory of fairness in automated decision-making—the normative theory of choosing standards. Instead, our question is: In the face of all these contrasting and sometimes conflicting standards, can standards nevertheless be chosen in an unbiased, or at least less biased, manner? Nozick spells the problem out:

There are many possible kinds of reasons for and against any belief—certainly any one concerning a controversial social or normative question—and there are many possible standards for evaluating such reasons. No one seeks out all possible reasons impartially and gives them equal weight—including those of us who make some effort to examine countervailing reasons—and the reasons that anyone accepts are not a random sampling of all the possible reasons. (Nozick 1993, pp. 105–106)

These observations may appear bleak: acknowledging first that our choices of standards in automated decision-making are influenced by unintentional factors such as those listed in Section 4 and then that even when we make intentional efforts to choose standards for good reasons, we are left with some second-level arbitrariness—uneven choice of standards—whatever we do. But at least this needs not be irrational. There are good reasons for wanting to equalize false positive rates, false negative rates, and positive predictive value across protected groups when developing decision-support systems. There are also good reasons for adopting a Kantian perspective on fairness in automated decision-making, just as there are good reasons for adopting rights-based, consequentialist, Rawlsian, etc. perspectives:

Each may be right: what each has noticed is a reason for their respective (opposed) conclusions. Each person also may be rational: believing things for reasons, reaching conclusions on the basis of evidence, invoking criteria for belief and evaluation that have much to be said for them. Yet each person is believing things for only some of the reasons, reaching conclusions on the basis of only some of the evidence, invoking only some of the criteria that have much to be said for them. The social factors (studied by the sociologists) act to narrow the range of possible relevant considerations that get considered. Within this range, our beliefs and actions may be rational. Even when not made relative to this range, they are not completely irrational—given that there is some reason for them, they are at least prima facie rational.” (Nozick 1993, p. 106, emphasis in original)

5.3 Piecemeal Alleviation of Second-Level Arbitrariness

We cannot eliminate second-level arbitrariness completely. But it does not follow that we cannot, and should not, alleviate it a bit, in a piecemeal fashion. As remarked in Section 2, arbitrariness is a matter of degree, and it can be decreased. In particular, three strategies suggest themselves: First, articulating and scrutinizing our choices of standards allows us to find reasons for and against them. Encouragingly, this is precisely what is currently unfolding in the literature on algorithmic fairness—the development of normative theories of choosing standards referred to above. Engaging in such activities allows us to become (more) rational in our choices and, plausibly, to avoid at least some of the worst mistakes. (Though some—especially consequentialist—normative theories may hold that it is better if you make the right choice for the wrong reasons than if you make the wrong choice for seemingly good reasons.)

Second, creating design situations with more diversity, whether by using particular methods to encourage lateral thinking or by recruiting people of greater diversity (with respect to competencies, perspectives, backgrounds, professions, ethical beliefs, etc.) thus broadening “the range of relevant considerations that get considered.” This is indeed a strategy that is generally acknowledged in the literature in various forms (some examples, far from exhaustive, include Friedman et al., 2013; Livermore 2016; Holstein et al., 2019; Kuhlman et al., 2020; Dexe et al.2020).

Third, ensuring proper competition and consumer choice. As noted by The Economist (2021), “[i]n a well ordered market, competition should introduce diversity quite fast. In the past, women and non-white people may have lacked purchasing power, but that is surely no longer so.” The problem is rather, as observed in Section 4, that in many markets the end-user is not the paying customer. In these areas, competition authorities may need to take action.Footnote 16

The second and the third strategies hint at the interplay between alleviating first- and second-arbitrariness: introducing a greater range of perspectives—including those of consumers who can vote with their feet—can be expected both to expose uneven application of existing standards and to expose designers to a wider range of possible standards to choose from.

Clearly, however, none of these strategies will remove bias altogether. At best we can asymptotically approach a completely unbiased state of affairs. This leads us to a few concluding remarks on this limit.

5.4 The Limits of Alleviation of Second-Level Arbitrariness

The question of how to choose normative standards is in no way limited to the context of automated decision-making. On the contrary, it is one of the big philosophical questions. Nagel, in The View from Nowhere, remarks:

In general, the problem of how to combine the enormous and disparate wealth of reasons that practical objectivity generates, together with the subjective reasons that remain, by a method that will allow us to act and choose in the world, is dauntingly difficult (Nagel 1986, p. 188)

This daunting difficultyFootnote 17 applies to the choice of normative standards in automated decision-making no less than it applies to similar choices in other contexts, and it is cause for pessimism. But Nagel still sees cause for optimism—belief in the possibility of moral progress. It is true that we should be skeptical about our current moral intuitions and beliefs, because they are influenced by “our starting points, the social influences pressing on us, and the confusion of our thought”—the familiar biases which are our concern in this article—and that “we are at a primitive stage of moral development” (Nagel1986, pp. 185–186). But, argues Nagel, it is also true that moral truth “could not be radically inaccessible in the way that the truth about the physical world might be” (p. 186), because it has to be suited to govern our daily conduct. Normative theories for humanity cannot be arbitrarily difficult to understand, because if we cannot understand them, then we cannot accept and internalize them.

This cause for optimism is our first lesson from Nagel. Even piecemeal alleviation of arbitrariness, as outlined above in the context of automated decision-making, is meaningful, and furthermore, our belief in this meaning is itself meaningful: “the idea of the possibility of moral progress is an essential condition of moral progress” (Nagel 1986, p. 186).

The second lesson from Nagel concerns the limits of the pursuit of objectivity as a method for moral progress. We remarked in Section 2 that the typology in Table 1 is not intended to delineate acts which are morally right from acts which are morally wrong under any particular normative theory. Specifically, it is not the case the removing all arbitrariness (or, more generally, all bias) is a necessary or sufficient condition for acting morally right. Nagel, while clearly advocating the pursuit of objectivity—we might call it the pursuit of being unbiased—reminds us of the limits of this pursuit:

This does not mean that greater detachment always takes us closer to the truth. Sometimes, to be sure, objectivity will lead us to regard our original inclinations as mistaken, and then we will try to replace them or bracket them as ineliminable but illusory. But it would be a mistake to try to eliminate perspective from our conception of ethics entirely—as much of a mistake as it would be to try to eliminate perspective from the universe. This itself must be objectively recognized. (Nagel 1986, p. 187)

Thus, it may be that the unreachable goal of eliminating all arbitrariness is anyway not the right goal—but that the piecemeal alleviation of arbitrariness is nevertheless meaningful.Footnote 18

These reflections on the limits of alleviation of arbitrariness are offered only tentatively. Surely, they are not the last word, but hopefully they are a meaningful contribution to an interesting discussion.

6 Conclusions

This article has elaborated two distinctions first made by Nozick (1993, pp. 103–106); (i) between first-level bias and second-level bias and (ii) between discrimination and arbitrariness, respectively. Applying these in the context of automated decision-making, a number of illuminating observations are made.

More precisely, surveying the empirical literature, we first observe that some reported bias in automated decision-making is, in our terminology, first-level arbitrariness. Distinguishing first-level arbitrariness from first-level discrimination is illuminating in that it shows more clearly why explainability is a prima facie good in systems for automated decision-making: explainability has a great potential to alleviate first-level arbitrariness, whereas it has only a limited potential to alleviate first-level discrimination.

We then apply the concept of second-level bias to automated decision-making and reveal that second-level arbitrariness is probably quite common—whereas proving second-level discrimination is a much more difficult empirical endeavor. Unfortunately, and in contrast to first-level arbitrariness, it is not always straightforward to detect second-level arbitrariness by, e.g., statistical tests which are often designed with first-level arbitrariness in mind.

In the final part of the article, we discuss the prospects for alleviating arbitrariness and argue that whereas automated tools can be powerful detectors of first-level arbitrariness, detecting and alleviating second-level arbitrariness is a more profound problem. More precisely, there are many contrasting and sometimes conflicting standards and even when we make intentional efforts to choose standards for good reasons, some second-level arbitrariness—uneven choice of standards—remains. However, we argue—at least tentatively—that this need not be irrational and that it is still meaningful to alleviate second-level arbitrariness in the piecemeal fashion which is available to us.