Above, we looked at two cases in which the people involved aim to contribute to a societal or common good and work carefully. They followed principles like respect for human autonomy, prevention of harm, fairness and explicability [21]. This can be challenging and complex enough. Below, we will explore one further level of complexity, namely potential unintended, undesirable, higher-order effects of using algorithms.
Such effects can occur, regardless of good intentions and a careful approach. That is why we call them ‘unintended, undesirable’. We propose that such an exploration is needed if we take Responsible Innovation (RI) seriously. Stilgoe et al. [40] argued that RI entails four key dimensions: anticipation, responsiveness, inclusion and reflexivity. We will focus on the first two: anticipation and responsiveness.Footnote 24 ‘Anticipation’, they write, ‘prompts researchers and organisations to ask ‘what if...?’ questions …, to consider contingency, … [it] involves systematic thinking aimed at increasing resilience, while revealing new opportunities for innovation’ (op. cit.: 1570). Anticipation involves exploration and speculation, envisioning various potential scenarios of what might happen. Not only of problems, by the way, also of opportunities. Regarding responsiveness, Stilgoe et al. comment that it refers to ‘a capacity to change shape or direction in response to stakeholder and public values and changing circumstances’ (op. cit.: 1572). Responsiveness is a necessary supplement to anticipation; one needs to be able to respond to issues that one anticipates. In addition, responsiveness requires anticipating changing circumstances to be able to respond adequately.
For our exploration of potential unintended, undesirable, higher-order effects, we will stay in the domain of algorithms that are used for decision support in the domain of justice and security. Let us further assume that the people involved in the design and deployment of these algorithms have good intentions and work carefully. They will typically focus on intended and desirable effects that the algorithm can help to realize. They will also make efforts to anticipate unintended, undesirable, first-order effects that can happen, e.g., when different values conflict in a relatively direct manner. When it is obvious that a lack of transparency can negatively impact fairness, people will work to improve transparency, to promote fairness.
They will, however, by definition, find it hard to anticipate unintended, undesirable, higher-order effects.
Now, what do we mean with higher-order effects? One way to understand these comes from systems thinking. In systems thinking one views different phenomena as parts of a larger system and looks at the relationships between these phenomena [27]. The feedback loop is a key concept here: imagine that A influences B, then information about the status of B regulates the influence of A on B. Feedback loops can be balancing; they steer parts of a system to some dynamic equilibrium. Think of the balancing feedback of a thermostat in a heating system. Or they can be reinforcing; they make parts of a system go increasingly up or increasingly down. They are sometimes referred to as virtuous cycles or viscous cycles. We are often able, to some extent, to anticipate first-order effects, like the influence of A on B. We are, however, much less able to anticipate higher-order effects, like the behaviour of a system that has multiple elements, multiple relationships and multiple balancing and reinforcing feedback loops.
We can illustrate what we mean with unintended, undesirable, higher-order effects with the anecdote of the Cobra effect.Footnote 25 In the time of the British rule of colonial India, the British wanted to get rid of venomous cobras in Delhi and offered a bounty for every dead cobra. The (first-order) intended and desirable effect was that people killed snakes for this reward. People, however, also began to breed cobras to claim their rewards. The government found out and stopped the reward program. The breeders then let their cobras go free, which were now worthless—which worsened the cobra plague. Officials could maybe have anticipated this effect if their analysis of the system had included variables and feedback loops for supply and demand, and for motivation and behaviour.
Now, if we envision a situation in which people design an algorithm. They focus on values A and B, and carefully combine and balance A and B. But then, after the system has been in use for a while, value C pops up, in a very disturbing way, seemingly out of nowhere. Or they create a careful balance between values P and Q, and then, as people use the system in ways slightly different from what they had intended, value Q goes off the rails, unexpectedly, and the balance between P and Q is gone.
A similar effect is known as Goodhart’s law,Footnote 26 named after economist Charles Goodhart: ‘When a measure becomes a target, it ceases to be a good measure.’ Informally, this is known as the KPI effect: when an organization introduces a Key Performance Indicator, the people on the floor find (creative) ways to satisfy this KPI—sometimes, however, in ways that hamper what the KPI tries to achieve. The organization meets its KPIs, but fails to realize the underlying goals. This draws attention to the need to be very careful when articulating, quantifying and measuring the intended, desirable outcomes one wishes to achieve.
Below, we will explore several potential unintended, undesirable, higher-order effects that may occur in the design and deployment of algorithms for decision support in the domain of justice and security. Our exploration is based on several general findings from the cases discussed above. (Please note, however, that our exploration is not intended as an assessment of what might happen in these particular cases.) We will, again, follow the principles of the High-Level Expert Group on Artificial Intelligence [21].
Respect for human autonomy
The deployment of algorithms can alter people’s ways of working and thus affect their autonomy. In the introduction we referred to algorithms in decision support, where agents need to combine the algorithm’s output with their professional discretion. The extremes of always strictly or slavishly following the algorithm and of always neglecting or overruling its output are rather ineffective or inefficient.
We may be able to anticipate unintended, undesirable, first-order effects and respond appropriately, e.g., by enabling people to inspect, question, modify or correct its functioning. But how may we anticipate and respond to higher-order effects? Shneiderman [38] provided a diagram that may be helpful—see Fig. 1. He advised exploring the top-right quadrant of high computer automation and high human control to create ‘reliable, safe and trustworthy’ AI systems (although there may be good reasons, in specific cases, to go for other quadrants: for example the combination of high human control and low computer automation for piano playing (‘human mastery’) or the combination of high computer automation and low human control for airbags (‘computer control’).
One way to explore potential, unintended, undesirable, higher-order effects, is to explore various ways in which the system and its usage may, over time, unintentionally, move across the plane in Fig. 1, away from the top-right quadrant. It may drift to excessive human control, where people need to micro-manage the system, which could be very inefficient, or even dangerous, e.g., in a situation in which a self-driving car very suddenly requires the driver, who is busy doing something else than driving, to take control of the steering wheel. Or towards excessive automation, where too many tasks are delegated to the system, so that people can no longer monitor its functioning in any meaningful way, or the system performs tasks that do require human perception, discretion and judgement. Computers are notoriously bad at the latter; they cannot take into account context and they lack common sense [26, 35].
Or the system may drift away towards too little human control; this can also happen because of people’s evolving practices, e.g., when people have learned to follow the algorithm’s output unthinkingly and routinely ‘click the okay button’. Or it may drift towards too little computer automation; people then need to perform too many routine tasks and effectively waste their time and energy—or worse, start to make mistakes, e.g., because of reduced concentration.
Another issue regarding respect for human autonomy is the combination of explicit knowledge and tacit knowledge. The former refers to data that is used as input for an algorithm and is associated with computer automation. The latter refers to information in people’s minds and bodies, which is associated with human control. Both types of knowledge are relevant for algorithms in decision support. Imagine an organization that procures such a system. They will typically want to realize benefits that outweigh the costs associated with using the system. Over time, they may unintentionally slide towards preferring explicit knowledge and computer automation over tacit knowledge and human control—it would be silly to buy an expensive system and not use it. There is ample (anecdotal) evidence of people feeling unhappy when their tacit knowledge, their abilities, skills, expertise, experience, are not valued and replaced by automation. This may even lead to an unintentional, undesirable focus on means, and losing sight of ends. Choosing for automation brings risks for the unintended and undesirable effect—over the course time—of prioritizing explicit knowledge and computer automation at the expense of tacit knowledge and human control.
Prevention of harm
A first step in anticipating and preventing potential harms involves assessing and evaluating the different pros and cons of using an algorithm (a future situation) in comparison to not using an algorithm (the current situation). It is indeed possible that not using the algorithm causes more harm than using it. This would be an argument in favour of deploying this algorithm. In addition, one would need to design and deploy measures to increase its benefits and to decrease its drawbacks.
Another way to anticipate and prevent potential harms of algorithms, is to create an error matrix. Such a matrix plots true positives, true negatives, false positives and false negatives. These errors can be viewed as first-order unintended, undesirable effects. One way to anticipate higher-order unintended, undesirable effects is to explore how this error matrix may evolve, as it is modified and fine-tuned over the course of time, either by people or ‘by itself’, in cases of machine learning.
For example, there may be unintentional incentives to promote the occurrence of false negatives. If we look back at Case A, this would refer to a person not being offered support to pay their fines, whereas this person would actually need such support. For the sake of argument, let us assume, in more general terms, that false negatives refer to advice that ‘no action’ is needed, which typically costs less money and time than action. Moreover, false negatives are likely to stay undetected; they typically do not appear in weekly, quarterly or yearly reports. This may unintentionally nudge the organization, over time, to modifying the system towards producing more false negatives—which may hamper the organization’s overall goals.
Alternatively, a system may unintentionally evolve, over time, towards producing more false positives. If we look back at Case B, this would refer to a person incorrectly receiving a high likelihood of violent behaviour, and an advice to act cautiously and carefully. For the sake of argument, let us assume, in more general terms, that false positives entail taking action, which costs money and time. Organizations typically steer away from costs and, unintentionally and over time, may modify the system towards producing less false positives. Having fewer errors does not need to be problematic, of course. It can, however, be a problem if the modified algorithm’s reduction of false positives leads to more false negatives. In Case B, this would refer to predictions of non-violent behaviour for people who will actually behave violently—an unintended and undesirable effect.
There remain questions regarding a fair balance between having false negatives, which may hamper the organization’s main goals, or false positives, which may involve wasting money and time, and harms.
One particular example of an unintended, undesirable, higher-order effect could be a drift towards ‘low hanging fruits’. An organization may gain insights in ‘what works best’ and drifts towards prioritizing cases that are very clearly true positive. In Case A, this would refer to people who are very willing and very able to pay their fines. One might say that they do not really need to be offered support. The organization, however, can be very successful if it targets them. Such a priority for ‘low hanging fruits’ may lead to a neglect of people who are less clearly true positive, who will miss out on the support they actually need.
Another higher-order, unintended harm can manifest when organizations collect and use data from multiple data sources, especially if these data sources pertain to different domains. Imagine an insurance company collecting data about their customers’ life styles. Or a care provider collecting data on their patients’ finances. Combining data from different sources is not necessarily always a bad idea. There are situations in which the public expects that different public service organizations collaborate and share information—of course following principles of legality and proportionality. The public will criticize the organizations involved if not collaborating and not sharing information resulted in harm that could have been prevented precisely by collaborating and sharing information.
Fairness
It almost goes without saying that we expect government organizations to comply to legislation and to ensure substantive fairness, an ‘equal and just distribution of both benefits and costs’ and procedural fairness, which refers to people’s abilities ‘to contest and seek effective redress’ [21], p 12). Making the algorithm substantively fair is necessary but not sufficient. The organizations involved will also need to organize procedural fairness, e.g., by organizing processes via which people at the receiving end of the decisions supported by the algorithm, are able to critique these decisions. We will further discuss this topic below, under explicability.
Our discussion of harms (above) focused on individuals. Harms can, however, also affect groups of people. In such cases, we can view these harms as systemic unfairness. There are ample examples of unfairness being repeated, propagated or exacerbated through the usage of algorithms [32]. Imagine an algorithm that puts a specific label on a relatively large number of people in a specific socio-economic or cultural group, then this may lead to stigmatization or discrimination of that group. There is a risk that people are reduced to labels and that the labels get reified. In very general terms, we can point at four sources for such unfairness:
-
the data that the algorithm uses as input—these data may refer to current unfair situations; when these data are used to train an algorithm, these unfair situations are likely to be repeated;
-
the algorithm itself, which can function unfairly—intentionally or unintentionally;
-
the process in which the algorithm’s output is used—which affects procedural fairness;
-
or the feedback loop, which feeds back information about the application of the algorithm’s output, so that the algorithm, or processes around it, can be corrected and modified.
This feedback loop is critical. ‘Without feedback’, Cathy O’Neil argued, ‘a statistical engine can continue spinning out faulty and damaging analysis while never learning from its mistakes’ (2016: p. 7). She stressed the need for properly functioning feedback loops, otherwise we risk ‘confusing [algorithms’] findings with on-the-ground reality’ (ibid: p. 12).
Finally, we must look at the larger picture. Promoting fairness in the design and deployment of algorithms must go hand in hand with questioning and critiquing the larger context in which these algorithms are used [3,3,4,4,5]. For example, in the infamous COMPAS case,Footnote 27 of an algorithm that assesses the likelihood of recidivism, one needs to make the algorithm more fair (or less unfair), but also make room to question and critique the role of racial discrimination in the judiciary system, and the larger systemic, racial inequalities and injustices in society.
Explicability
Explicability can be understood as having instrumental value in that it contributes to other values or principles. According to Hayes et al. [20], explicability (or in their words: ‘accountability/transparency’) contributes to autonomy and to fairness. Autonomy, of both those who use the algorithm (‘human decision makers’) and those at the receiving end (‘data subjects’), critically depends on their abilities to understand and explain the algorithm’s functioning. Moreover, explicability is critical for people’s abilities to question and critique the algorithm’s fairness: to find an appropriate balance of agency between people and technology; to inspect and evaluate the various types of errors; and to organize processes via which people can critique and provide pushback, and seek correction and redress. There are several domains of knowledge dedicated to promoting explicability, e.g., XAI (Explainable AI) and FAT (Fairness, Accountability and Transparency; which focuses on more than explicability)—a discussion of which is outside our scope.
Two issues, which we also encountered in our cases, are, however, worth mentioning. First, there is the ‘problem of many hands’ [8]. This refers to the problem that in a complex system, with many actors and many moving parts, it can be hard to attribute responsibility. If we want to explore unintended, undesirable, higher-order effects, we need to look at the larger processes in which algorithms are used, at organizations that use the algorithms. A decision based partially on an algorithm’s output can only be understood and explained if the processes and organization are understandable and explainable. One will need to avoid situations where citizens’ questions get a reply like: ‘Computer says no. I don’t know why. You will need to go elsewhere. I don’t know where.’
Second, there are different types of algorithms with different properties and levels of explicability. In the two cases, we saw that the people involved chose for a simple decision-tree rather than a complex deep-learning (Case A), and for using ‘expert knowledge’ rather than ‘data mining’ (Case B); in both cases, they chose the former because it typically provides better explicability than the latter. Looking forward, it would be wise to keep an eye on developments in FAT and XAI; these fields may provide solutions that combine autonomy, fairness, accuracy and privacy. Organizations need to explore, innovate, experiment and learn.