The tragedy of the AI commons

Abstract

Policy and guideline proposals for ethical artificial intelligence research have proliferated in recent years. These are supposed to guide the socially-responsible development of AI for a common good. However, there typically exist incentives for non-cooperation (i.e., non-adherence to such policies and guidelines); and, these proposals often lack effective mechanisms to enforce their own normative claims. The situation just described constitutes a social dilemma—namely, a situation where no one has an individual incentive to cooperate, though mutual cooperation would lead to the best outcome for all involved. In this paper, we use stochastic evolutionary game dynamics to model this social dilemma in the context of the ethical development of artificial intelligence. This formalism allows us to isolate variables that may be intervened upon, thus providing actionable suggestions for increased cooperation amongst numerous stakeholders in AI. Our results show how stochastic effects can help make cooperation viable in such a scenario. They suggest that coordination for a common good should be attempted in smaller groups in which the cost of cooperation is low, and the perceived risk of failure is high. This provides insight into the conditions under which we should expect such ethics proposals to be successful with regard to their scope, scale, and content.

Appendices

Appendix A: Game theory

In this brief appendix, we provide some further game-theoretic background than we had space to discuss in Sect. 3. For more comprehensive introductions to game theory, see, e.g., (Aumann & Hart, 1992, 1994, 2002; Maynard Smith, 1982; Neumann & Morgenstern, 1944; Weibull, 1997; Young & Zamir, 2014).

1.1 Game-theoretic analysis of cooperation and conflict

Cooperative behaviour persists in human and non-human animal populations alike, but it provides something of an evolutionary puzzle Axelrod and Hamilton (1981); Darwin (1871); Hauert et al. (2006); Moyano and Sánchez (2009); Nowak (2012); Nowak et al. (2004); Taylor et al. (2004): How can cooperation be maintained despite incentives for non-cooperative behaviour (i.e., defection)? Evolutionary game theory provides useful tools for analysing the evolution of cooperative behaviour quantitatively in both human and non-human animals.²²

Game theory can be used to study the ways in which independent choices between actors interact to produce outcomes.²³

In game theory, a game is determined by the payoffs. For example, the payoff matrix for a generic, $2\times 2$, symmetric, normal form game is displayed in Fig. 5.

Fig. 5

Payoff matrix for a generic, $2\times 2$, symmetric, normal form game

Each actor (Player 1 and Player 2) in this example can choose one of two strategies, C or D. The payoffs to each of the players are given by the respective entries in each cell—i.e., the first number in the top-right cell (b) is the payoff afforded to Player 1 when she plays C and her partner plays D; the second number (c) is the payoff afforded to Player 2 in the same situation (i.e., when Player 2 plays D and Player 1 plays C).

As discussed in the paper, social dilemmas are games where (i) the payoff to each individual for non-cooperative behaviour is higher than the payoff for cooperative behaviour, and (ii) every individual receives a lower payoff when everyone defects than they would have, had everyone cooperated (Dawes, 1980).

When $c> a> d > b$, in Fig. 5, we have a Prisoner’s Dilemma.²⁴ Note that when both actors cooperate (i.e., both play C), their payoff is higher than if they both defect ($a > d$), thus satisfying criterion (ii) mentioned above. However, each actor has an individual incentive to defect (i.e., play D) regardless of what the other actor does; Player 1 would prefer to defect when Player 2 cooperates ($c > a$), and she would prefer to defect when Player 2 defects ($d > b$)—and mutatis mutandis for Player 2. This satisfies criterion (i) above.

In this case, we say that defect is a strictly dominant strategy for each player, which leads to the unique Nash equilibrium: $\langle D , D \rangle$—that is, a combination of strategies where no actor can increase her payoff by unilateral deviation from her strategy. The ‘dilemma’ is that mutual cooperation yields a better outcome for all parties than mutual defection, but, from an individual perspective, it is never rational to cooperate.

1.2 Evolutionary game dynamics

In an evolutionary context, the payoffs are identified with reproductive fitness, so that more-successful strategies are more likely to propagate, reproduce, be replicated, be imitated, etc. This provides a natural way to incorporate dynamics to the underlying game.

There are two natural interpretations of evolutionary game dynamics. The first is biological, where strategies are encoded in the genome of individuals, and those who are successful pass on their genes at higher rates; the second is cultural, where successful behaviours are reproduced through learning and imitation. We are primarily concerned with processes of cultural evolution. This process should be familiar to those in AI/ML who work on multi-agent reinforcement learning (MARL) (Littman, 1994; Shapley, 1953; Zhang et al., 2019).

In addition to the game, an evolutionary model requires a specification of the dynamics—namely, a set of rules for determining how the strategies of actors in a population will update (under a cultural interpretation), or how the proportions of strategies being played in the population will shift as they proliferate or are driven to extinction (under a biological interpretation). Evolutionary game dynamics are often studied in infinite populations using deterministic differential equations. For example, the replicator dynamic (Taylor & Jonker, 1978) captures how strategies with higher-than-average fitness tend to increase, and strategies with lower-than-average fitness tend to decrease. A population state is evolutionarily stable only if it is an asymptotically stable rest point of the dynamics (Maynard Smith, 1982).

1.3 Stochastic game dynamics

In finite populations, stochastic game dynamics are used to study the selection of traits with frequency-dependent fitness (Liu et al., 2011; Ohtsuki et al., 2007; Sigmund, 2010; Szabo et al., 2009; Szolnoki et al., 2009; Traulsen et al., 2006a).

A standard stochastic game dynamics that is used extensively is the Moran process. This is a simple birth-death process where an individual is chosen proportional to their fitness and replaces a randomly chosen individual with an offspring of its own type (Altrock & Traulsen, 2009; Claussen & Traulsen, 2005; Huang & Traulsen, 2010; Liu et al., 2017, 2015; Taylor et al., 2006; Traulsen et al., 2007; Wu et al., 2010, 2015).

In the standard Fermi process, which we discuss in Sect. 3, an individual is chosen randomly from a finite population, and its reproductive success is evaluated by comparing its payoff to a second, randomly-selected individual from the population (Liu et al., 2017; Traulsen et al., 2006b, 2007).

As mentioned in Sect. 3, the pairwise comparison of payoffs of the focal individual and the role model informs the probability, p, that the focal individual copies the strategy of the role model; the probability function, called the Fermi function, was presented in Eq. 3, and repeated here for convenience:

$$\begin{aligned} p = \left[ 1 + e^{\lambda (\pi _{f} - \pi _{r})} \right] ^{-1}. \end{aligned}$$

Again, if both individuals have the same payoff, the focal individual randomises between the two strategies. Note, then, that the focal individual does not always switch to a better strategy; the individual may switch to one that is strictly worse.

When the intensity of selection $\lambda =0$, selection is random; when selection is weak ($\lambda \ll 1$), p reduces to a linear function of the payoff difference; when $\lambda = 1$, our model gives us back the replicator dynamic; and, when $\lambda \rightarrow \infty$, we get the best-response dynamic (Fudenberg & Tirole, 1991).

Evolutionary game dynamics have been used to shed light upon many aspects of human behaviour, including altruism,²⁵ moral behaviour,²⁶empathy,²⁷ social learning,²⁸ social norms,²⁹ and the evolution of communication, proto-language, and compositional syntax,³⁰ among many others. See (Ross, 2019) for further details.

Appendix B: Technical details

In this brief appendix, we provide some further formal details for our model than we had space to discuss in Sect. 3.

1.1 Mean payoffs

Recall that the payoffs to each cooperators, C, and defectors, D, in a group of size N are given as a function of the number of cooperators in that group, $n_C$, as follows:

$$\begin{aligned} \pi _{C} (n_C)&= b \cdot \Theta (n_{C} - n^{*}) + b \cdot (1 - rm) \cdot \Theta (n_{C} - n^{*}) - cb, \\ \pi _{D} (n_C)&= \pi _C (n_C) + cb, \end{aligned}$$

where $\Theta$ is the Heaviside step function. The mean payoffs to each type in a population of size Z, where groups are determined by random mixing, is then given as a function of the total fraction of cooperators in the population, $x_{C}=n_C^Z/Z$, as follows:

$$\begin{aligned} \Pi _{C} (x_{C})&= \sum _{n_{C}=0}^{N} \frac{n_{C}}{N} \left( {\begin{array}{c}N\\ n_{C}\end{array}}\right) x_{C}^{n_C} (1 - x_{C})^{N - n_{C}} \pi _{C}(n_C), \\ \Pi _{D} (x_{C})&= \sum _{n_C=0}^{N} \frac{N-n_C}{N} \left( {\begin{array}{c}N\\ n_{C}\end{array}}\right) x_{C}^{n_C} (1-x_{C})^{N-n_{C}}\pi _{D}(n_C). \end{aligned}$$

1.2 Fermi dynamics

The Fermi dynamics uses the average payoffs to each type to determine the probability that a randomly-chosen individual from the population will imitate the strategy of a second randomly-chosen individual from the population. Such a change will produce one of three outcomes: the number of cooperators in the population, $k=n_C^Z$, will increase, decrease, or remain the same. This is captured by the following transition probabilities, which yield a tri-diagonal transition matrix, T, for our birth-death process:

$$\begin{aligned} T^+(k)&= (1-\mu ) \frac{k}{Z}\frac{Z-k}{Z-1} \left( 1 + e^{\lambda (\Pi _{C} - \Pi _{D})} \right) ^{-1} + \frac{\mu }{2}\\ T^-(k)&= (1-\mu ) \frac{Z-k}{Z} \frac{k}{Z-1} \left( 1 + e^{\lambda (\Pi _{D} - \Pi _{C})} \right) ^{-1} + \frac{\mu }{2} \\ T^0(k)&=1-T^+(k)-T^-(k) \end{aligned}$$

where $\lambda$ is the inverse temperature associated with the influence of selection versus drift, and $\mu$ is the rate of mutation. This produces an ergodic Markov process.

1.3 Gradient of selection

The gradient of selection of the process captures the expected direction of selection as a function of the number of cooperators in the population, k, in a way that is analogous to the mean-field dynamics for the infinite-population case. This is given by

$$\begin{aligned} G(k) = T^+(k)-T^-(k) = \frac{k}{Z}\frac{Z-k}{Z-1}\left( \tanh \frac{\lambda }{2}\left( \Pi _C(k)-\Pi _D(k)\right) \right) , \end{aligned}$$

where $G(k)>0$ implies that selection favours cooperation, and $G(k)<0$ implies that defection is favoured.

1.4 Stationary distribution

The stationary distribution of the process captures the long run distribution of time the process spends at each state. For an ergodic process, the stationary distribution is known to be unique and independent of initial conditions of that process. We compute is as follows.

$$\begin{aligned} \sigma _k&= \frac{ \prod ^k_{i=1} \frac{T^+(j-1)}{T^-(j)} }{\sum ^{Z}_{i=1} \prod ^{i}_{j=1} \frac{T^+(j-1)}{T^-(j)}}\quad \text {for} \ k \in \{ 1, \dots , Z \}. \end{aligned}$$

The stationary distribution can also be approximated via the Chapman–Kolmogorov equation which states that nth step transition matrix corresponds to the nth power of the one-step transition matrix, $T_t=T^t$. Thus, we get that $\sigma$ corresponds to any row of the matrix given by $\lim _{t\rightarrow \infty } T^t$.

Proofs

Here we demonstrate several propositions which elucidate the general relationship between selection for cooperation under the Fermi dynamics and the parameters of the strategic interaction.

We say that selection for a strategy, $\sigma$, under the dynamics is increasing in parameter x if the transition probability $T^+(k)$ from a state with k individuals playing strategy $\sigma$ to one with $k+1$ individuals playing $\sigma$ increases as x increases, for every interior state. That is $x < x'$ implies $T^+(k;x) < T^+(k;x')$ for all $k \in \{1,\dots ,Z-1\}$.

For the following proofs, we fix the initial endowment of agents as some positive constant, $b>0$, without loss of generality, and we assume non-extreme values of the strategic parameters of interest: $N\le Z \in {\mathbb {N}}$; $r \in (0,1)$; $m\in (0,1)$; $c\in (0,1)$; $p^{*} \in (\frac{1}{N},\frac{N-1}{N})$; $\mu \in (0,1)$; and $\lambda \in {\mathbb {R}}_{>0}$. Note that allowing for extreme values of the parameters makes it so the following inequalities hold only weakly.

Lemma C.1

Selection for a cooperation under the Fermi dynamics increases (decreases) as the differences of its mean payoff with that of defection increases (decreases).

Proof

Recall that the transition probability from a state with k cooperators to one with $k+1$ cooperators is given by

$$\begin{aligned} T^+(k) = (1-\mu ) \frac{k}{Z}\frac{Z-k}{Z-1} \left( 1 + e^{\lambda (\Pi _{C} - \Pi _{D})} \right) ^{-1} + \frac{\mu }{2}. \end{aligned}$$

So, for any (non-extremal) values of k, $\lambda$, and $\mu$, we have it that $T^+$ is proportional to the logit function, $\left( 1 + e^{\lambda (\Pi _{C} - \Pi _{D})} \right) ^{-1}$, which in turn clearly increases (decreases) as the difference of mean payoffs, $\Pi _{C} - \Pi _{D}$, increases (decreases). $\square$

Proposition C.2

Selection for cooperation decreases as the cost to cooperation increases.

Proof

Consider the difference in mean payoffs between cooperators and defectors, $\Pi _{C}(k;c) - \Pi _{D}(k;c)$, as a function of the cost of cooperation, $c \in (0,1)$, and then fix each of the other parameters at some non-extremal values.

Observe that for any fixed number of cooperators, $k \in \{1,\dots ,Z-1\}$, the difference in mean cost of cooperation

$$\begin{aligned} \Pi _{C}(k;c) - \Pi _{D}(k;c) \propto \sum ^N_{n_C=0}\pi _{C}(n_C;c) - \pi _{D}(n_C;c) \propto -cb. \end{aligned}$$

Since $b>0$, increasing cost of cooperation, c, decreases $\Pi _{C} - \Pi _{D}$, as required.

By lemma C.1, it follows that selection for cooperation decreases as the cost of cooperation increases. $\square$

Proposition C.3

Selection for cooperation decreases as the size of cooperative groups increases.

Proof

Consider the differences in mean payoffs between cooperators and defectors, $\Pi _C(k;N)-\Pi _D(k;N)$, as a function of the size of cooperative groups, $N \in {\mathbb {N}}$, and fix each other parameter at some non-extremal value.

Reformulate the difference in mean payoffs in terms of the fraction of the expected fractions of each cooperators and defectors in successful cooperative groups, p, q respectively:

$$\begin{aligned} \Pi _{C} - \Pi _{D} = (p \Pi _{C;s} + (1-p) \Pi _{C;f}) - (q \Pi _{D;s} + (1-q) \Pi _{D;f}), \end{aligned}$$

where the subscripts s and f denote when the payoff is for success or failure. We pair the payoffs terms to get

$$\begin{aligned} (p \Pi _{C;s}-(q \Pi _{D;s}) - ((1-p) \Pi _{C;f})-(1-q) \Pi _{D;f}), \end{aligned}$$

and then use the fact that $\Pi _{D;s}=\Pi _{C;s}+cb$ and $\Pi _{D;f}=\Pi _{C;f}+cb$, and some algebra, to simplify the expression to: $(p-q)\Pi _{C;s}+(q-p)\Pi _{C;f}-cb$.

Since the payoff for success is greater than failure, $\Pi _{C;s}>\Pi _{C;f}$, it follows that the difference in average payoffs is decreasing in the difference of the fraction $p-q$ of successful cooperators and defectors. Taking the derivative of the difference of fractions with respect to N yields

$$\begin{aligned} \begin{aligned}&\frac{d}{dN}[p(N)-q(N)]=\frac{d}{dN}\left[ \sum ^{N}_{n_C=\lceil p^*N \rceil }\left( {\begin{array}{c}N\\ n_C\end{array}}\right) (k/Z)^{n_C}(1-k/Z)^{N-n_C} \frac{2n_C-N}{N}\right] \\&\propto -\sum ^{N}_{n_C=\lceil p^*N \rceil }N^{-2} \end{aligned}, \end{aligned}$$

which is strictly negative.

Hence the difference in the fractions of cooperators and defectors who succeed and fail is decreasing in group size, and so the difference in mean payoffs between cooperators and defectors is decreasing. By Lemma C.1, it follows that selection for cooperation decreases as group size increases. $\square$

Proposition C.4

Selection for cooperation increases as the product of the perceived risk and magnitude of the consequences of failing to successfully cooperate increases.

Proof

Consider the differences in mean payoffs to between cooperators and defectors,

$$\begin{aligned} \Pi _C(k;r,m)-\Pi _D(k;r,m), \end{aligned}$$

as a function of the the product of the perceived probability, $0<r<1$, and magnitude, $0<m<1$, such that $0<rm<r'm'<1$.

Observe that

$$\begin{aligned} \begin{aligned}&\Pi _{C}(k;r,m) - \Pi _{D}(k;r,m) \propto \sum ^N_{n_C=0}\pi _{C}(n_C;r,m) - \pi _{D}(n_C;r,m) \\&=\pi _{C}(0;r,m) - \pi _{D}(0;r,m) + (N-1)cb + \pi _{C}(N;r,m) - \pi _{D}(N;r,m) \\&\propto \pi _{D}(N;r,m) - \pi _{C}(0;r,m).\end{aligned} \end{aligned}$$

When $c_N=N$, we have $\pi _{C} (N;r,m) = b(1-c)$ (all cooperators; cooperation succeeds) and when $c_N=0$, we have $\pi _{C}(0;r,m) = b (1 - rm - c)$ (all defectors; cooperation fails). Thus

$$\begin{aligned} \pi _{D}(N;r,m) - \pi _{C}(0;r,m)=brm, \end{aligned}$$

and since $b>0$, this is increasing in rm.

Hence the difference in mean payoffs, $\Pi _{C}(k;r,m) - \Pi _{D}(k;r,m)$, is also increasing in rm. By lemma C.1, it follows that selection for cooperation increases as the product of the risk and magnitude of the failure to cooperate increases. $\square$