Instrumental divergence

Abstract

The thesis of instrumental convergence holds that a wide range of ends have common means: for instance, self preservation, desire preservation, self improvement, and resource acquisition. Bostrom contends that instrumental convergence gives us reason to think that “the default outcome of the creation of machine superintelligensome of the ‘convergence is existential catastrophe”. I use the tools of decision theory to investigate whether this thesis is true. I find that, even if intrinsic desires are randomly selected, instrumental rationality induces biases towards certain kinds of choices. Firstly, a bias towards choices which leave less up to chance. Secondly, a bias towards desire preservation, in line with Bostrom’s conjecture. And thirdly, a bias towards choices which afford more choices later on. I do not find biases towards any other of the convergent instrumental means on Bostrom’s list. I conclude that the biases induced by instrumental rationality at best weakly support Bostrom’s conclusion that machine superintelligence is likely to lead to existential catastrophe.

Appendix

Let ${\mathcal {W}}$ be the collection of ways the world might be, for all Sia is in a position to know. I’ll assume that ${\mathcal {W}}$ is finite, with cardinality N. Fix some enumeration of the worlds in ${\mathcal {W}}$, $W_1, W_2, \dots , W_N$. I assume that there is a fixed collection of available acts, ${\mathcal {A}}$, between which Sia must choose. We can represent each of the possible desires Sia might hold with a function $D: {\mathcal {W}}\rightarrow \mathbb {R}$. The interpretation is that D(W) measures how well satisfied Sia’s desires are at the world W.

I’ll suppose that, in order for us to say which of the acts in ${\mathcal {A}}$ are more or less rational than which others, we need to be given one more ingredient: we’ll need, for each $A \in {\mathcal {A}}$, a suppositional probability function $P_A$. Since ${\mathcal {W}}$ is finite, we can take each of these probability functions to be defined over the powerset $\mathscr {P}({\mathcal {W}})$.

For each $A \in {\mathcal {A}}$, we can use the suppositional probability function $P_A$ and the desire function D to calculate Sia’s expected utility for A.

Definition 1

The expected utility of $A \in {\mathcal {A}}$, relative to the desires D, is the value which the suppositional probability function $P_A$ expects D to take on.

$$\begin{aligned} {\mathbb {E}}_{P_A}[D] = \sum _{W \in {\mathcal {W}}} D(W) \cdot P_A(W) \end{aligned}$$

Both causal and evidential decision theory say that an act A is more rational than another, B, iff A’s expected utility is greater than B’s. They disagree over how to understand the probability functions $P_A$. Evidentialists say that $P_A(X)$ is the conditional probability function $P(X \mid A)$.¹⁷ Causalists say that $P_A(X)$ is the probability function P imaged on the performance of A. They take for granted an imaging function, $I: {\mathcal {A}}\times {\mathcal {W}}\rightarrow (\mathscr {P}({\mathcal {W}}) \rightarrow [0, 1])$, from pairs of acts and worlds to probability distributions. The interpretation is that I(A, W)(X) is how likely it is that the propositions X would be true, were you to choose A at the world W. Then, causalists say that $P_A(X) = \sum _{W \in {\mathcal {W}}} I(A, W)(X) \cdot P(W)$.¹⁸ Other decision theories, like the functional decision theory from Yudkowsky and Soares (2018) and Soares and Levinstein (2020), will also take this form, though they will understand the suppositional probability distributions differently.

Proposition 1

For any $A, B \in {\mathcal {A}}$, if $P_A \ne P_B$, then there are infinitely many desires D such that the expected utility of A is greater than the expected utility of B, relative to D.

Proof

If $P_A \ne P_B$, then there is some set of worlds $X \subseteq {\mathcal {W}}$ such that $P_A(X) > P_B(X)$. Select any two numbers $x, y \in \mathbb {R}$ such that $x > y$ and consider a desire function D such that, for all $W \in X$, $D(W) = x$, and for all $W \notin X$, $D(W) = y$. Then, the expected utility of A, relative to D, will be $x P_A(X) + y[1-P_A(X)]$ and the expected utility of B, relative to D, will be $xP_B(X) + y[1-P_B(X)]$. Since $P_A(X) > P_B(X)$ and $x>y$, the expected utility of A will exceed the expected utility of B, relative to these desires. Since there are infinitely many choices of x and y such that $x>y$, there are infinitely many such desires.

Note that, by just taking another instance of the proposition in which we exchange A and B, we also get that there are infinitely many desires such that the expected utility of B exceeds the expected utility of A, relative to those desires. Note also that, on the causalist’s understanding, $P_A \ne P_B$ iff the expected consequences of A are different from the expected consequences of B.

Finally, note that proposition 1 doesn’t depend upon whether we are considering a ‘small world’ or ‘grand world’ decision. If it is a grand world decision, so that P is defined over propositions about which choice Sia makes, then $P_A(W) > 0$ implies that $P_B(W) = 0$, for every $B \ne A$. Then,

Proposition 2

In any grand world decision, and any available act in the decision, A, there are infinitely many desires which make A more rational than every other alternative.

Proof

Select any two numbers $x, y \in \mathbb {R}$ such that $x > y$ and consider a desire function D such that, for all $W \in A$, $D(W) = x$, and for all $W \notin A$, $D(W) = y$. Then, the expected utility of A will be $\sum _W P_A(W) \cdot D(W) = \sum _{W \in A} P_A(W) \cdot x = x$, and the expected utility of every other act, B, will be $\sum _{W} P_B(W) \cdot D(W) = \sum _{W \in B} P_B(W) \cdot y = y$. So the expected utility of A will exceed the expected utility of every other act. Since there are infinitely many choices of x and y such that $x>y$, there are infinitely many such desires.

Let $\mathfrak {D}$ be the set of all desires Sia could have, $\mathfrak {D} = \{ D: {\mathcal {W}}\rightarrow \mathbb {R} \}$. We can define a probability distribution, Q, over a $\sigma$-field of subsets of $\mathfrak {D}$, $\mathfrak {F} \subseteq \mathscr {P}(\mathfrak {D})$. That is, $\mathfrak {F}$ is a set of propositions about Sia’s desires such that (i) $\mathfrak {D} \in \mathfrak {F}$, (ii) $X^c \in \mathfrak {F}$ whenever $X \in \mathfrak {F}$, and (iii) $\bigcup _{i=1}^\infty X_i \in \mathfrak {F}$ whenever $X_1, X_2, \dots \in \mathfrak {F}$. The interpretation is that Q(Y) is our probability that Sia’s desires fall somewhere within the set $Y \subseteq \mathfrak {D}$.

I’ll suppose that our probability distribution Q satisfies the following four conditions. To explain the first condition: let ‘$D_i$’ be a random variable which takes a desire function $D \in \mathfrak {D}$ to the real number $D(W_i)$. Then, I’ll suppose that, for every $x \in \mathbb {R}$, the proposition $D_i \leqslant x$ is included in $\mathfrak {F}$. This way, the cumulative distribution function $Q(D_i \leqslant x)$ will be well-defined, for every $i \in \{ 1 \dots N \}$ and every $x \in \mathbb {R}$. I’ll also suppose that this function is absolutely continuous.

• (1) For every $i \in \{ 1 \dots N \}$ and every $x \in \mathbb {R}$, $D_i \leqslant x \in \mathfrak {F}$, and $Q(D_i \leqslant x)$ is absolutely continuous.

If this first condition is satisfied, then for each world $W_i$, we can define a probability density function $q_i(x) = (d/dx) Q(D_i \leqslant x)$.

Secondly, I’ll assume that, for any two worlds $W_i, W_j \in {\mathcal {W}}$, we don’t have any reason to think that $W_i$ will better satisfy Sia’s desires than $W_j$ will. So, for any $i, j \in \{ 1 \dots N \}$, our expectation of the value of $D_i$ should equal our expectation of the value of $D_j$.

• (2) For any $i, j \in \{ 1 \dots N \}$,

  $$\begin{aligned} {\mathbb {E}}_Q[D_i] = \int _{-\infty }^\infty x \cdot q_{i}(x) \,\, dx = \int _{-\infty }^\infty x \cdot q_j(x) \,\, dx = {\mathbb {E}}_Q[D_j] \end{aligned}$$

I’ll call this common expectation ‘$\mu$’. $\mu$ is our expectation of the degree to which Sia’s desires will be satisfied at any particular world.

Thirdly, I’ll assume that we’ve no more reason to think that Sia’s desires are going to be satisfied to degree $\mu + x$ than we have to think that her desires are going to be satisfied to degree $\mu - x$. That is: I’ll assume that each probability density function $q_i$ is symmetric around the mean $\mu$.

• (3) For each $i \in \{ 1 \dots N \}$ and each $x \in \mathbb {R}$, $q_i(\mu + x) = q_i(\mu - x)$.

And finally, I’ll assume that learning how well satisfied Sia’s desires are at some worlds doesn’t tell us how well satisfied her desires are at other worlds.

• (4) The random variables $D_1, D_2, \dots D_N$ are mutually independent.

If these four conditions are satisfied, then I’ll say that Sia’s desires are ‘sampled randomly’.

Proposition 3

If Sia’s desires are sampled randomly, then, for any two acts $A, B \in {\mathcal {A}}$, the probability that Sia’s desires make A more rational than B is equal to the probability that Sia’s desires make B more rational than A.

Proof

First, notice that whether A is more or less rational than B does not change if we replace Sia’s desires with a positive affine transformation of those desires. That is, if $\hat{D}$ is a positive affine transformation of D, then ${\mathbb {E}}_{P_A}[D] \geqslant {\mathbb {E}}_{P_B}[D]$ iff ${\mathbb {E}}_{P_A}[\hat{D}] \geqslant {\mathbb {E}}_{P_B}[\hat{D}]$. So, if Sia’s desires are D, we can let $\hat{D} =_{df} D - \mu$, where $\mu$ is our expectation of the degree to which Sia’s desires will be satisfied at any world (measured in the units of D). Having performed this transformation, our expectation of the degree to which Sia’s desires will be satisfied, in the units of $\hat{D}$, will be zero. For ${\mathbb {E}}_Q[\hat{D}_i] = {\mathbb {E}}_Q[D_i - \mu ] = {\mathbb {E}}_Q[D_i] - \mu = \mu - \mu = 0$. I will use the units of the ‘shifted’ scale $\hat{D}$ for the remainder of the proof.

Next, consider the random variable $Z = {\mathbb {E}}_{P_A}[\hat{D}] - {\mathbb {E}}_{P_B}[\hat{D}]$. If Z is positive, then Sia’s desires make A more rational than B. If Z is negative, then her desires make B more rational than A. Zero, and her desires make A and B equally rational. Note that Z is a linear combination of the random variables $\hat{D}_i$

$$\begin{aligned} Z&= \sum _{i=1}^N \hat{D}_i \cdot P_A(W_i) \,-\, \sum _{i=1}^N \hat{D}_i \cdot P_B(W_i) \\&= \sum _{i=1}^N \left[ P_A(W_i) - P_B(W_i) \right] \cdot \hat{D}_i \end{aligned}$$

Let $c_i = P_A(W_i) - P_B(W_i)$. Then, $Z = \sum _{i=1}^N c_i \cdot \hat{D}_i$.

Let $\varphi _i(t) =_{df} {\mathbb {E}}_Q[e^{t \hat{D}_i \sqrt{-1} }]$ be the characteristic function of the random variable $\hat{D}_i$. The characteristic function of a random variable is real-valued iff that variable is probabilistically symmetric about the origin—that is, for each x, the probability that the variable takes on a value greater than x is equal to the probability that it takes on a value less than $-x$ (see Billingsley, 1986, problem 26.2.) By assumption, each random variable $D_i$ is symmetric about their common mean $\mu$, so the ‘shifted’ variables $\hat{D}_i$ are symmetric about the origin. So each $\varphi _i(t)$ is real-valued. Given any N mutually independent random variables $V_1, V_2, \dots V_N$ with characteristic functions $\varphi _{V_1}(t), \varphi _{V_2}(t), \dots , \varphi _{V_N}(t)$, the characteristic function for their linear combination $U = \sum _{i=1}^N c_i \cdot V_i$ is $\varphi _U(t) = \prod _{i=1}^N \varphi _{V_i}(c_i t)$.¹⁹ Since by assumption the random variables $\hat{D}_i$ are mutually independent, and since $Z = {\mathbb {E}}_{P_A}[\hat{D}] - {\mathbb {E}}_{P_B}[\hat{D}]$ is a linear combination of the $\hat{D}_i$, $Z = \sum _{i=1}^N c_i \cdot \hat{D}_i$, the characteristic function for Z, $\varphi _Z(t)$, is $\prod _{i=1}^N \varphi _i(c_i t)$. The product of N real-valued functions is real-valued. So $\varphi _Z$ is real-valued. So Z is also probabilistically symmetric about the origin. So $Q(Z > 0) = Q(Z < 0)$. So Sia’s desires are just as likely to make A more rational than B as they are to make B more rational than A.

Let’s consider an additional assumption about our probability distribution, Q. If our probabilities are distributed independently and identically for each random variable $D_i$, and, moreover, this distribution is a Gaussian or normal distribution $D_i {\mathop {\sim }\limits ^{\text {\tiny {iid}}}} \mathcal {N}(\mu , \sigma )$, then I’ll say that Sia’s desires are sampled normally from the space of all possible desires.

Proposition 4

If Sia’s desires are sampled normally from the space of all possible desires, then, in any grand world decision, the probability that her desires will rationalise choosing A increases with $\Vert P_A \Vert = \left( \sum _{W \in {\mathcal {W}}} P_A(W)^2\right) ^{1/2}$.

Proof

As explained in the proof of proposition 3, we can re-scale Sia’s desires by taking $\hat{D}_i = D_i - \mu$. Then, we will have the variables $\hat{D}_i {\mathop {\sim }\limits ^{\text {\tiny {iid}}}} \mathcal {N}(0, \sigma )$. Since this is a grand world decision, there is no world $W \in {\mathcal {W}}$ such that $P_A(W)$ and $P_B(W)$ are both positive, for any $A \ne B$. So Sia’s expected utilities for acts are independent (since the $\hat{D}_i$s are independent). Let $Y_i$ be Sia’s expected utility for $A_i$. Then, $Y_i = \sum _j \hat{D}_j \cdot P_{A_i}(W_j)$, which is a linear combination of the variables $\hat{D}_j$. Since each $\hat{D}_j$ is an iid normal random variable with mean 0 and variance $\sigma ^2$, their linear combination will be a normal variable with mean 0 and variance $\sigma ^2 \cdot \sum _j P_{A_i}(W_j)^2 = \sigma ^2 \Vert P_{A_i} \Vert ^2$. So $X_i = Y_i / (\sigma \cdot \Vert P_{A_i} \Vert )$ will have a standard normal distribution. That is, if $q_i$ is our probability density function for the random variable $X_i$, then $q_i(x) = \phi (x)$, where $\phi (x)$ is the standard normal distribution,

$$\begin{aligned} \phi (x) =_{df} \frac{1}{\sqrt{2 \pi }} e^{-x^2/2} \end{aligned}$$

Since the $X_i$ are independent, their joint probability density is just the product of the marginal densities,

$$\begin{aligned} q(x_1, x_2, \dots , x_N) = \prod _{i=1}^n \phi (x_i) \end{aligned}$$

Sia’s expected utility for $A_i$ is greater than her expected utility for $A_j$ exactly if $Y_i > Y_j$, which is so exactly if $(\sigma \cdot \Vert P_{A_i}\Vert ) X_i > (\sigma \cdot \Vert P_{A_j} \Vert ) X_j$, which is so exactly if $X_j < (\Vert P_{A_i} \Vert / \Vert P_{A_j} \Vert ) X_i$. Without loss of generality, consider the probability that $A_1$ maximises expected utility. That probability is given by

$$\begin{aligned}& \int _{-\infty }^\infty \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_2} \Vert ) x_1} \dots \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_n} \Vert )x_1} q(x_1, x_2, \dots , x_n) \,\,dx_n \dots dx_2 dx_1 \\&= \int _{-\infty }^\infty \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_2} \Vert ) x_1} \dots \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_n} \Vert )x_1} \prod _{i=1}^n \phi (x_i) \,\,dx_n \dots dx_2 dx_1 \\&= \int _{-\infty }^\infty \phi (x_1) \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_2} \Vert )x_1} \phi (x_2) \dots \int _{-\infty }^{(\Vert P_{A_1} \Vert / \Vert P_{A_n} \Vert )x_1} \phi (x_n) \,\,dx_n \dots dx_2 dx_1 \end{aligned}$$

In general,

$$\begin{aligned} \int _{-\infty }^c \phi (x) \,\, dx = \Phi (c) \end{aligned}$$

(1)

where $\Phi$ is the cumulative density function for the standard normal distribution. Then, the probability that $A_1$ maximises expected utility is given by

$$\begin{aligned} \int _{-\infty }^\infty \phi (x_1) \cdot \prod _{i=2}^N \Phi ( x_1 \Vert P_{A_1} \Vert / \Vert P_{A_i} \Vert ) \,\, dx_1 \end{aligned}$$

$\Phi (x)$ is an increasing function of x. So, as $\Vert P_{A_1} \Vert$ gets larger (if everything else is held fixed), each of the factors in the product in (1) will become larger. Since $\phi (x)$ is non-negative, this will mean that the value of the integral will become larger. So the probability that $A_1$ maximises expected utility will be larger. $A_1$ was arbitrary, so what goes for it goes for every other option; in general, for any $A \in {\mathcal {A}}$, as $\Vert P_A \Vert$ increases (with everything else held fixed), the probability that A maximises expected utility increases.

Proposition 5

If Sia’s desires are sampled randomly, then, in a grand world decision with n available acts, the probability that she chooses any given act is at least $1/2^{n-1}$.

Proof

Let ${\mathcal {A}}= \{ A_1, A_2, \dots , A_n \}$, and take any $A_i \in {\mathcal {A}}$. Without loss of generality, let it be $A_1$. Since $Q(D_i \leqslant x)$ is absolutely continuous, the probability that Sia’s expected utility for $A_1$ is equal to her expected utility for $A_j$ ($j \ne i$) is zero. So we can ignore this possibility when calculating the probability that A is uniquely rational. If we write ‘$A_1 \succ A_i$’ for ‘${\mathbb {E}}_{P_{A_1}}[D] > {\mathbb {E}}_{P_{A_i}}[D]$’, the probability that $A_1$ is uniquely rational is

$$\begin{aligned} Q(A_1 \succ A_2 \wedge A_1 \succ A_3 \wedge \dots \wedge A_1 \succ A_n) = \left[ \prod _{i=2}^{n-1} Q\left( A_1 \succ A_i \,\,\Big |\,\, \bigwedge _{j=i+1}^n A_1 \succ A_j \right) \right] \cdot Q(A_1 \succ A_n) \end{aligned}$$

Because $P_{A_1}(A_1) = 1$ and $P_{A_i}(A_i) = 1$, there is no world W such that both $P_{A_1}$ and $P_{A_i}$ give W positive probability (for any $i > 1$). So ${\mathbb {E}}_{P_{A_1}}[D] = \sum _{j} P_{A_1}(W_j) \cdot D_j$ and ${\mathbb {E}}_{P_{A_i}}[D] = \sum _j P_{A_i}(W_j) \cdot D_j$ will be independent. So, for each $i \geqslant 2$, $Q\left( A_1 \succ A_i \,\,\mid \,\, \bigwedge _{j=i+1}^n A_1 \succ A_j \right) \geqslant Q(A_1 \succ A_i)$.

By proposition 3, $Q(A_1 \succ A_i) = Q(A_i \succ A_1) = 1/2$. So

$$\begin{aligned} \prod _{i=2}^{n-1} Q\left( A_1 \succ A_i \,\,\Big |\,\, \bigwedge _{j=i+1}^n A_1 \succ A_j \right) \cdot Q(A_1 \succ A_n) \geqslant (1/2)^{n-1} \end{aligned}$$