Jump Markov chains and rejection-free Metropolis algorithms

Abstract

We consider versions of the Metropolis algorithm which avoid the inefficiency of rejections. We first illustrate that a natural Uniform Selection algorithm might not converge to the correct distribution. We then analyse the use of Markov jump chains which avoid successive repetitions of the same state. After exploring the properties of jump chains, we show how they can exploit parallelism in computer hardware to produce more efficient samples. We apply our results to the Metropolis algorithm, to Parallel Tempering, to a Bayesian model, to a two-dimensional ferromagnetic 4$\times$4 Ising model, and to a pseudo-marginal MCMC algorithm.

Appendix: Proof of Proposition 1

Lemma 15

For the Uniform Selection chain of Fig. 6, let $s(x) = \mathbf{P}($hit 4 before 0$\, | \, X_0=x)$. Then $s(0)=0$, $s(1)=3/7$, $s(2)=4/7$, $s(3)=13/21$, and $s(4)=1$.

Proof

Clearly $s(0)=0$ and $s(4)=1$. Also, by conditioning on the first step, for $1 \le x \le 3$ we have $s(x) = p_{x,x-1} \, s(x-1) + p_{x,x+1} \, s(x+1)$. In particular, $s(1) = (1/4) s(0) + (3/4) s(2) = (3/4) s(2)$, and $s(2) = (1/4) s(1) + (3/4) s(3)$, and $s(3) = (8/9) s(2) + (1/9) s(4) = (8/9) s(2) + (1/9)$. We solve these equations using algebra. Substituting the first equation into the second, $s(2) = (1/4)(3/4) s(2) + (3/4) s(3)$, so $(13/16) s(2) = (3/4) s(3)$, so $s(3) = (13/16)(4/3) s(2) = (13/12) s(2)$. Then the third equation gives $(13/12) s(2) = (8/9) s(2) + (1/9)$, so $(7/36) s(2) = (1/9)$, so $s(2)=(1/9)(36/7) = 4/7$. Then $s(1) = (3/4) s(2) = (3/4)(4/7) = 3/7$, and $s(3) = (8/9) s(2) + (1/9) = (8/9) (4/7) + (1/9) = 13/21$, as claimed. $\square$

Lemma 16

Suppose the Uniform Selection chain for Example 2 begins at state $x=4a$ for some positive integer a. Let C be the event that the chain hits $4(a+1)$ before hitting $4(a-1)$. Then $q := \mathbf{P}(C) = 9/17 > 1/2$.

Proof

By conditioning on the first step, we have that

$$\begin{aligned} q= & {} \mathbf{P}(C \, | \, X_0=4a) \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \\= & {} \mathbf{P}(X_1=4a+1) \ \mathbf{P}(C \, | \, X_0=4a+1) + \mathbf{P}(X_1=4a-1) \ \mathbf{P}(C \, | \, X_0=4a-1)\\= & {} (1/2) \ \mathbf{P}(C \, | \, X_0=4a+1) + (1/2) \ \mathbf{P}(C \, | \, X_0=4a-1) \, . \end{aligned}$$

But from $4a+1$, by Lemma 15, we either reach $4a+4$ before returning to 4a (and “win”) with probability 3/7, or we first return to 4a (and “start over”) with probability 4/7. Similarly, from $4a-1$, we either return to 4a (and “start over”) with probability 13/21, or we reach $4a-4$ before returning to 4a (and “lose”) with probability 8/21. Hence,

$$\begin{aligned} q = (1/2) \, [ (3/7) + (4/7) q ] + (1/2) \, [ (13/21) q + 0] \, . \end{aligned}$$

That is, $q = (3/14) + (2/7) q + (13/42) q = (3/14) + (25/42) q$. Hence, $q = (3/14) \bigm / (17/42) = 9/17 > 1/2$. $\square$

We then have:

Corollary 17

Suppose the Uniform Selection chain for Example 2 begins at state $4a \ge 8$ for some positive integer $a \ge 2$. Then the probability it will ever reach the state 4 is $(8/9)^{a-1} < 1$.

Proof

Consider a sub-chain $\{\tilde{X}_n\}$ of $\{X_n\}$ which just records new multiples of 4. That is, if the original chain is at the state 4b, then the new chain is at b. Then, we wait until the original reaches either $4(b-1)$ or $4(b+1)$ at which point the next state of the new chain is $b-1$ or $b+1$ respectively. Then Lemma 16 says that this new chain is performing simple random walk on the positive integers, with up-probability 9/17 and down-probability 8/17. Then it follows from the Gambler’s Ruin formula (e.g. Rosenthal 2006, equation 7.2.7) that, starting from state a, the probability that the new chain will ever reach the state 1 is equal to $[(8/17)/(9/17)]^{a-1} = (8/9)^{a-1} < 1$, as claimed. $\square$

Since the chain starting at 4a for $a \ge 2$ cannot reach state 3 without first reaching state 4, Proposition 1 follows immediately from Corollary 17.

If we instead cut off the example at the state 4L, then the Gambler’s Ruin formula (e.g. Rosenthal 2006, equation 7.2.2) says that from the state $4(L-1)$, the probability of reaching the state 4 before returning to the state 4L is $[(9/8)^1-1] \bigm / [(9/8)^{L-2}-1] < (8/9)^{L-1}$ (since $[A-1] \bigm / [B-1] < A/B$ whenever $1<A<B$), so the expected number of attempts to reach state 4 from state 4L is more than $(9/8)^{L-1}$.