The expected bit complexity of the von Neumann rejection algorithm

Abstract

In 1952, von Neumann introduced the rejection method for random variate generation. We revisit this algorithm when we have a source of perfect bits at our disposal. In this random bit model, there are universal lower bounds for generating a random variate with a given density to within an accuracy $\epsilon$ derived by Knuth and Yao, and refined by the authors. In general, von Neumann’s method fails in this model. We propose a modification that insures proper behavior for all Riemann-integrable densities on compact sets, and show that the expected number of random bits needed behaves optimally with respect to universal lower bounds. In particular, we introduce the notion of an oracle that evaluates the supremum and infimum of a function on any rectangle of ${\mathbb {R}}^{d}$, and develop a quadtree-style extension of the classical rejection method.

Appendices

Appendix 1: Riemann integrability and the sup/inf oracle

In this section, we contruct a family of densities that are not Riemann-integrable for which the oracle is useless. Let $\delta \in [0,1\slash 3)$ be a parameter, and let $I_{\delta }\subseteq [0,1]$ be a Cantor-like set constructed below. Setting

$$\begin{aligned} f_{\delta }(x)=\frac{1}{\lambda (I_{\delta })}\mathbbm {1}_{\{x\in I_{\delta }\}}, \end{aligned}$$

where $\lambda$ is the Lebesgue measure, we have (see below) $\lambda (I_{\delta })=(1-3\delta )\slash (1-2\delta )$. The Lebesgue measure of the set of discontinuities is $1-\lambda (I_{\delta })$ and is zero only if $\delta =0$. The case of $\delta =0$ corresponds to the usual uniform density on [0, 1] that is Riemann-integrable. All cases of $\delta \in (0,1\slash 3)$ are Lebesgue integrable but not Riemann-integrable because the set of discontinuities is non-zero and yet possess a cumulative distribution function. For every $\epsilon >0$ and every $x\in I_{\delta }$ we have

$$\begin{aligned}&\inf _{[x-\epsilon ,x+\epsilon ]}{f(x)}=0,\\ \text {and}&\\&\sup _{[x-\epsilon ,x+\epsilon ]}{f(x)}=\frac{1-2\delta }{1-3\delta }. \end{aligned}$$

Since these boundaries are invariant under changes of $\epsilon$, Algorithm 4, when used for rejection, say, from a uniform density, has an infinite loop with positive probability. It is, therefore, essential that Riemann-integrable densities are considered as that the supremum and infimum returned by the oracle over given small intervals converge to each other as intervals shrink.

For the construction of $I_{\delta }$, we recursively remove middle open subintervals of geometrically decreasing sizes. Let $I_{j,k}\subset [0,1]$ for $j\in {\mathbb {N}}\setminus \{0\}$ and $k=0,\ldots {},2^{j}-1$ be the $2^{j}$ closed subintervals that are left once the middle parts are removed from the previous subintervals $I_{j-1,k}$ with $k=0$ corresponding to the leftmost subinterval and so on. Initially, $I_{0,0}=[0,1]$. For all $j\in {\mathbb {N}}\setminus \{0\}$ and $k\in \{0,\ldots {},2^{j}-1\}$, the length of a removed middle part is $\delta ^{j}$. For all $j\in {\mathbb {N}}\setminus \{0\}$, the total length not removed at the j-th step is $2^{j}\lambda (I_{j,0})$ because the subintervals are of the same length. Let $I_{\delta }$ be the limiting subset of [0, 1] that is left, i.e.,

$$\begin{aligned} I_{\delta }=\bigcap _{j=1}^{\infty }{\bigcup _{k=0}^{2^{j}-1}{I_{j,k}}}. \end{aligned}$$

We compute $\lambda (I_{\delta })$ as follows:

$$\begin{aligned} \lambda (I)&=\lim _{j\rightarrow \infty }{\sum _{k=0}^{2^{j}-1}{\lambda (I_{j,k})}}\,\,(\hbox {by the definition of }I_{\delta })\\&=\lim _{j\rightarrow \infty }{2^{j}\lambda (I_{j,0})},\\ \lambda (I_{1,0})&=\frac{1}{2}-\frac{\delta }{2},\\ \lambda (I_{j,0})&=\frac{1}{2}\lambda (I_{j-1,0})-\frac{\delta ^{j}}{2}\\&=\frac{1}{2^{j}}-\frac{\delta }{2^{j}}-\frac{\delta ^{2}}{2^{j-1}}-\ldots -\frac{\delta ^{j}}{2}\,\,\hbox {for }j\ge 1, \end{aligned}$$

and therefore

$$\begin{aligned} 2^{j}\lambda (I_{j,0})=1-\delta \sum _{i=0}^{j-1}{(2\delta )^{i}}=1-\delta \left( \frac{(2\delta )^{j}-1}{2\delta -1}\right) , \end{aligned}$$

so that

$$\begin{aligned} \lambda (I_{\delta })=\frac{1-3\delta }{1-2\delta }. \end{aligned}$$

Appendix 2: A naive modification to the general rejection method that is incorrect

A trivial, but incorrect, modification to Algorithm 3 would be:

This attempt leads to failure. Let A be the support of $X_{\epsilon }$, which is necessarily contained in a fixed countable subset of $[0,1]^{d}$. Then given any Riemann-integrable density g, take a finite subset $A^{\star }$ of A, and modify g on $A^{\star }$ by setting

$$\begin{aligned} f(x) = \left\{ \begin{array}{ll} g(x) &{} \text {if } x \notin A^{\star },\\ C &{} \text {if } x \in A^{\star }. \end{array} \right. \end{aligned}$$

We have that f is still a Riemann-integrable density bounded by C (and its set of discontinuities is of measure 0 because $A^{\star }$ is countable) but since $f(X_{\epsilon })=C$ if $X_{\epsilon }\in A^{\star }$, we accept all $X_{\epsilon }\in A^{\star }$, regardless of the density g we started with. Since ${\mathbf {P}}\{X_{\epsilon }\in A^{\star }\}>0$, we make an error with positive probability.

If we set

$$\begin{aligned} f(x) = \left\{ \begin{array}{ll} g(x) &{} \text {if } x \notin A,\\ 0 &{} \text {if } x \in A, \end{array} \right. \end{aligned}$$

then g is no longer Riemann-integrable, and in that case, $f(X_{\epsilon })=0$ with probability one, and therefore, the algorithm loops forever, regardless of the choice of g.

This simple example shows the necessity of the oracle and of the condition of Riemann-integrability.