Variance Reduction for Matrix Computations with Applications to Gaussian Processes

Abstract

In addition to recent developments in computing speed and memory, methodological advances have contributed to significant gains in the performance of stochastic simulation. In this paper we focus on variance reduction for matrix computations via matrix factorization. We provide insights into existing variance reduction methods for estimating the entries of large matrices. Popular methods do not exploit the reduction in variance that is possible when the matrix is factorized. We show how computing the square root factorization of the matrix can achieve in some important cases arbitrarily better stochastic performance. In addition, we detail a factorized estimator for the trace of a product of matrices and numerically demonstrate that the estimator can be up to 1,000 times more efficient on certain problems of estimating the log-likelihood of a Gaussian process. Additionally, we provide a new estimator of the log-determinant of a positive semi-definite matrix where the log-determinant is treated as a normalizing constant of a probability density.

Appendices

A Proof of Theorem 1

The following proof is not explicitly stated in [15] and so we include it here for completeness.

Proof

To see this, first observe that when $\mathbf {C} = \mathbf {B}$, $\sum _{j}b^2_{i,j}=a_{i,i}$. Therefore, from (3) we can deduce $\mathbb {V}\mathrm {ar}_G[a_{i,i}]= 2a^2_{i,i}.$ Then, applying Cauchy-Schwarz inequality to (3), we obtain the inequality,

$$ \mathbb {V}\mathrm {ar}_G[a_{i,i}] \ge (\boldsymbol{b}_{i,:})^\top (\boldsymbol{c}_{i,:}) + a_{i,i}^2. $$

Since $a_{i,i}^2$ remains the same for every decomposition of the form $\mathbf {A} = \mathbf {B} \mathbf {C}^\top$ and Cauchy-Schwarz inequality holds with equality if and only if $\boldsymbol{b}_{i,:}$ and $\boldsymbol{c}_{i, :}$ are linearly dependent, we conclude that $\mathbb {V}\mathrm {ar}_G[a_{i,i}]$ is minimized when $\mathbf {C} = \mathbf {B}$.

B Uniform Spherical Estimator

1.1 B.1 Proof of Theorem 2

Proof

Suppose $\boldsymbol{Z} = (Z_1,\dots ,Z_n)^{\top }$ is uniformly distributed on the surface of the unit radius n-dimensional sphere centered at the origin. Using spherical coordinates, it is not difficult to show that the vector $\boldsymbol{X}=\boldsymbol{Z}\odot \boldsymbol{Z}$ follows a Dirichlet distribution over the simplex:

$$ f(\boldsymbol{x})=\frac{\pi ^{n/2}}{\varGamma (n/2)}\prod _{i=1}^n x_i^{-1/2},\quad x_i\in [0,1],\quad \sum _{i=1}^nx_i=1. $$

Therefore, using the well-known mean and variance formula for the Dirichlet distribution we have,

$$\begin{aligned} \mathbb {E}[\boldsymbol{X}]=\boldsymbol{1}/n,\quad \mathbb {V}\mathrm {ar}[\boldsymbol{X}]=\frac{1}{n(n/2+1)}\left[ \mathbf {I}_n-\frac{1}{n}\boldsymbol{1}\boldsymbol{1}^\top \right] . \end{aligned}$$

(26)

Using spherical coordinates, we also know that $\mathbb {E}[Z_iZ_j]=0$ for $i\ne j$. Therefore,

$$\begin{aligned} \mathbb {E}[\boldsymbol{Z}\boldsymbol{Z}^{\top }]=\frac{1}{n}\mathbf {I}_n, \end{aligned}$$

and,

$$\begin{aligned} \mathbb {E}\left[ n\left( \mathbf {B} \boldsymbol{Z}\right) \left( \mathbf {C} \boldsymbol{Z}\right) ^{\top }\right] =n\mathbb {E}\left[ \mathbf {B}\boldsymbol{Z} \boldsymbol{Z}^{\top }\mathbf {C}^\top \right] = \mathbf {A}. \end{aligned}$$

To prove the variance, we first note that,

$$ \mathbb {E}[(\hat{a}_{i,j}/n)^2]=\mathbb {E}[(\left[ \mathbf {B} \boldsymbol{Z}\right] _i\times \left[ \mathbf {C} \boldsymbol{Z}\right] _j)^2] = \sum _{k,l,m,n}b_{i,k}c_{j,l}b_{i,m}c_{j,n}\mathbb {E}[Z_{k} Z_{l} Z_{m} Z_{n}]. $$

Representing $\boldsymbol{Z}$ in spherical coordinates, we obtain the formula,

$$\begin{aligned} \mathbb {E}[Z_{k} Z_{l} Z_{m} Z_{n}] = c_1[\delta _{kl}\delta _{mn}+\delta _{km}\delta _{ln}+\delta _{kn}\delta _{lm}]+(c_2-3c_1)[\delta _{kl}\delta _{lm}\delta _{mn}]. \end{aligned}$$

The constants $c_1$ and $c_2$ are given by,

$$\begin{aligned} c_1&= \mathbb {V}\mathrm {ar}[\boldsymbol{X}]_{p,q}+ \mathbb {E}[\boldsymbol{X}]_p^2= \frac{1}{n(n+2)},\qquad p \ne q,\qquad \text {and}\\ c_2&=\mathbb {V}\mathrm {ar}[\boldsymbol{X}]_{p,p}+\mathbb {E}[\boldsymbol{X}]_p^2= \frac{3}{n(n+2)}. \end{aligned}$$

Thus $c_2-3c_1=0$, and,

$$\begin{aligned} \mathbb {E}[(a_{i,j}/n)^2]&= c_1\sum _{k,l,m,n}b_{ik}c_{jl}b_{im}c_{jn}[\delta _{kl}\delta _{mn}+\delta _{km}\delta _{ln}+\delta _{kn}\delta _{lm}]\\&= c_1\left[ (\boldsymbol{b}_{i,:}^{\top }\boldsymbol{c}_{j,:})^2 +(\boldsymbol{b}_{i,:}^{\top }\boldsymbol{b}_{i,:})(\boldsymbol{c}_{j,:}^{\top }\boldsymbol{c}_{j,:})+(\boldsymbol{b}_{i,:}^{\top }\boldsymbol{c}_{j,:})^2\right] \\&= c_1\left[ 2a_{i,j}^2 +\Vert \boldsymbol{b}_{i,:}\Vert ^2\Vert \boldsymbol{c}_{i,:}\Vert ^2\right] . \end{aligned}$$

Therefore,

$$\begin{aligned} \mathbb {V}\mathrm {ar}_{S}[(\hat{a}_{i,j}/n)]&= \mathbb {E}\left[ (\hat{a}_{i,j}/n)^2\right] -a^2_{i,j}/n^2\\&= (2c_1-1/n^2)a_{i,j}^2+c_{1}\Vert \boldsymbol{b}_{i,:}\Vert ^2\Vert \boldsymbol{c}_{j,:}\Vert ^2.\\ \end{aligned}$$

Hence, the variance for $\hat{a}_{i,j}$ is,

$$\begin{aligned} \mathbb {V}\mathrm {ar}_{S}[(\hat{a}_{i,j})]&= n^2\mathbb {V}\mathrm {ar}_{S}[(\hat{a}_{i,j}/n)]\\&= \frac{n-2}{n+2}a_{i,j}^2+\frac{n}{n+2}\Vert \boldsymbol{b}_{i,:}\Vert ^2\Vert \boldsymbol{c}_{j,:}\Vert ^2. \end{aligned}$$

1.2 B.2 Proof of Theorem 4

Proof

Suppose $\mathbf {\Sigma }'$ has eigenvalues $\{\lambda _i\}$, and orthonormal eigendecomposition $\mathbf {Q}\mathbf {\Lambda } \mathbf {Q}^{\top }$. Then, as the random variable that is distributed uniformly on the surface of the n-dimensional sphere centered at the origin is invariant to orthogonal rotations, we obtain,

$$ \mathbb {V}\mathrm {ar}[\boldsymbol{Z}^\top \mathbf {\Sigma }'\boldsymbol{Z}]=\mathbb {V}\mathrm {ar}[\boldsymbol{Z}^\top \mathbf {\Lambda }\boldsymbol{Z}]=\mathbb {V}\mathrm {ar}[\boldsymbol{\lambda }^\top \boldsymbol{X}]=\boldsymbol{\lambda }^\top \mathbb {V}\mathrm {ar}[\boldsymbol{X}]\boldsymbol{\lambda }. $$

We notice that ${\text {tr}}(\mathbf {\Sigma }^\top \mathbf {\Sigma })=\boldsymbol{\lambda }^\top \boldsymbol{\lambda }$. Therefore,

$$\begin{aligned} \mathbb {V}\mathrm {ar}[n\boldsymbol{Z}^\top \mathbf {\Sigma }'\boldsymbol{Z}]=\frac{n}{n+2}\times 2\left( {\text {tr}}(\mathbf {\Sigma }'^\top \mathbf {\Sigma }')-\frac{(\sum _i\mathbf {\Sigma }'_{i,i})^2}{n}\right) . \end{aligned}$$

To see that this gives the variance for $n\boldsymbol{Z}^{\top }\mathbf {\Sigma }\boldsymbol{Z}$, we note,

$$\begin{aligned} \boldsymbol{Z}^{\top }\mathbf {\Sigma }'\boldsymbol{Z} = \boldsymbol{Z}^{\top }\left[ (\mathbf {\Sigma } + \mathbf {\Sigma }^{\top })/2\right] \boldsymbol{Z}=\boldsymbol{Z}^{\top }\mathbf {\Sigma }\boldsymbol{Z}. \end{aligned}$$

We note that when $\boldsymbol{Z}$ is distributed uniformly on the unit radius complex sphere instead, the variance formula is given in Tropp [20].

C Log-Determinant

Suppose $\mathbf {\Sigma }\in \mathbb {R}^{n\times n}$ is a PSD matrix. Let $\boldsymbol{Z}\sim \mathcal {N}(\mathbf {0},\mathbf {I}_n)$. Then, $\boldsymbol{Z}$ can be represented as $\boldsymbol{Z}==R\boldsymbol{\varTheta }$ with $\boldsymbol{\varTheta }$ being uniformly distributed on the surface of the unit radius n-dimensional sphere centered at the origin and $R\sim \chi _n$, independently. Then using the standard normal as a change of measure we get the following,

$$ \begin{aligned} \frac{1}{(2\pi )^{n/2}}\int \exp (-\boldsymbol{z}^\top \mathbf {\Sigma }^{-1}\boldsymbol{z}/2)\mathbf {d} \boldsymbol{Z}&=\mathbb {E} \left[ \exp (-\boldsymbol{Z}^\top \mathbf {\Sigma }^{-1}\boldsymbol{Z}/2+\Vert \boldsymbol{Z}\Vert ^2/2)\right] \\&=\mathbb {E} \left[ \exp (-R^2\boldsymbol{\varTheta }^\top \mathbf {\Sigma }^{-1}\boldsymbol{\varTheta }/2+R^2/2)\right] \\&=\frac{1}{2^{n/2}\varGamma (n/2)}\mathbb {E} \left[ \int _0^\infty r^{n/2-1}\exp (-r\boldsymbol{\varTheta }^\top \mathbf {\Sigma }^{-1}\boldsymbol{\varTheta }/2)\mathbf {d} r\right] \\&=\mathbb {E} \left[ (\boldsymbol{\varTheta }^\top \mathbf {\Sigma }^{-1}\boldsymbol{\varTheta })^{-n/2} \right] \\&\approx \frac{1}{M}\sum _{i=1}^M(\boldsymbol{\varTheta }_i^\top \mathbf {\Sigma }^{-1}\boldsymbol{\varTheta }_i)^{-n/2}. \end{aligned} $$

We use the following integral formula to evaluate the integral on line 3,

$$ \int _0^\infty r^{n/2-1}\exp (-r\alpha /2)\mathbf {d} r=\alpha ^{-n/2}2^{n/2}\varGamma (n/2). $$

Thus a conditional Monte Carlo estimate for the log-determinant is,

$$\begin{aligned} \widehat{\ln |\mathbf {\Sigma }|}=2\ln \left[ \frac{1}{M}\sum _{i=1}^M(\boldsymbol{\varTheta }_i^\top \mathbf {\Sigma }^{-1}\boldsymbol{\varTheta }_i)^{-n/2}\right] . \end{aligned}$$