Model Selection for Gaussian Process Regression

Abstract

Gaussian processes are powerful tools since they can model non-linear dependencies between inputs, while remaining analytically tractable. A Gaussian process is characterized by a mean function and a covariance function (kernel), which are determined by a model selection criterion. The functions to be compared do not just differ in their parametrization but in their fundamental structure. It is often not clear which function structure to choose, for instance to decide between a squared exponential and a rational quadratic kernel. Based on the principle of posterior agreement, we develop a general framework for model selection to rank kernels for Gaussian process regression and compare it with maximum evidence (also called marginal likelihood) and leave-one-out cross-validation. Given the disagreement between current state-of-the-art methods in our experiments, we show the difficulty of model selection and the need for an information-theoretic approach.

N.S. Gorbach and A.A. Bian—These two authors contributed equally.

Appendices

Appendix

A Propositions of Gaussian distribution

This is a collection of properties related to Gaussian distributions for the derivations in Sect. 2.2.

Proposition 1

If

$$ \begin{bmatrix} {\varvec{t}} \\ {\varvec{u}} \end{bmatrix} \thicksim {{\mathrm{\mathcal {N}}}}\left( {\begin{bmatrix} {\varvec{\mu }} \\ {\varvec{r}} \end{bmatrix}, \begin{bmatrix} \varvec{\varSigma }&\varvec{A} \\ \varvec{A}^\intercal&\varvec{V} \end{bmatrix}}\right) $$

then

[19, Theorem 3.3.4].

Proposition 2

If $\varvec{\varLambda }$ is symmetric positive-definite, then

[20, 14].

Proposition 3

It holds that,

where ${\varvec{r}}= \sum _{k = 1}^{K} \varvec{\varSigma }_{k} ^ {-1} {\varvec{\mu }}_{k}$ and $\varvec{\varLambda }= \sum _{k = 1}^{K} \varvec{\varSigma }_{k} ^ {-1}$.

Proof

We shorten to move this factor $\gamma$ independent of ${\varvec{x}}$ out of the integral as in

The remaining integral can be calculated by Proposition 2.

Proposition 4

If $\varvec{\varSigma }$ is symmetric positive-definite, then $\varvec{\varSigma }$ is invertible and $\varvec{\varSigma } ^ {-1}$ is symmetric positive-definite [12, 430].

Proposition 5

If $\varvec{\varSigma }$ is symmetric positive-definite and $\varvec{A}$ has full row rank, then $\varvec{A}\varvec{\varSigma }\varvec{A}^\intercal$ is symmetric positive-definite [12, Observation 7.1.8.(b)].

Proposition 6

For $\varvec{A}\in \mathbb {R}^ {D \times N}$ of full row rank, the density

has the equivalent form as , where ${\varvec{r}}= \varvec{A}\varvec{\varSigma } ^ {-1} {\varvec{\mu }}$ and $\varvec{\varLambda }= \varvec{A}\varvec{\varSigma } ^ {-1} \varvec{A}^\intercal$.

Proof

First, we separate a factor independent of ${\varvec{x}}$ in

Therefore,

$$ p\left( {{\varvec{x}}}\right) = \frac{\exp \left( {{\varvec{x}}^\intercal \left( {{\varvec{r}}- \frac{1}{2} \varvec{\varLambda }{\varvec{x}}}\right) }\right) }{\int _{\mathbb {R}^ {D}}^{} \exp \left( {{\varvec{x}}^\intercal \left( {{\varvec{r}}- \frac{1}{2} \varvec{\varLambda }{\varvec{x}}}\right) }\right) \,\text{ d }^ {D} {\varvec{x}}}. $$

We now calculate the integral. From Proposition 4 and Proposition 5, one can see that $\varvec{\varLambda }$ is symmetric positive-definite, so that Proposition 2 can be applied to find

Finally, one gets