Multiple instance learning via Gaussian processes

Abstract

Multiple instance learning (MIL) is a binary classification problem with loosely supervised data where a class label is assigned only to a bag of instances indicating presence/absence of positive instances. In this paper we introduce a novel MIL algorithm using Gaussian processes (GP). The bag labeling protocol of the MIL can be effectively modeled by the sigmoid likelihood through the max function over GP latent variables. As the non-continuous max function makes exact GP inference and learning infeasible, we propose two approximations: the soft-max approximation and the introduction of witness indicator variables. Compared to the state-of-the-art MIL approaches, especially those based on the Support Vector Machine, our model enjoys two most crucial benefits: (i) the kernel parameters can be learned in a principled manner, thus avoiding grid search and being able to exploit a variety of kernel families with complex forms, and (ii) the efficient gradient search for kernel parameter learning effectively leads to feature selection to extract most relevant features while discarding noise. We demonstrate that our approaches attain superior or comparable performance to existing methods on several real-world MIL datasets including large-scale content-based image retrieval problems.

Appendix: Gaussian process models and approximate inference

We review the GP regression and classification models as well as the Laplace method for approximate inference.

1.1 Gaussian process regression

When the output variable $y$ is a real-valued scalar, one natural likelihood model from $f$ to $y$ is the additive Gaussian noise model, namely

$$\begin{aligned} P(y|f) = \fancyscript{N}(y;f,\sigma ^2) = \frac{1}{\sqrt{2\pi \sigma ^2}} \exp \bigg (-\frac{(y-f)^2}{2\sigma ^2}\bigg ), \end{aligned}$$

(33)

where $\sigma ^2$, the output noise variance, is another set of hyperparameters (together with the kernel parameters $\varvec{\beta }$). Given the training data $\{(\mathbf{x}_i,y_i)\}_{i=1}^n$ and with (33), we can compute the posterior distribution for the training latent variables $\mathbf{f}$ analytically, namely

$$\begin{aligned} P(\mathbf{f}|\mathbf{y},\mathbf{X}) \propto P(\mathbf{y}|\mathbf{f}) P(\mathbf{f}|\mathbf{X}) \!=\! \fancyscript{N}(\mathbf{f}; \mathbf{K}(\mathbf{K}+\sigma ^2\mathbf{I})^{-1}\mathbf{y}, (\mathbf{I} \!-\! \mathbf{K}(\mathbf{K}\!+\!\sigma ^2\mathbf{I})^{-1})\mathbf{K}).\qquad \quad \end{aligned}$$

(34)

When the posterior of $\mathbf{f}$ is obtained, we can readily compute the predictive distribution for a test output $y_*$ on $\mathbf{x}_*$. We first derive the posterior for $f_*$, the latent variable for the test point, by marginalizing out $\mathbf{f}$ as follows:

$$\begin{aligned} P(f_*|\mathbf{x}_*,\mathbf{y},\mathbf{X})&= \int P(f_*|\mathbf{x}_*,\mathbf{f},\mathbf{X})P(\mathbf{f}|\mathbf{y},\mathbf{X}) d\mathbf{f} \nonumber \\&= \fancyscript{N}(f_*; \mathbf{k}(\mathbf{x}_*)^{\top } (\mathbf{K}+\sigma ^2\mathbf{I})^{-1} \mathbf{y}, k(\mathbf{x}_*,\mathbf{x}_*) \nonumber \\&- \mathbf{k}(\mathbf{x}_*)^{\top } (\mathbf{K}+\sigma ^2\mathbf{I})^{-1} \mathbf{k}(\mathbf{x}_*)). \end{aligned}$$

(35)

Then it is not difficult to see that the posterior for $y_*$ can be obtained by marginalizing out $f_*$, namely

$$\begin{aligned} P(y_*|\mathbf{x}_*,\mathbf{y},\mathbf{X})&= \int P(y_*|f_*) P(f_*|\mathbf{x}_*,\mathbf{y},\mathbf{X}) d f_* \nonumber \\&= \fancyscript{N}(y_*; \mathbf{k}(\mathbf{x}_*)^{\top } (\mathbf{K}+\sigma ^2\mathbf{I})^{-1} \mathbf{y}, k(\mathbf{x}_*,\mathbf{x}_*) \nonumber \\&- \mathbf{k}(\mathbf{x}_*)^{\top } (\mathbf{K}+\sigma ^2\mathbf{I})^{-1} \mathbf{k}(\mathbf{x}_*) + \sigma ^2). \end{aligned}$$

(36)

So far, we have assumed that the hyperparameters (denoted by $\varvec{\theta }=\{{\varvec{\beta }}, \sigma ^2\}$) of the GP model are known. However, it is a very important issue to estimate $\varvec{\theta }$ from the data, the task often known as the GP learning. Following the fully Bayesian treatment, it would be ideal to place a prior on $\varvec{\theta }$ and compute a posterior distribution for $\varvec{\theta }$ as well, however, due to the difficulty of the integration, it is usually handled by the evidence maximization, also referred to as the empirical Bayes. The evidence in this case is the data likelihood,

$$\begin{aligned} P(\mathbf{y}|\mathbf{X},{\varvec{\theta }}) = \int P(\mathbf{y}|\mathbf{f},\sigma ^2) P(\mathbf{f}|\mathbf{X},{\varvec{\beta }}) d\mathbf{f} = \fancyscript{N}(\mathbf{y}; \mathbf{0}, \mathbf{K}_{\varvec{\beta }} + \sigma ^2\mathbf{I}). \end{aligned}$$

(37)

We then maximize (37), which is equivalent to solving the following optimization problem:

$$\begin{aligned} {\varvec{\theta }}^* = \arg \max _{\varvec{\theta }} \ \log P(\mathbf{y}|\mathbf{X},{\varvec{\theta }}) = \arg \min _{\varvec{\theta }} \ |\mathbf{K}_{\varvec{\beta }} + \sigma ^2\mathbf{I}| + \mathbf{y}^{\top } (\mathbf{K}_{\varvec{\beta }} + \sigma ^2\mathbf{I})^{-1} \mathbf{y}. \end{aligned}$$

(38)

(38) is non-convex in general, and one can find a local minimum using the quasi-Newton or the conjugate gradient search.

1.2 Gaussian process classification

In the classification setting where the output variable $y$ takes a binary¹¹ value from $\{+1,-1\}$, we have several choices for the likelihood model $P(y|f)$. Two most popular ways to link the real-valued variable $f$ to the binary $y$ are the sigmoid and the probit.

$$\begin{aligned} P(y|f) = \left\{ \begin{array}{ll} \frac{1}{1+\exp (-yf)} &{} \text {(sigmoid)} \\ \varPhi (yf) &{} \text {(probit)} \end{array} \right. \end{aligned}$$

(39)

where $\varPhi (\cdot )$ is the cumulative normal function.

We let $l(y,f;{\varvec{\gamma }}) = -\log P(y|f,{\varvec{\gamma }})$, where ${\varvec{\gamma }}$ denotes the (hyper)parameters of the likelihood model.¹² Likewise, we often drop the dependency on ${\varvec{\gamma }}$ for the notational simplicity.

Unlike the regression case, it is unfortunate that the posterior distribution $P(\mathbf{f}|\mathbf{y},\mathbf{X})$ has no analytic form as it is a product of the non-Gaussian $P(\mathbf{y}|\mathbf{f})$ and the Gaussian $P(\mathbf{f}|\mathbf{X})$. One can consider three standard approximation schemes within the Bayesian framework: (i) Laplace approximation, (ii) variational methods, and (iii) sampling-based approaches (e.g., MCMC). It is often the cases that the third method is avoided due to the computational overhead (as we sample $\mathbf{f}$s, $n$-dimensional vectors, where $n$ is the number of training samples). In the below, we briefly review the Laplace approximation.

1.3 Laplace approximation

The Laplace approximation essentially replaces the product $P(\mathbf{y}|\mathbf{f}) P(\mathbf{f}|\mathbf{X})$ by a Gaussian with the mean equal to the mode of the product, and the covariance equal to the inverse Hessian of the product evaluated at the mode. More specifically, we let

$$\begin{aligned} S(\mathbf{f})&= -\log \big ( P(\mathbf{y}|\mathbf{f}) P(\mathbf{f}|\mathbf{X}) \big ) = \sum _{i=1}^n l(y_i,f_i) + \frac{1}{2} \mathbf{f}^{\top } \mathbf{K}^{-1} \mathbf{f} \nonumber \\&+ \frac{1}{2} \log |\mathbf{K}| + \frac{n}{2}\log 2 \pi . \end{aligned}$$

(40)

Note that (40) is a convex function of $\mathbf{f}$ since the Hessian of $S(\mathbf{f})$, ${\varvec{\varLambda }}+\mathbf{K}^{-1}$, is positive definite where ${\varvec{\varLambda }}$ is the diagonal matrix with entries $[{\varvec{\varLambda }}]_{ii} = \frac{\partial ^2 l(y_i,f_i)}{\partial f_i^2}$. The minimum of $S(\mathbf{f})$ can be attained by a gradient search. We denote the optimum by $\mathbf{f}^{MAP} = \arg \max _{\mathbf{f}} S(\mathbf{f})$. Letting ${\varvec{\varLambda }}^{MAP}$ be ${\varvec{\varLambda }}$ evaluated at $\mathbf{f}^{MAP}$, we approximate $S(\mathbf{f})$ by the following quadratic function (i.e., using the Taylor expansion)

$$\begin{aligned} S(\mathbf{f}) \approx S(\mathbf{f}^{MAP}) + \frac{1}{2} (\mathbf{f}-\mathbf{f}^{MAP})^{\top } ({\varvec{\varLambda }}^{MAP}+\mathbf{K}^{-1}) (\mathbf{f}-\mathbf{f}^{MAP}), \end{aligned}$$

(41)

which essentially leads to a Gaussian approximation for $P(\mathbf{f}|\mathbf{y},\mathbf{X})$, namely

$$\begin{aligned} P(\mathbf{f}|\mathbf{y},\mathbf{X}) \approx \fancyscript{N}(\mathbf{f}; \mathbf{f}^{MAP}, ({\varvec{\varLambda }}^{MAP}+\mathbf{K}^{-1})^{-1}). \end{aligned}$$

(42)

The data likelihood (i.e., evidence) immediately follows from the similar approximation,

$$\begin{aligned} P(\mathbf{y}|\mathbf{X},{\varvec{\theta }}) \approx \exp (-S(\mathbf{f}^{MAP})) (2\pi )^{n/2} |{\varvec{\varLambda }}^{MAP} + \mathbf{K}_{\varvec{\beta }}^{-1}|^{-1/2}, \end{aligned}$$

(43)

which can be maximized by gradient search with respect to the hyperparameters ${\varvec{\theta }}=\{{\varvec{\beta }},{\varvec{\gamma }}\}$.

Using the Gaussian approximated posterior, it is easy to derive the predictive distribution for $f_*$:

$$\begin{aligned} P(f_*|\mathbf{x}_*,\mathbf{y},\mathbf{X})&= \fancyscript{N}(f_*; \mathbf{k}(\mathbf{x}_*)^{\top } \mathbf{K}^{-1} \mathbf{f}^{MAP}, k(\mathbf{x}_*,\mathbf{x}_*) \nonumber \\&- \mathbf{k}(\mathbf{x}_*)^{\top } (({\varvec{\varLambda }}^{MAP})^{-1}+\mathbf{K}^{-1})^{-1} \mathbf{k}(\mathbf{x}_*)). \end{aligned}$$

(44)

Finally, the predictive distribution for $y_*$ can be obtained by marginalizing out $f_*$, which has a closed form solution for the probit likelihood model as follows¹³:

$$\begin{aligned} P(y_*=+1|\mathbf{x}_*,\mathbf{y},\mathbf{X}) = \varPhi \Big (\frac{\bar{\mu }}{\sqrt{1+\bar{\sigma }^2}}\Big ), \end{aligned}$$

(45)

where $\bar{\mu }$ and $\bar{\sigma }^2$ are the mean and the variance of $P(f_*|\mathbf{x}_*,\mathbf{y},\mathbf{X})$, respectively.