Kernel discriminant analysis and clustering with parsimonious Gaussian process models

Abstract

This work presents a family of parsimonious Gaussian process models which allow to build, from a finite sample, a model-based classifier in an infinite dimensional space. The proposed parsimonious models are obtained by constraining the eigen-decomposition of the Gaussian processes modeling each class. This allows in particular to use non-linear mapping functions which project the observations into infinite dimensional spaces. It is also demonstrated that the building of the classifier can be directly done from the observation space through a kernel function. The proposed classification method is thus able to classify data of various types such as categorical data, functional data or networks. Furthermore, it is possible to classify mixed data by combining different kernels. The methodology is as well extended to the unsupervised classification case and an EM algorithm is derived for the inference. Experimental results on various data sets demonstrate the effectiveness of the proposed method. A Matlab toolbox implementing the proposed classification methods is provided as supplementary material.

Appendix: Proofs

Proof of Proposition 1

Recalling that $d_{\max }=\max (d_{1},\ldots , d_{k})$, the classification function can be rewritten as:

$$\begin{aligned} D_{i}(\varphi (x))&= \sum _{j=1}^{r}\frac{1}{\lambda _{ij}}\langle \varphi (x)-\mu _{i},q_{ij}\rangle _{L_{2}}^{2}\\&+\sum _{j=1}^{d_{i}}\log (\lambda _{ij})\!+\!\sum _{j=d_{i}+1}^{d_{\max }}\log (\lambda )\!-\!2\log (\pi _{i})+\gamma , \end{aligned}$$

where $\gamma =(r-d_{\max })\log (\lambda )$ is a constant term which does not depend on the index $i$ of the class. In view of the assumptions, $D_{i}(\varphi (x))$ can be also rewritten as:

$$\begin{aligned} D_{i}(\varphi (x))&= \sum _{j=1}^{d_{i}}\frac{1}{\lambda _{ij}}\langle \varphi (x)-\mu _{i},q_{ij}\rangle _{L_{2}}^{2}\\&+\frac{1}{\lambda }\sum _{j=d_{i}+1}^{r}\langle \varphi (x)-\mu _{i},q_{ij}\rangle _{L_{2}}^{2}\\&+ \sum _{j=1}^{d_{i}}\log (\lambda _{ij})+(d_{\max }-d_{i})\log (\lambda )\\&-2\log (\pi _{i})+\gamma . \end{aligned}$$

Introducing the norm $||.||_{L_{2}}$ associated with the scalar product $\langle .,.\rangle _{L_{2}}$ and in view of Proposition 1 of (Shorack and Wellner (1986), p. 208), we finally obtain:

$$\begin{aligned} D_{i}(\varphi (x))&= \sum _{j=1}^{d_{i}}\left( \frac{1}{\lambda _{ij}}-\frac{1}{\lambda }\right) \langle \varphi (x)-\mu _{i},q_{ij}\rangle _{L_{2}}^{2}\\&+\frac{1}{\lambda }||\varphi (x)-\mu _{i}||_{L_{2}}^{2}\\&+ \sum _{j=1}^{d_{i}}\log (\lambda _{ij})+(d_{\max }-d_{i})\log (\lambda )\\&-2\log (\pi _{i})+\gamma , \end{aligned}$$

which is the desired result. $\square$

Proof of Proposition 2

The proof involves three steps.

• (i) Computation of the projection $\langle \varphi (x)-\hat{\mu }_{i},\hat{q}_{ij}\rangle _{L_{2}}$ : Since $(\hat{\lambda }_{ij},\hat{q}_{ij})$ is solution of the Fredholm-type equation, it follows that, for all $t\in J$,

  $$\begin{aligned} \hat{\lambda }_{ij}\hat{q}_{ij}(t)&= \int _{J}\hat{{\varSigma }}_{i}(s,t)\hat{q}_{ij}(s)ds\nonumber \\&= \frac{1}{n_{i}}\sum _{x_{\ell }\in C_{i}}\langle \varphi (x_{\ell })-\hat{\mu }_{i},\hat{q}_{ij}\rangle _{L_{2}}(\varphi (x_{\ell })(t)\nonumber \\&-\hat{\mu }_{i}(t)). \end{aligned}$$

  (8)

  This implies that $\hat{q}_{ij}$ lies in the linear subspace spanned by the $(\varphi (x_{\ell })-\hat{\mu }_{i}),\,x_{\ell }\in C_{i}$. As a consequence, the rank of the operator $\hat{{\varSigma }}_{i}$ is finite and is at most $r_{i}=\min (n_{i},r)$. It therefore exists $\beta _{ij\ell }\in {\mathbb {R}}$ such that:

  $$\begin{aligned} \hat{q}_{ij}=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(\varphi (x_{\ell })-\hat{\mu }_{i}) \end{aligned}$$

  (9)

  leading to:

  $$\begin{aligned} \langle \varphi (x)-\hat{\mu }_{i},\hat{q}_{ij}\rangle _{L_{2}}=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }\rho _{i}(x,x_{\ell }), \end{aligned}$$

  (10)

  for all $j=1,\ldots ,r_{i}$. The estimated classification function has therefore the following form:

  $$\begin{aligned} \hat{D}_{i}(\varphi (x))&=\frac{1}{n_{i}}\sum _{j=1}^{d_{i}}\frac{1}{\hat{\lambda }_{ij}}\left( \frac{1}{\hat{\lambda }_{ij}}-\frac{1}{\hat{\lambda }}\right) \left( \sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }\rho _{i}(x,x_{\ell })\right) ^{2}\\&\qquad +\frac{1}{\hat{\lambda }}\rho _{i}(x,x)\\&\qquad +\sum _{j=1}^{d_{i}}\log (\hat{\lambda }_{ij})+(d_{\max }-d_{i})\log (\hat{\lambda })-2\log (\hat{\pi }_{i}), \end{aligned}$$

  for all $i=1,\ldots ,k$.

• (ii) Computation of the $\beta _{ij\ell }$ and $\hat{\lambda }_{ij}$: Replacing (9) in the Fredholm-type equation (8) it follows that

  $$\begin{aligned}&\frac{1}{n_{i}}\sum _{x_{\ell },x_{\ell '}\in C_{i}}\beta _{ij\ell '}(\varphi (x_{\ell })-\hat{\mu }_{i})\rho _{i}(x_{\ell },x_{\ell '})\\&\qquad \qquad =\hat{\lambda }_{ij}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}(\varphi (x_{\ell '})-\hat{\mu }_{i}). \end{aligned}$$

  Finally, projecting this equation on $\varphi (x_{m})-\hat{\mu }_{i}$ for $x_{m}\in C_{i}$ yields

  $$\begin{aligned}&\frac{1}{n_{i}}\sum _{x_{\ell },x_{\ell '}\in C_{i}}\beta _{ij\ell '}\rho _{i}(x_{\ell },x_{m})\rho _{i}(x_{\ell },x_{\ell '})\\&\qquad \qquad =\hat{\lambda }_{ij}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}\rho _{i}(x_{m},x_{\ell '}). \end{aligned}$$

  Recalling that $M_{i}$ is the matrix $n_{i}\times n_{i}$ defined by $(M_{i})_{\ell ,\ell '}=\rho _{i}(x_{\ell },x_{\ell '})/n_{i}$ and introducing $\beta _{ij}$ the vector of ${\mathbb {R}}^{n_{i}}$ defined by $(\beta _{ij})_{\ell }=\beta _{ij\ell }$, the above equation can be rewritten as $M_{i}^{2}\beta _{ij}=\hat{\lambda }_{ij}M_{i}\beta _{ij}$ or, after simplification $M_{i}\beta _{ij}=\hat{\lambda }_{ij}\beta _{ij}.$ As a consequence, $\hat{\lambda }_{ij}$ is the $j$th largest eigenvalue of $M_{i}$ and $\beta _{ij}$ is the associated eigenvector for all $1\le j\le d_{i}$. Let us note that the constraint $\Vert \hat{q}_{ij}\Vert =1$ can be rewritten as $\beta _{ij}^{t}\beta _{ij}=1$.

• (iii) Computation of $\hat{\lambda }$: Remarking that trace $(\hat{{\varSigma }}_{i})\!=\!\text{ trace }(M_{i}) +{{\sum _{j=r_{i}+1}^{r}\hat{\lambda }_{ij}}}$, it follows:

  $$\begin{aligned} \hat{\lambda }=\frac{1}{\sum _{i=1}^{k}\hat{\pi _{i}}(r_{i}-d_{i})}\sum _{i=1}^{k}\hat{\pi }_{i}\left( {\mathrm {trace}}(M_{i})-\sum _{j=1}^{d_{i}}\hat{\lambda }_{ij}\right) , \end{aligned}$$

  and the proposition is proved. $\square$

Proof of Proposition 3

It is sufficient to prove that $\hat{q}_{ij}$ and $\hat{\lambda }_{ij}$ are respectively the $j$th normed eigenvector and eigenvalue of $\hat{{\varSigma }}_{i}$. First,

$$\begin{aligned} \hat{{\varSigma }}_{i}\hat{q}_{ij}&=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\frac{1}{n_{i}}\sum _{x_{\ell '}\in C_{i}}(x_{\ell '}-\bar{\mu }_{i})(x_{\ell '}\\&\quad -\bar{\mu }_{i})^{t}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(x_{\ell }-\bar{\mu }_{i})\\&=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\frac{1}{n_{i}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }(x_{\ell '}-\bar{\mu }_{i})(x_{\ell '}\\&\quad \quad -\bar{\mu }_{i})^{t}(x_{\ell }-\bar{\mu }_{i})\\&=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\frac{1}{n_{i}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }(x_{\ell '}-\bar{\mu }_{i})\rho _{i}(x_{\ell },x_{\ell '})\\&=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}(M_{i})_{\ell ,\ell '}\beta _{ij\ell }(x_{\ell '}-\bar{\mu }_{i})\\&=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell '}\in C_{i}}(M_{i}\beta _{ij})_{\ell '}(x_{\ell '}-\bar{\mu }_{i}), \end{aligned}$$

and remarking that $\beta _{ij}$ is eigenvector of $M_{i}$, it follows:

$$\begin{aligned} \hat{{\varSigma }}_{i}\hat{q}_{ij}=\hat{\lambda }_{ij}\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}(x_{\ell '}-\bar{\mu }_{i})=\hat{\lambda }_{ij}\hat{q}_{ij}. \end{aligned}$$

Second, straightforward algebra shows that

$$\begin{aligned} ||\hat{q}_{ij}||^{2}&=\frac{1}{n_{i}\hat{\lambda }_{ij}}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(x_{\ell })-\bar{\mu }_{i})^{t}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}(x_{\ell '}-\bar{\mu }_{i})\\&=\frac{1}{n_{i}\hat{\lambda }_{ij}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }\beta _{ij\ell '}(x_{\ell }-\bar{\mu }_{i})^{t}(x_{\ell '}-\bar{\mu }_{i})\\&=\frac{1}{\hat{\lambda }_{ij}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}(M_{i})_{\ell ,\ell '}\beta _{ij\ell }\beta _{ij\ell '}\\&=\frac{1}{\hat{\lambda }_{ij}}\hat{q}_{ij}^{t}M_{i}\hat{q}_{ij}=1, \end{aligned}$$

and the result is proved. $\square$

Proof of Proposition 4

For all $\ell =1,\ldots ,L$, the $\ell$th coordinate of the mapping function $\varphi (x)$ is defined as the $\ell$th coordinate of the function $x$ expressed in the truncated basis $\{b_{1},\ldots ,b_{L}\}$. More specifically,

$$\begin{aligned} x(t)=\sum _{\ell =1}^{L}\varphi _{\ell }(x)b_{\ell }(t), \end{aligned}$$

for all $t\in [0,1]$ and thus, for all $j=1,\ldots ,L$, we have

$$\begin{aligned} \gamma _{j}(x)&= \int _{0}^{1}x(t)b_{j}(t)dt \\&= \sum _{\ell =1}^{L}\varphi _{\ell }(x)\int _{0}^{1}b_{j}(t)b_{\ell }(t)dt=\sum _{\ell =1}^{L}B_{j\ell }\varphi _{\ell }(x). \end{aligned}$$

As a consequence, $\varphi (x)=B^{-1}\gamma (x)$ and $K(x,y)=\gamma (x)^{t}B^{-1}\gamma (y)$. Introducing

$$\begin{aligned} \bar{\gamma }_{i}=\frac{1}{n_{i}}\sum _{x_{j}\in C_{i}}\gamma (x_{j}), \end{aligned}$$

it follows that $\rho _{i}(x,y)=(\gamma (x)-\bar{\gamma }_{i})^{t}B^{-1}(\gamma (y)-\bar{\gamma }_{i})$. Let us first show that $\hat{q}_{ij}$ is eigenvector of $B^{-1}\hat{{\varSigma }}_{i}$. Recalling that

$$\begin{aligned} \hat{q}_{ij}=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(\gamma (x_{\ell })-\bar{\gamma }_{i}), \end{aligned}$$

we have

$$\begin{aligned} B^{-1}\hat{{\varSigma }}_{i}\hat{q}_{ij}&= \frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\frac{1}{n_{i}}\sum _{x_{\ell '}\in C_{i}}(\gamma (x_{\ell '})-\bar{\gamma }_{i})(\gamma (x_{\ell '})\\&\quad -\bar{\gamma }_{i})^{t}B^{-1}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(\gamma (x_{\ell })-\bar{\gamma }_{i})\\&= \frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\frac{1}{n_{i}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }(\gamma (x_{\ell '})\\&-\bar{\gamma }_{i})(\gamma (x_{\ell '})-\bar{\gamma }_{i})^{t}B^{-1}(\gamma (x_{\ell })-\bar{\gamma }_{i})\\&= \frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\frac{1}{n_{i}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }(\gamma (x_{\ell '})\\&-\bar{\gamma }_{i})\rho _{i}(x_{\ell },x_{\ell '})\\&= \frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell '},x_{\ell }\in C_{i}}(M_{i})_{\ell ,\ell '}\beta _{ij\ell }(\gamma (x_{\ell '})-\bar{\gamma }_{i})\\&= \frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell '}\in C_{i}}(M_{i}\beta _{ij})_{\ell '}(\gamma (x_{\ell '})-\bar{\gamma }_{i}). \end{aligned}$$

Remarking that $\beta _{ij}$ is eigenvector of $M_{i}$, it follows:

$$\begin{aligned} B^{-1}\hat{{\varSigma }}_{i}\hat{q}_{ij}&= \hat{\lambda }_{ij}\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}B^{-1}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}(\gamma (x_{\ell '})-\bar{\gamma }_{i})\\&= \hat{\lambda }_{ij}\hat{q}_{ij}. \end{aligned}$$

Let us finally compute the norm of $\hat{q}_{ij}$:

$$\begin{aligned} ||\hat{q}_{ij}||^{2}&=\frac{1}{n_{i}\hat{\lambda }_{ij}}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }(\gamma (x_{\ell })\\&\quad -\bar{\gamma }_{i})^{t}B^{-1}\sum _{x_{\ell '}\in C_{i}}\beta _{ij\ell '}(\gamma (x_{\ell '})-\bar{\gamma }_{i})\\&=\frac{1}{n_{i}\hat{\lambda }_{ij}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}\beta _{ij\ell }\beta _{ij\ell '}(\gamma (x_{\ell })\\&\quad -\bar{\gamma }_{i})^{t}B^{-1}(\gamma (x_{\ell '})-\bar{\gamma }_{i})\\&=\frac{1}{\hat{\lambda }_{ij}}\sum _{x_{\ell '},x_{\ell }\in C_{i}}(M_{i})_{\ell ,\ell '}\beta _{ij\ell }\beta _{ij\ell '}\\&=\frac{1}{\hat{\lambda }_{ij}}\hat{q}_{ij}^{t}M_{i}\hat{q}_{ij}=1, \end{aligned}$$

and the result is proved. $\square$

Proof of Proposition 5

Let us first set the estimate of $\mu _{i}$at iteration $s$ to its empirical counterpart conditionally on the current value of the model parameter $\theta ^{(s-1)}$:

$$\begin{aligned} \hat{\mu }_{i}^{(s)}(t)=\frac{1}{n_{i}^{(s)}}\sum _{j=1}^{n}t_{ij}^{(s-1)}\varphi (x_{j})(t), \end{aligned}$$

where $n_{i}^{(s-1)}=\sum _{j=1}^{n}t_{ij}^{(s-1)}$. Replacing $\hat{\mu }_{i}$ by $\hat{\mu }_{i}^{(s)}$ in the proof of Proposition 2 , we get:

$$\begin{aligned} \hat{q}_{ij}=\frac{1}{\sqrt{n_{i}\hat{\lambda }_{ij}}}\sum _{x_{\ell }\in C_{i}}\beta _{ij\ell }\sqrt{t_{\ell i}}(\varphi (x_{\ell })-\hat{\mu }_{i}). \end{aligned}$$

(11)

According to Bayes’ rule and Eq. (1), the computation of $t_{ij}^{(s)}={\mathbb {E}}(Z_{j}=i|x_{j},\theta ^{(s-1)})$ conditionally on the current value of the model parameter $\theta ^{(s-1)}$ can be finally written as:

$$\begin{aligned} t_{ij}^{(s)}&= {\mathbb {E}}(Z_{j}=i|x_{j},\theta ^{(s-1)})=P(Z_{j}=i|x_{j},\theta ^{(s-1)})\\&= {{\frac{\exp \left( -\frac{1}{2}D_{i}^{(s-1)}\left( \varphi (x_{j})\right) \right) }{\sum _{\ell =1}^{k}\exp \left( -\frac{1}{2}D_{\ell }^{(s-1)}\left( \varphi (x_{j})\right) \right) }},} \end{aligned}$$

for $j=1,\ldots ,n$ and $i=1,\ldots ,k$. The result follows. $\square$