From Covariance Matrices to Covariance Operators: Data Representation from Finite to Infinite-Dimensional Settings

Abstract

This chapter presents some of the recent developments in the generalization of the data representation framework using finite-dimensional covariance matrices to infinite-dimensional covariance operators in Reproducing Kernel Hilbert Spaces (RKHS). We show that the proper mathematical setting for covariance operators is the infinite-dimensional Riemannian manifold of positive definite Hilbert–Schmidt operators, which are the generalization of symmetric, positive definite (SPD) matrices. We then give the closed form formulas for the affine-invariant and Log-Hilbert–Schmidt distances between RKHS covariance operators on this manifold, which generalize the affine-invariant and Log-Euclidean distances, respectively, between SPD matrices. The Log-Hilbert–Schmidt distance in particular can be used to design a two-layer kernel machine, which can be applied directly to a practical application, such as image classification. Experimental results are provided to illustrate the power of this new paradigm for data representation.

Appendices

Appendix

Proofs of Mathematical Results

Proof

(of Proposition 1) For the first kernel, we have the property that the sum and product of positive definite kernels are also positive definite. Thus from the positivity of the inner product $\langle A, B\rangle _{F}$, it follows that $K(A,B) = (c + \langle A, B\rangle _{\mathrm{logE}})^d$ is positive definite, as in the Euclidean setting.

For the second kernel, since $(\mathrm{{Sym}}^{++}(n), \odot , \circledast , \langle \;, \;\rangle _{\mathrm{logE}})$ is an inner product space, it follows that the kernel

$$ K(A,B) = \exp (-d_{\mathrm{logE}}^p(A,B) /\sigma ^2) = \exp (-||\log (A) - \log (B)||_{F}^p /\sigma ^2) $$

is positive definite for $0 < p \le 2$ by a classical result due to Schoenberg on positive definite functions and the imbeddability of metric spaces into Hilbert spaces (see [35], Theorem 1 and Corollary 1).

Proof

(of Lemma 1) Recall that we have $\varPhi (\mathbf {x})^{*}\varPhi (\mathbf {x}) = K[\mathbf {x}]$, $\varPhi (\mathbf {y})^{*}\varPhi (\mathbf {y}) = K[\mathbf {y}]$, $\varPhi (\mathbf {x})^{*}\varPhi (\mathbf {y}) = K[\mathbf {x},\mathbf {y}]$. By definition of the Hilbert–Schmidt norm and property of the trace operation, we have

$$\begin{aligned}&||C_{\varPhi (\mathbf {x})} - C_{\varPhi (\mathbf {y})}||_{\mathrm{HS}}^2 = \left\| \frac{1}{m}\varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*} - \frac{1}{m}\varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}\right\| ^2_{\mathrm{HS}} \\&= \frac{1}{m^2}||\varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*}||^2_{\mathrm{HS}} - \frac{2}{m^2}\langle \varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*}, \varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}\rangle _{\mathrm{HS}} \\&+ \frac{1}{m^2}||\varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}||^2_{\mathrm{HS}} \\&= \frac{1}{m^2}\mathrm{tr}[\varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*}\varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*}] - \frac{2}{m^2}\mathrm{tr}[\varPhi (\mathbf {x})J_m\varPhi (\mathbf {x})^{*}\varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}] \\&+ \frac{1}{m^2}\mathrm{tr}[\varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}\varPhi (\mathbf {y})J_m\varPhi (\mathbf {y})^{*}] \\&= \frac{1}{m^2}\mathrm{tr}[(K[\mathbf {x}]J_m)^2- 2K[\mathbf {y},\mathbf {x}]J_mK[\mathbf {x},\mathbf {y}]J_m + (K[\mathbf {y}]J_m)^2] \\&= \frac{1}{m^2}[\langle J_mK[\mathbf {x}], K[\mathbf {x}]J_m\rangle _F -2 \langle J_mK[\mathbf {x},\mathbf {y}], K[\mathbf {x},\mathbf {y}]J_m\rangle _F + \langle J_mK[\mathbf {y}], K[\mathbf {y}]J_m\rangle _F]. \end{aligned}$$

This completes the proof of the lemma.   $\square$

Lemma 2

Let B be a constant with $0< B < 1$. Then for all $|x| \le B$,

$$\begin{aligned} |\log (1+x)| \le \frac{1}{1-B}|x|. \end{aligned}$$

(5.65)

Proof

For $x \ge 0$, we have the well-known inequality $0 \le \log (1+x) \le x$, so clearly $0 \le \log (1+x) < \frac{1}{1-B}x$. Consider now the case $-B \le x \le 0$. Let

$$\begin{aligned} f(x) = \log (1+x) - \frac{1}{1-B}x. \end{aligned}$$

We have

$$\begin{aligned} f^{'}(x) = \frac{1}{1+x} - \frac{1}{1-B} \le 0, \end{aligned}$$

with $f^{'}(-B) = 0$. Thus the function f is decreasing on $[-B, 0]$ and reaches its minimum at $x= 0$, which is $f(0) = 0$. Hence we have for all $-1 < -B \le x \le 0$

$$\begin{aligned} 0 \ge \log (1+x) \ge \frac{1}{1-B}x \Rightarrow |\log (1+x)| \le \frac{1}{1-B}|x|, \end{aligned}$$

as we claimed.    $\square$

Proof

(of Propositions 2 and 3) We first show that for an operator $A+\gamma I > 0$, where A is self-adjoint, compact, the operator $\log (A+\gamma I) \notin \mathrm{HS}(\mathscr {H})$ if $\gamma \ne 1$. Since A is compact, it has a countable spectrum $\{\lambda _k\}_{k\in \mathbb {N}}$, with $\lim _{k \rightarrow \infty }\lambda _k = 0$, so that $\lim _{k \rightarrow \infty }\log (\lambda _k +\gamma ) = \log (\gamma )$. Thus if $\gamma \ne 1$, so that $\log \gamma \ne 0$, we have

$$\begin{aligned} ||\log (A+\gamma I)||^2_{\mathrm{HS}} = \sum _{k=1}^{\infty }[\log (\lambda _k + \gamma )]^2 = \infty . \end{aligned}$$

Hence $\log (A+\gamma I) \notin \mathrm{HS}(\mathscr {H})$ if $\gamma \ne 1$.

Assume now that $\gamma = 1$. We show that $\log (A+I) \in \mathrm{HS}(\mathscr {H})$ if and only if $A \in \mathrm{HS}(\mathscr {H})$. For the first direction, assume that $B = \log (A+I) \in \mathrm{HS}(\mathscr {H})$. By definition, we have $A+ I = \exp (B) \Longleftrightarrow A = \exp (B) - I = \sum _{k=1}^{\infty }\frac{B^k}{k!}$, with

$$\begin{aligned} ||A||_{\mathrm{HS}} = \left\| \sum _{k=1}^{\infty }\frac{B^k}{k!}\right\| _{\mathrm{HS}} \le \sum _{k=1}^{\infty }\frac{||B||_{\mathrm{HS}}^k}{k!} = \exp (||B||_{\mathrm{HS}}) -1 < \infty . \end{aligned}$$

This shows that $A \in \mathrm{HS}$. Conversely, assume $A \in \mathrm{HS}(\mathscr {H})$, so that

$$\begin{aligned} ||A||_{\mathrm{HS}}^2 = \sum _{k=1}^{\infty }\lambda _k^2 < \infty , \end{aligned}$$

and that $A+I >0$, so that $\log (A+I)$ is well-defined and bounded, with eigenvalues $\{\log (\lambda _k+1)\}_{k=1}^{\infty }$. Since $\lim _{k \rightarrow \infty }\lambda _k = 0$, for any constant $0< \varepsilon < 1$, there exists $N = N(\varepsilon )$ such that $|\lambda _k| < \varepsilon$ $\forall k \ge N$. By Lemma 2, we have

$$\begin{aligned} ||\log (A+I)||^2_{\mathrm{HS}}&= \sum _{k=1}^{\infty }[\log (\lambda _k + 1)]^2 = \sum _{k=1}^{N-1}[\log (\lambda _k+1)]^2 + \sum _{k=N}^{\infty }[\log (\lambda _k+1)]^2 \\&\le \sum _{k=1}^{N-1}[\log (\lambda _k+1)]^2 + \frac{1}{1-\varepsilon }\sum _{k=N}^{\infty }\lambda _k^2 < \infty . \end{aligned}$$

This shows that $\log (A+I) \in \mathrm{HS}(\mathscr {H})$, which completes the proof.    $\square$

Proof

(Proof of Proposition 4) Since the identity operator I commutes with any operator A, we have the decomposition

$$\begin{aligned} \log (A+ \gamma I) = \log \left( \frac{A}{\gamma } + I\right) + (\log \gamma ) I. \end{aligned}$$

We first note that the operator $\log \left( \frac{A}{\gamma } + I\right)$ is compact, since it possesses a countable set of eigenvalues $\{\log (\frac{\lambda _k}{\gamma } + 1)\}_{k\in \mathbb {N}}$ satisfying $\lim _{k \rightarrow \infty }\log (\frac{\lambda _k}{\gamma } +1) = 0$.

If A is Hilbert–Schmidt, then by Proposition 2, we have $\log \left( \frac{A}{\gamma } + I\right) \in \mathrm{HS}(\mathscr {H})$, and thus $\log (A+\gamma I) \in \mathscr {H}_{\mathbb {R}}$. By definition of the extended Hilbert–Schmidt norm,

$$\begin{aligned} ||\log (A+ \gamma I)||^2_{\mathrm{eHS}} = \left\| \log \left( \frac{A}{\gamma } + I\right) \right\| ^2_{\mathrm{HS}} + (\log \gamma )^2 < \infty . \end{aligned}$$

Conversely, if $\log (A+\gamma I) \in \mathscr {H}_{\mathbb {R}}$, then together with the fact that $\log \left( \frac{A}{\gamma } + I\right)$ is compact, the above decomposition shows that we must have $\log \left( \frac{A}{\gamma } + I\right) \in \mathrm{HS}(\mathscr {H})$ and hence $A \in \mathrm{HS}(\mathscr {H})$ by Proposition 2.   $\square$

Proof

(of Proposition 5) Since $(A+\gamma I) > 0$, $(B+ \mu I) > 0$, it is straightforward to see that $(A+\gamma I)^{-1/2}(B+\mu I)(A+\gamma I)^{-1/2} > 0$. Using the identity

$$\begin{aligned} (A+ \gamma I)^{-1} = \frac{1}{\gamma }I -\frac{A}{\gamma }(A+\gamma I)^{-1}, \end{aligned}$$

we obtain

$$\begin{aligned}&(A+\gamma I)^{-1/2}(B+ \mu I)(A+\gamma I)^{-1/2} \\&= \frac{\mu }{\gamma }I + (A+\gamma I)^{-1/2}B(A+\gamma I)^{-1/2} - \frac{\mu }{\gamma }A(A+\gamma I)^{-1} = Z + \nu I, \end{aligned}$$

where $\nu = \frac{\mu }{\gamma }$ and $Z = (A+\gamma I)^{-1/2}B(A+\gamma I)^{-1/2} - \frac{\mu }{\gamma }A(A+\gamma I)^{-1}$. It is clear that $Z = Z^{*}$ and that $Z \in \mathrm{HS}(\mathscr {H})$, since $\mathrm{HS}(\mathscr {H})$ is a two-sided ideal in $\mathscr {L}(\mathscr {H})$. It follows that $\log (Z + \gamma I) \in \mathscr {H}_{\mathbb {R}}$ by Proposition 4. Thus the geodesic distance

$$\begin{aligned} d_{\mathrm{aiHS}}[(A+\gamma I), (B+ \mu I)]&= ||\log [(A+\gamma I)^{-1/2}(B+ \mu I)(A+\gamma I)^{-1/2}]||_{\mathrm{eHS}} \\&= ||\log (Z+\nu I)||_{\mathrm{eHS}} \end{aligned}$$

is always finite. Furthermore, by Proposition 2, $\log (\frac{Z}{\nu } +I) \in \mathrm{HS}(\mathscr {H})$ and thus by definition of the extended Hilbert–Schmidt norm, when $\dim (\mathscr {H}) = \infty$,

$$\begin{aligned} d^2_{\mathrm{aiHS}}[(A+\gamma I), (B+ \mu I)] = ||\log (Z+\nu I)||_{\mathrm{eHS}}^2 = ||\log \left( \frac{Z}{\nu }+I\right) ||^2_{\mathrm{HS}} + (\log \nu )^2. \end{aligned}$$

This completes the proof.   $\square$

Proof

(of Proposition 6) By Proposition 4, $(A+\gamma I), (B+\mu I) \in \Sigma (\mathscr {H}) \Longleftrightarrow \log (A+\gamma I), \log (B+ \mu I) \in \mathscr {H}_{\mathbb {R}}$ . It follows that $[\log (A+\gamma I) - \log (B+\mu I)] \in \mathscr {H}_{\mathbb {R}}$, so that $||\log (A+\gamma I) - \log (B+\mu I)||_{\mathrm{eHS}}$ is always finite.

Furthermore, by Proposition 2, $\log \left( \frac{A}{\gamma }+I\right) , \log \left( \frac{B}{\mu }+I\right) \in \mathrm{HS}(\mathscr {H})$ and by definition of the extended Hilbert–Schmidt norm, when $\dim (\mathscr {H}) = \infty$,

$$\begin{aligned}&||\log (A+\gamma I) - \log (B+ \mu I)||_{\mathrm{eHS}}^2 = \left\| \log \left( \frac{A}{\gamma } + I\right) - \log \left( \frac{B}{\mu } + I\right) + \left( \log \frac{\gamma }{\mu }\right) I\right\| ^2_{\mathrm{eHS}}\\&= \left\| \log \left( \frac{A}{\gamma } + I\right) - \log \left( \frac{B}{\mu } + I\right) \right\| ^2_{\mathrm{HS}} + \left( \log \frac{\gamma }{\mu }\right) ^2. \end{aligned}$$

This completes the proof.    $\square$