Singular Value Decomposition of Operators on Reproducing Kernel Hilbert Spaces

Abstract

Reproducing kernel Hilbert spaces (RKHSs) play an important role in many statistics and machine learning applications ranging from support vector machines to Gaussian processes and kernel embeddings of distributions. Operators acting on such spaces are, for instance, required to embed conditional probability distributions in order to implement the kernel Bayes rule and build sequential data models. It was recently shown that transfer operators such as the Perron–Frobenius or Koopman operator can also be approximated in a similar fashion using covariance and cross-covariance operators and that eigenfunctions of these operators can be obtained by solving associated matrix eigenvalue problems. The goal of this paper is to provide a solid functional analytic foundation for the eigenvalue decomposition of RKHS operators and to extend the approach to the singular value decomposition. The results are illustrated with simple guiding examples.

A Appendix

1.1 A.1 Proof of Block SVD

Proof

(Lemma 2). Let A admit the SVD given in (2). Then by the definition of T, we have

$$\begin{aligned} T (\pm u_i , v_i) = (A v_i , A^* u_i) = \pm \sigma _i (\pm u_i, v_i) \end{aligned}$$

for all $i \in I$. For any element $(f,h) \in {{\,\mathrm{span}\,}}\{(\pm u_i , v_i)\}_{i \in I}^{\perp }$, we can immediately deduce

$$\begin{aligned} 0 = \left\langle (f,h),\, (\pm u_i,v_i) \right\rangle _{\oplus } = \pm \left\langle f,\, u_i \right\rangle _F + \left\langle h,\, v_i \right\rangle _H \end{aligned}$$

for all $i \in I$ and hence $f \in {{\,\mathrm{span}\,}}\{u_i \}_{i \in I}^\perp$ and $h \in {{\,\mathrm{span}\,}}\{v_i \}_{i \in I}^\perp$. Using the SVD of A in (2), we therefore have

It now remains to show that $\left\{ \tfrac{1}{\sqrt{2}} (\pm u_i,v_i) \right\} _{i \in I}$ is an orthonormal system in $F \oplus H$, which is clear since $\left\langle (\pm u_i,v_i),\, ( \pm u_j,v_j) \right\rangle _{\oplus } = 2\,\delta _{ij}$ and $\left\langle (-u_i,v_i),\, (u_j,v_j) \right\rangle _{\oplus } = 0$ for all $i,j \in I$. Concluding, T has the form (3) as claimed.    $\square$

1.2 A.2 Derivation of the Empirical CCA Operator

The claim follows directly when we can show the identity

and its analogue for the feature map $\Psi$. Let be the eigendecomposition of the Gramian. We know that in this case we have the SVD of the operator $\Phi \Phi ^\top = \sum _{i \in I} \lambda _i (\lambda _i^{-1/2}\Phi u_i) \otimes (\lambda _i^{-1/2} \Phi u_i)$, since

We will write this operator SVD for simplicity as $\Phi \Phi ^\top = (\Phi U \Lambda ^{-1/2}) \Lambda (\Lambda ^{-1/2} U \Phi ^\top )$ with an abuse of notation. Note that we can express the inverted operator square root elegantly in this form as $(\Phi \Phi ^\top )^{-1/2} = (\Phi U \Lambda ^{-1/2}) \Lambda ^{-1/2} (\Lambda ^{-1/2} U \Phi ^\top ) = (\Phi U) \Lambda ^{-3/2} (U \Phi ^\top )$. Therefore, we immediately get

which proves the claim. In the regularized case, all operations work the same with an additional $\epsilon$-shift of the eigenvalues, i.e., the matrix $\Lambda$ is replaced with the regularized version $\Lambda + \epsilon \mathrm {I}$.