Kernel methods for regression in continuous time over subsets and manifolds

Abstract

This paper derives error bounds for regression in continuous time over subsets of certain types of Riemannian manifolds. The regression problem is typically driven by a nonlinear evolution law taking values on the manifold, and it is cast as one of optimal estimation in a reproducing kernel Hilbert space. A new notion of persistency of excitation (PE) is defined for the estimation problem over the manifold, and rates of convergence of the continuous time estimates are derived using the PE condition. We discuss and analyze two approximation methods of the exact regression solution. We then conclude the paper with some numerical simulations that illustrate the qualitative character of the computed function estimates. Examples of function estimates generated over a trajectory of the Lorenz system and based on experimental motion capture data are included.

Appendix

1.1 Background on Galerkin approximations

Let U be a real Hilbert space, $A\in {\mathcal {L}}(U)$ be a bounded linear operator on U, $b\in U$ be a fixed element of U, and suppose we seek to find $u\in U$ that satisfies the operator equation

$$\begin{aligned} Au=b. \end{aligned}$$

It is customary that the existence and uniqueness of the solution of this equation is established by studying the associated bilinear form $a(\cdot ,\cdot ):U\times U \rightarrow \mathbb {R}$ given by $a(u,v):=(Au,v)$ for all $u,v\in U$. Then, the operator equation above is equivalent to finding the $u\in U$ for which

$$\begin{aligned} a(u,v)=\langle b,v\rangle _U \quad \text { for all } v\in U. \end{aligned}$$

(19)

The Lax–Milgram theorem given below stipulates a concise pair of conditions that ensure the well-posedness of the operator equation.

Theorem 3

(Lax–Milgram theorem [4]) The bilinear form $a(\cdot ,\cdot ):U\times U \rightarrow \mathbb {R}$ is bounded if there is a constant $C_1>0$ such that

$$\begin{aligned} |a(u,v)|\le C_1 \Vert u\Vert _U \Vert v\Vert _U \quad \text { for all } u,v\in U, \end{aligned}$$

and it is coercive if there is a constant $C_2>0$ such that

$$\begin{aligned} |a(u,u)|\ge C_2 \Vert u\Vert ^2_U \quad \text { for all } u\in U. \end{aligned}$$

If the bilinear form $a(\cdot ,\cdot )$ is bounded and coercive, then $A^{-1}\in {\mathcal {L}}(U)$ and there is a unique solution $u\in U$ to Eq. (19).

Proof

The first condition above, continuity of the bilinear form, ensures that $A\in {\mathcal {L}}(U)$ by definition. The coercivity condition implies that the nullspace of A is just $\{0\}$. As a result, we know that A is one-to-one. This means that the operator $A^{-1}:\text {range}(A)\rightarrow U$ is well defined. From the coercivity condition, we also conclude that

$$\begin{aligned} \Vert A^{-1}b\Vert ^2\le & {} \frac{1}{C_2}|\langle A\cdot A^{-1}b,A^{-1}b\rangle _U|\\ {}\le & {} \frac{1}{C_2} \Vert b\Vert _U \Vert A^{-1}b\Vert _U \end{aligned}$$

for every $b\in \text {range}(A)$. This means that $A^{-1}\in {\mathcal {L}}(\text {range}(A),U)$ and $\Vert A^{-1}\Vert \le 1/C_2$.

One implication of the fact that $A^{-1}\!\in \! {\mathcal {L}}(\text {range}(A),U)$ is that $\text {range}(A)$ is closed. Suppose that $\{b_k\}_{k\in \mathbb {N}}\subset \text {range}(A)$ and $b_k\rightarrow {\bar{b}}$. By construction there is a sequence $\{u_k\}_{k\in \mathbb {N}}\subset U$ such that $Au_k=b_k$. But we have

$$\begin{aligned} \Vert u_m-u_n\Vert= & {} \Vert A^{-1}(y_m-y_n)\Vert _U\\\le & {} \Vert A^{-1}\Vert \cdot \Vert y_m-y_n\Vert \rightarrow 0, \end{aligned}$$

and $\{u_k\}_{k\in \mathbb {N}}$ is a Cauchy sequence in the complete space U. There is a limit $u_k\rightarrow {\bar{u}}\in U$. By the continuity of the operator A, we know that $A{\bar{u}}={\bar{b}}$, hence ${\bar{b}}\in \text {range}(A)$. The range of A is consequently closed.

It only remains to show that $\text {range}(A)=U$. Suppose to the contrary there is a ${\bar{b}} \ne 0$ with ${\bar{b}} \in (\overline{\text {range}(A)})^\perp$. Since $\mathcal {N}(A^*) = (\overline{\text {range}(A)})^\perp$, we know that

$$\begin{aligned} \langle A^*b,w \rangle _U = \langle b,Aw \rangle _U = 0 \end{aligned}$$

By the coercivity condition, we must have $0=\langle A{\bar{b}},{\bar{b}}\rangle _U\ge C_2 \Vert {\bar{b}}\Vert _U^2 \not = 0$. But this is a contradiction and $\text {range}(A)$ is all of U. $\square$

Next, we discuss how error bounds are derived for Galerkin approximations $u_N$ of the solution u of the operator equations above. Let $U_N\subseteq U$ be a finite-dimensional subspace of U. By definition, the Galerkin approximation $u_N\in U_N$ is the unique solution of the equation

$$\begin{aligned} a(u_N,v_N)=\langle b,v_N\rangle _U \quad \text { for all } v_N\in U_N. \end{aligned}$$

(20)

The theorem below summarizes one of the well-known bounds on the error $u-u_N$ between the Galerkin approximation $u_N\in U_N$ and the true solution $u\in U$.

Theorem 4

(Cea’s Lemma, [4]) Suppose that the hypotheses of the Lax–Milgram Theorem 3 hold. There is a unique solution $u_N\in U_N$ of the Galerkin Equation  20. The error $u-u_N$ is a-orthogonal to the subspace $U_N$ in the sense that

$$\begin{aligned} a(u-u_N,v_N)=0 \quad \text { for all } v_N\in U_N. \end{aligned}$$

We also have the error bound

$$\begin{aligned} \Vert u-u_N\Vert _U\le & {} \frac{C_1}{C_2} \min _{v_N\in U_N} \Vert u-v_N\Vert _U\\ {}= & {} \frac{C_1}{C_2}\Vert (I-\varPi _N)u\Vert _U \end{aligned}$$

where $\varPi _N$ is the U-orthogonal projection of U onto $U_N$.

Proof

First, note that when $a(\cdot ,\cdot )$ satisfies the boundedness and coercivity conditions, its restriction $a:U_N\times U_N \rightarrow \mathbb {R}$ to $U_N$ satisfies the boundedness and coercivity conditions with the same constants relative to $U_N$. This means that the Galerkin equations have a unique solution $u_N\in U_N$. Since Eq. (19) holds for all $v\in U$, it holds for all $v_N\in U_N$. We can subtract Eqs. 19 and 20 for each $v_N\in U_N$ and obtain

$$\begin{aligned} a(u-u_N,v_N)=0 \end{aligned}$$

for each $v_N\in U_N$. Using the boundedness and coercivity of the bilinear form, as well as the $a-$orthogonality of the error, we have

$$\begin{aligned} C_2\Vert u-u_N\Vert _U^2&\le a(u-u_n,u-u_N)\\&=a(u-u_n,u-v_N)\\&\le C_1\Vert u-u_N\Vert _U \Vert u-v_N\Vert _U \end{aligned}$$

for any $v_N\in U_N$. The theorem now follows after canceling the common term on the right and left. $\square$