Distribution-free learning theory for approximating submanifolds from reptile motion capture data

Abstract

This paper describes the formulation and experimental testing of an estimation of submanifold models of animal motion. It is assumed that the animal motion is supported on a configuration manifold, Q, and that the manifold is homeomorphic to a known smooth, Riemannian manifold, S. Estimation of the configuration submanifold is achieved by finding an unknown mapping, $\gamma$, from S to Q. The overall problem is cast as a distribution-free learning problem over the manifold of measurements. This paper defines sufficient conditions that show that the rates of convergence in $L^2_\mu (S)$ of approximations of $\gamma$ correspond to those known for classical distribution-free learning theory over Euclidean space. This paper concludes with a study and discussion of the performance of the proposed method using samples from recent reptile motion studies.

Appendices

A. Appendix

A.1 Lebesgue and Sobolev function spaces

The goal of this section is to provide a brief review of the Lebesgue spaces, which we denote $L^p_\mu (\Omega )$, and the Sobolev Spaces that are denoted $W^{r,p}(\Omega )$. Both are defined over some particular domain $\Omega$. In order to provide some motivation toward understanding these function spaces, we first introduce the space of continuous functions over the domain $\Omega$. We denote this space $C(\Omega )$. The conventional norm $\Vert \cdot \Vert _{C(\Omega )}$ on $C(\Omega )$ is given by

$$\begin{aligned} \Vert f\Vert _{C(\Omega )} = \sup _{x\in \Omega }|f(x)| \end{aligned}$$

With this norm, the space is complete.In a complete metric space, Cauchy sequences , which are sequences in the space whose elements get closer to one another as the sequence approaches $\infty$. converge to a limit in the space. We note that the distances between elements, and the convergence, are based on a metric induced by the norm. We can make this space a vector space with the following two operations.

$$\begin{aligned} (f+g)(x)= & {} f(x) + g(x) \qquad f,g \in C(\Omega )\\ a(f(x))= & {} af(x) \qquad f \in C(\Omega ), a \in {\mathbb {R}} \end{aligned}$$

Given these two operations, $C(\Omega )$ becomes a normed linear space. As mentioned in the paper, our expected risk function is based on convergence using the $2-$ norm, denoted $\Vert \cdot \Vert _2$ defined by

$$\begin{aligned} \Vert f\Vert _2 = \Big (\int _\Omega |f(x)|^2d\mu \Big )^{\frac{1}{2}} \end{aligned}$$

If we instead use the $2-$norm on the space, $C(\Omega )$, it is well known that the space is no longer complete. However, the space $L^2_\mu (\Omega )$, which consists of functions f where

$$\begin{aligned} \Vert f\Vert _2 = \left( \int _\Omega |f(x)|^2 d\mu \right) ^{\frac{1}{2}} < \infty \end{aligned}$$

is complete with the metric induced by the 2 norm. We can generalize this notion to the so-called $p-$norms. If we consider any positive real number p, the $\mu$-square integrable Lebesgue functions, denoted $L^p_\mu (\Omega )$, consist of spaces of measurable functions f for which

$$\begin{aligned} \Vert f\Vert _p = \left( \int _\Omega |f(x)|^p d\mu \right) ^{\frac{1}{p}} < \infty \end{aligned}$$

For functions in $L^p_\mu (\Omega )$, we can use the norm to get a sense of the “size” of the function. Additionally, we can utilize topological notions in convergence proofs. For the space $L^p_\mu (\Omega )$, if we have disjoint subsets $\{A_k\}_{k=1}^n$ of $\Omega$ where $\Omega = \cup _{k=1}^\infty A_k$. If we define $1_{A_k}$ as the characteristic function of the set $A_k$, the finite-dimensional space of functions f of the form

$$\begin{aligned} f = \sum _{k=1}^n a_k 1_{A_k} \end{aligned}$$

are dense in $L^p_\mu (\Omega )$.

While we know that the functions in this space exhibit these nice properties, we are often interested in the regularity of our functions in a particular space. It is well known that taking the derivative of functions in $L^P_\mu (\Omega )$ can result in functions that are not in $L^P_\mu (\Omega )$. Thus, we would like a space of functions which when we take the derivative still behave nicely. These are the Sobolev Spaces $W^{r,p}(\Omega )$, which are functions with derivatives that are in $L^P_\mu (\Omega )$. There are many different ways to define Sobolev Spaces over various domains. Conventionally, a norm for a Sobolev Space $W^{r,p}(\Omega )$ can be defined as

$$\begin{aligned} \Vert f\Vert _{r,p} = \sum _{0\le |a| \le r}\left( \int _\Omega |D^a f(x)|^p d\mu \right) ^{\frac{1}{p}} < \infty \end{aligned}$$

where a is a multi-index. For functions in $W^{r,p}(\Omega )$, we have most of the convenient properties of functions in $L^p_\mu (\Omega )$. In fact, the $L^p_\mu (\Omega )$ space can be thought of as a special case of a Sobolev space $W^{r,p}(\Omega )$ when $r = 0$. We make a final remark that there are many important details about these spaces omitted because covering all the fundamentals would require a much larger discussion. A much more detailed motivation can begin from a more elementary functional analysis texts such as [54]. Specifically, details and motivation toward Sobolev Spaces can be found in [55].

A.2 Kernel method estimation

This section outlines the minimization over ${\mathbb {H}}_n^S$ and solves for $\gamma _{n,m}$ in terms of the kernel basis. For a particular dimension j, any estimate can be represented by a linear combination of n radial basis functions with centers $\xi$. In other words, $\gamma ^j_{n,m}(s) = \sum _{k=1}^n \alpha _k {\mathfrak {K}}^S_{\xi _k}(s)$. Given our fixed basis, we see that equation 9 can be solved now by determining the optimal coefficients $\{\alpha _k^*\}_{k=1}^n$. Specifically, choosing a minimizing $\gamma ^j_{n,m}$ is found by finding optimal coefficients $\varvec{\alpha }^*$ where

$$\begin{aligned} \alpha _1^*,...,\alpha _n^*= {{\,\mathrm{arg\,min}\,}}_{\alpha _1,... \alpha _n}\frac{1}{m}\sum _{i=1}^m\Vert q^j_i - \sum _{k=1}^n \alpha _k {\mathfrak {K}}^S_{\xi _k}(s)\Vert _X^2 \end{aligned}$$

If we arrange the coefficients, $\alpha _k$, into a n-dimensional vector $\varvec{\alpha } = [\alpha _1, \alpha _2, \dots , \alpha _n]^T$, create an m-dimensional vector of measurement outputs of the $j^{th}$ dimension $\varvec{q}^j = [q^j_1, q^j_2, \dots , q^j_m]^T$, and denote our $n \times m$ Kernel matrix $\varvec{K}$ where

$$\begin{aligned} \varvec{K}_{ij} = {\mathfrak {K}}^S_{\xi _i}(s_j) \end{aligned}$$

This matrix is a tall matrix with rows equal to the large number of samples m and columns n equal to the dimension of the function space. With these matrices we can reform our minimization with matrix representations to finding the optimal vector $\varvec{\alpha }$ in ${\mathbb {R}}^n$

$$\begin{aligned} \varvec{\alpha }^*= {{\,\mathrm{arg\,min}\,}}_{\varvec{\alpha }\in {\mathbb {R}}^n} \Vert \varvec{q}^j - \varvec{K}\varvec{\alpha }\Vert _X^2 \end{aligned}$$

(14)

for $X \approx {\mathbb {R}}^d$ is a Hilbert Space, which means we have an inner product related to the norm by

$$\begin{aligned} \Vert \varvec{v}\Vert _X^2 = (\varvec{v},\varvec{v})_X \end{aligned}$$

for any $v \in X$. Our minimization can now be interpreted as minimizing the least squares error on an n-dimensional function space

$$\begin{aligned} \varvec{\alpha }^*= {{\,\mathrm{arg\,min}\,}}_{\varvec{\alpha }\in {\mathbb {R}}^n} (\varvec{q}^j - \varvec{K}\varvec{\alpha },\varvec{q}^j - \varvec{K}\varvec{\alpha })_X \end{aligned}$$

We can determine the optimal coefficients $\varvec{\alpha }^*$ by taking the partial derivative and setting it equal to zero. The optimal vector $\varvec{\alpha }^*$ is, therefore,

$$\begin{aligned} \varvec{\alpha }^* = (\varvec{K}^T\varvec{K})^{-1}\varvec{K}^T\varvec{q}^j \end{aligned}$$

As mentioned before, matrix inversion can prove to be computationally costly. Furthermore, ill-conditioned matrices can lead to poor estimates. We make one remark that the optimal approximation from $H^S_n$, where we build estimates from piece-wise constants, does not face these same issues.

A.3 Proof of Theorem 1

Proof

When we define $\gamma ^j_{n,m}:=\sum _{k=1,\ldots ,n(\ell )}\alpha _k 1_{S_{\ell ,k}}(\cdot )$, where $\alpha _k$ represent the weighted coefficients of our piece-wise constants $1_{S_{\ell ,k}}(\cdot )$, we have the explicit representation of $E^j_m(\gamma ^j_{n,m})$ given by

$$\begin{aligned} E_m^j(\gamma _{n,m}^j) = \frac{1}{m}\sum _{i=1}^m\left( x_i^j - \sum _{k=1}^{n(\ell )}\alpha _k 1_{S_{\ell ,k}}(s_i)\right) ^2. \end{aligned}$$

We can also write this sum as

$$\begin{aligned} E_m^j(\gamma _{n,m}^j) = \frac{1}{m}\sum _{k=1}^{n(\ell )}\sum _{i=1}^m\left( 1_{S_{\ell ,k}}(s_i)x_i^j - \alpha _k 1_{S_{\ell ,k}}(s_i)\right) ^2, \end{aligned}$$

and this summation can be reordered as

$$\begin{aligned}&E_m^j(\gamma _{n,m}^j) = \frac{1}{m}\sum _{k=1}^{n(\ell )}E_{m,n}^j(\alpha _k)\\&\quad E_{m,n}^j(\alpha _k) = \sum _{i=1}^m(1_{S_k}(s)x_i^j - \alpha _k1_{S_k}(s))^2 \end{aligned}$$

with each $E_{m,n}^j(\alpha _k)$ depending on a single variable $\alpha _k$. By taking the partial derivative $D_{\alpha _k}(E^j_{m})=0$, we see that for the optimal choice of coefficients ${\hat{a}}_i$

$$\begin{aligned} {\hat{a}}_i = \frac{\sum _{i=1}^m1_{S_{\ell ,k}}(s_i)x_i^{j}}{\sum _{i=1}^m1_{S_{\ell ,k}}(s_i)} \qquad \text {for} \quad i = 1,\dots ,n(\ell ), \end{aligned}$$

which establishes the form of solution given in the claim. $\square$

A.4 Proof of Theorem 2

We now turn to the consideration of the error bound in the theorem.

Proof

From the triangle inequality

$$\begin{aligned} \Vert \gamma _\mu - \gamma _{n,m}\Vert _{L^2_\mu } \le \Vert \gamma _\mu - \Pi _n^S\gamma _\mu \Vert _{L^2_\mu } + \Vert \Pi _n^S\gamma _\mu -\gamma _{n,m}\Vert _{L^2_\mu }, \end{aligned}$$

we can bound the first term above by $n^{-r}$ from the definition of the linear approximation space $A^{r,2}(L^2_\mu (S))$. The bound in the theorem is proven if we can show that there is a constant $C_2$ such that $\Vert \Pi _n^S \gamma _\mu ^j - \gamma _{n,m}\Vert _{L^2_\mu (S)}\le C_2 n(\ell )\log (n(\ell ))/m$. We establish this bound by an extension to functions on the manifold S of the proof in [56], which is given for functions defined on ${\mathbb {R}}^p$ for some $p\ge 1$. The expression above for $\gamma ^j_{n,m}$ can be written in the form $\gamma ^j_{n,m}:=\sum _{k=1}^{n(\ell )}\alpha _k1_{S_{\ell ,k}}$, and that for $\Pi ^S_n\gamma ^j_\mu$ can be written as $\gamma ^j_{\mu }:=\sum _{k=1}^{n(\ell )}{\hat{\alpha }}^j_{\ell ,k}1_{S_{\ell ,k}}$. In terms of these expansions, we write the error as

$$\begin{aligned} \Vert \Pi ^S_n \gamma ^j_\mu -\gamma _{n,m}\Vert ^2_{L^2_\mu (S)}=\sum _{\ell =1}^{n(\ell )} \left( \alpha _{\ell ,k}^j-{\hat{\alpha }}_{\ell ,k}^j \right) ^2\mu _S(S_{\ell ,k}). \end{aligned}$$

Let $\epsilon >0$ be an arbitrary, but fixed, positive number. We define the set of indices ${\mathcal {I}}(\epsilon )$ that denote subsets $S_{n,k}$ that have, in a sense, small measure,

$$\begin{aligned} {\mathcal {I}}(\epsilon )&:= \left\{ k\in \{1,\ldots ,n(\ell )\}\ \biggl | \ \mu _S(S_{\ell ,k})\le \frac{1}{8N(\ell ){\bar{X}}^2} \right\} \end{aligned}$$

where ${\bar{X}} = \sup _{s\in S}\Vert \gamma (s)\Vert _X$. We define the complement $\tilde{{\mathcal {I}}} (\epsilon ):=\{ i\in \{1\ldots n(\ell )\} \ | \ k\not \in {\mathcal {I}}\}$, and then set the associated sums $S_{{\mathcal {I}}}:=\sum _{k\in {\mathcal {I}}}(\alpha _{\ell ,k}^j-{\hat{\alpha }}_{\ell ,k}^j)^2\mu _S(S_{\ell ,k})$ and $S_{\tilde{{\mathcal {I}}}}:=\sum _{k\in \tilde{{\mathcal {I}}}}(\alpha _{\ell ,k}^j-{\hat{\alpha }}_{\ell ,k}^j)^2\mu _S(S_{\ell ,k})$. The bound in Equation 2 follows if we can demonstrate a concentration of measure formula that has the form

$$\begin{aligned}&\text {Prob}\left( \Vert \Pi ^S_n \gamma ^j_\mu -\gamma ^j_{n,m} \Vert ^2_{L^2_\mu }> \epsilon ^2 \right) \nonumber \\&=\text {Prob}(S_{{\mathcal {I}}}+S_{\tilde{{\mathcal {I}}}}>\epsilon ^2)\le be^{cm\epsilon ^2/n(\ell )} \end{aligned}$$

(15)

for some constants b, c. See [40, 56] for a discussion of such concentration inequalities. The fact that such a concentration inequality implies the bound in expectation in Equation 2 is proven in [56] on page 1311 for functions over Euclidean space. The argument proceeds exactly in the same way for the problem at hand by integration of the distribution function defined by Equation 15 over the manifold S. To establish the concentration inequality, let us define two events

$$\begin{aligned} E_{{\mathcal {I}}+\tilde{{\mathcal {I}}}}(\epsilon )&:=\left\{ z\in {\mathbb {Z}}^m\ | \ S_{{\mathcal {I}}}+S_{\tilde{{\mathcal {I}}}}\ge \epsilon ^2 \right\} , \\ E_{\tilde{{\mathcal {I}}}}(\epsilon )&:=\left\{ z\in {\mathbb {Z}}^m\ | \ S_{\tilde{{\mathcal {I}}}}\ge \frac{1}{2}\epsilon ^2. \right\} . \end{aligned}$$

We can compute directly from the definitions of the coefficients $\alpha _k,{\hat{\alpha }}^j_{\ell ,k}$, and using the compactness of $\gamma (S)\subset X$, that $S_{{\mathcal {I}}}\le \epsilon ^2/2$ for any $\epsilon >0$. Since we always have

$$\begin{aligned}&\sum _{k\in \tilde{I}}\left( \alpha _k-{\hat{\alpha }}^j_{\ell ,k} \right) ^2\mu _S(S_{\ell ,k}) \\&> \epsilon ^2 - \sum _{k\in {\mathcal {I}}}\left( \alpha _k-{\hat{\alpha }}^j_{\ell ,k} \right) ^2 \mu _S(S_{\ell ,k})> \frac{1}{2}\epsilon ^2, \end{aligned}$$

we know that $E_{{\mathcal {I}}+\tilde{{\mathcal {I}}}}(\epsilon )\subseteq E_{\tilde{{\mathcal {I}}}}(\epsilon )$ for any $\epsilon >0$. If the inequality $S_{\tilde{{\mathcal {I}}}}>\epsilon ^2/2$, then we know there is at least one $\tilde{k}\in \tilde{{\mathcal {I}}}$ such that

$$\begin{aligned} S_{\tilde{i}}(\epsilon ) :=(\alpha _k-{\hat{\alpha }}^j_{\ell ,k})^2\mu _S(S_{\ell ,k})>\frac{1}{2 (\#\tilde{{\mathcal {I}}})}\epsilon ^2>\frac{1}{2N(\epsilon )}\epsilon ^2. \end{aligned}$$

When we define the event $E_i(\epsilon ):=\{z\in {\mathbb {Z}}^m \ | \ S_i(\epsilon )> \epsilon ^2/2n(\ell )$ for each $i\in \{1,\ldots , n(\ell )\}$, we conclude

$$\begin{aligned} E_{{\mathcal {I}}+\tilde{{\mathcal {I}}}}\subseteq E_{\tilde{{\mathcal {I}}}}(\epsilon ) \subseteq \bigcup _{i\in \tilde{{\mathcal {I}}}} E_i. \end{aligned}$$

By the monotonicity of measures, we conclude that $\text {Prob}\left( {\mathcal {I}}+\tilde{{\mathcal {I}}} \right) \le \sum _{i\in \tilde{{\mathcal {I}}}} \text {Prob}(E_i)$. But we can show, again by a simple modification of the arguments on pages 1310 of [56], that $\text {Prob}(E_i)\lesssim e^{-cm\epsilon ^2/n(\ell )}$. The analysis proceeds as in that reference by using Bernstein’s inequality for random variables defined over the probability space $(S,\Sigma _S, \mu _S)$ instead of over Euclidean space. $\square$