Learning from non-random data in Hilbert spaces: an optimal recovery perspective

Abstract

The notion of generalization in classical Statistical Learning is often attached to the postulate that data points are independent and identically distributed (IID) random variables. While relevant in many applications, this postulate may not hold in general, encouraging the development of learning frameworks that are robust to non-IID data. In this work, we consider the regression problem from an Optimal Recovery perspective. Relying on a model assumption comparable to choosing a hypothesis class, a learner aims at minimizing the worst-case error, without recourse to any probabilistic assumption on the data. We first develop a semidefinite program for calculating the worst-case error of any recovery map in finite-dimensional Hilbert spaces. Then, for any Hilbert space, we show that Optimal Recovery provides a formula which is user-friendly from an algorithmic point-of-view, as long as the hypothesis class is linear. Interestingly, this formula coincides with kernel ridgeless regression in some cases, proving that minimizing the average error and worst-case error can yield the same solution. We provide numerical experiments in support of our theoretical findings.

Appendix

Below, we generalize the result of [3] in two directions. For the first direction, instead of assuming that the target function $f_0$ itself is well approximated by elements of a set V, we assume that it is some linear transform T applied to $f_0$ that is well approximated. This translates in the modification (44) of the approximability set. The novelty occurs not for invertible transforms, but for noninvertible ones (e.g. when T represents a derivative, as in (36)). For the second direction, instead of attempting to recover $f_0$ in full, we assume that we only need to estimate a quantity $Q(f_0)$ depending on $f_0$, such as its integral. Although we focus on the extreme situations where Q is the identity or where Q is a linear functional, the case of an arbitrary linear map Q is covered. Leaving the introduction of the transform T aside, one useful consequence of the result below is that knowledge of (a basis for) the space V is not needed, since only the values of the $\ell _i(v_j)$’s and $Q(v_j)$’s are required to form $(Q \circ R^{\mathrm{opt}})(\mathbf {y})$.

Theorem 4

Let $\mathcal {F},\mathcal {H},\mathcal {Z}$ be three normed spaces, $\mathcal {H}$ being a Hilbert space, and let V be a subspace of $\mathcal {H}$. Consider a linear quantity of interest $Q: \mathcal {F}\rightarrow \mathcal {Z}$ and a linear map $T: \mathcal {F}\rightarrow \mathcal {H}$. For $\mathbf {y}\in \mathbb {R}^m$, define $R^{\mathrm{opt}}(\mathbf {y}) \in \mathcal {F}$ as a solution to

$$\begin{aligned} \underset{f\in \mathcal {F}}{{\text {minimize}}}\, \Vert Tf - P_V(Tf)\Vert _{\mathcal {H}} \qquad \text{ s.to } L(f) = \mathbf {y}. \end{aligned}$$

(43)

Then the learning/recovery map $Q \circ R^{\mathrm{opt}}: \mathbb {R}^m \rightarrow \mathcal {Z}$ is locally optimal over the model set

$$\begin{aligned} \mathcal {K}= \{ f \in \mathcal {F}: \mathrm{dist}(Tf,V) \le \epsilon \} \end{aligned}$$

(44)

in the sense that, for any $z \in \mathcal {Z}$,

$$\begin{aligned} \sup _{f \in \mathcal {K}, L(f) = \mathbf {y}} \Vert Q(f) - Q \circ R^{\mathrm{opt}}(f)\Vert _{\mathcal {Z}} \le \sup _{f \in \mathcal {K}, L(f) = \mathbf {y}} \Vert Q(f) - z \Vert _{\mathcal {Z}}. \end{aligned}$$

(45)

Proof

Let us introduce the compatibility indicator

$$\begin{aligned} \mu := \sup _{u \in \ker (L) \setminus \{0\}} \frac{\Vert Q(u)\Vert _\mathcal {Z}}{\mathrm{dist}(Tu,V)}. \end{aligned}$$

(46)

Given $\mathbf {y}\in \mathbb {R}^m$, let ${\hat{f}} = R^{\mathrm{opt}}(\mathbf {y})$ denote the solution to (43). We shall establish (45) by showing on the one hand that

$$\begin{aligned} \sup _{\begin{array}{c} f \in \mathcal {K}\\ L(f) = \mathbf {y} \end{array}} \Vert Q(f) - Q({\hat{f}})\Vert _{\mathcal {Z}} \le \mu \big [ \epsilon ^2 - \Vert T{\hat{f}} - P_V(T {\hat{f}})\Vert _\mathcal {H}^2 \big ]^{1/2} \end{aligned}$$

(47)

and on the other hand that, for any $z \in \mathcal {Z}$.

$$\begin{aligned} \sup _{\begin{array}{c} f \in \mathcal {K}\\ L(f) = \mathbf {y} \end{array}} \Vert Q(f) - z \Vert _{\mathcal {Z}} \ge \mu \big [ \epsilon ^2 - \Vert T{\hat{f}} - P_V(T {\hat{f}})\Vert _\mathcal {H}^2 \big ]^{1/2}. \end{aligned}$$

(48)

Let us start with (47). Considering an arbitrary $u \in \ker (L)$, notice that the quadratic expression $t \in \mathbb {R}$ given by

$$\begin{aligned}&\Vert T({\hat{f}} + t u) - P_V(T({\hat{f}} + t u))\Vert _{\mathcal {H}}^2 = \Vert T{\hat{f}} - P_V(T{\hat{f}} )\Vert _{\mathcal {H}}^2 + 2 t \langle T{\hat{f}} \nonumber \\&\quad -P_V(T{\hat{f}} ),Tu - P_V(Tu) \rangle + \mathcal {O}(t^2) \end{aligned}$$

(49)

is miminized at the point $t=0$. This forces the linear term $\langle T{\hat{f}} - P_V(T{\hat{f}} ),Tu - P_V(Tu) \rangle$ to vanish, in other words

$$\begin{aligned} \langle T{\hat{f}} - P_V(T{\hat{f}} ),Tu \rangle = 0 \qquad \text{ for } \text{ any } u \in \ker (L). \end{aligned}$$

(50)

Now, considering $f \in \mathcal {K}$ such that $L(f) = \mathbf {y}$ written as $f = {\hat{f}} + u$ for some $u \in \ker (L)$, the fact that $f \in \mathcal {K}$ reads

$$\begin{aligned} \epsilon ^2 \ge \Vert T({\hat{f}} + u) - P_V(T({\hat{f}} + u))\Vert _{\mathcal {H}}^2 = \Vert T{\hat{f}} - P_V(T{\hat{f}})\Vert _{\mathcal {H}}^2 + \Vert Tu - P_V(Tu)\Vert _{\mathcal {H}}^2. \end{aligned}$$

(51)

Rearranging the latter gives

$$\begin{aligned} \mathrm{dist}(Tu,V) \le \big [ \epsilon ^2 - \Vert T{\hat{f}} - P_V(T {\hat{f}})\Vert _\mathcal {H}^2 \big ]^{1/2}. \end{aligned}$$

(52)

It remains to take the definition (46) into account in order to bound $\Vert Q(f) - Q({\hat{f}}) \Vert _{\mathcal {Z}} = \Vert Q(u) \Vert _{\mathcal {Z}}$ and arrive at (47).

Turning to (48), we consider $u \in \ker (L)$ such that

$$\begin{aligned} \Vert Q(u)\Vert _{\mathcal {Z}}&= \mu \, \mathrm{dist}(Tu, V), \end{aligned}$$

(53)

$$\begin{aligned} \Vert Tu - P_V(Tu)\Vert _\mathcal {H}&= \big [ \epsilon ^2 - \Vert T{\hat{f}} - P_V(T {\hat{f}})\Vert _\mathcal {H}^2 \big ]^{1/2}. \end{aligned}$$

(54)

It is clear that $f^\pm := {\hat{f}} \pm u$ both satisfy $L(f^\pm ) = \mathbf {y}$, while $f^\pm \in \mathcal {K}$ follows from

$$\begin{aligned} \Vert T f^\pm - P_V(T f^\pm )\Vert _\mathcal {H}^2&= \Vert (T{\hat{f}} - P_V(T {\hat{f}}) ) \pm (T u - P_V(T u) ) \Vert _\mathcal {H}^2\nonumber \\&= \Vert T{\hat{f}} - P_V(T {\hat{f}}) \Vert ^2 + \Vert Tu - P_V(T u) \Vert _\mathcal {H}^2 = \epsilon ^2. \end{aligned}$$

(55)

Therefore, for any $z \in \mathcal {Z}$,

$$\begin{aligned} \sup _{\begin{array}{c} f \in \mathcal {K}\\ L(f) = \mathbf {y} \end{array}} \Vert Q(f) - z \Vert _{\mathcal {Z}}&\ge \max \{ \Vert Q(f^+) - z \Vert _{\mathcal {Z}}, \Vert Q(f^-) - z \Vert _{\mathcal {Z}}\} \nonumber \\&\ge \frac{1}{2} \big ( \Vert Q(f^+) - z \Vert _{\mathcal {Z}} + \Vert Q(f^-) - z \Vert _{\mathcal {Z}}\} \big )\nonumber \\&\ge \frac{1}{2} \Vert Q(f^+ - f^-) \Vert _{\mathcal {Z}} = \Vert Q(u) \Vert _{\mathcal {Z}}. \end{aligned}$$

(56)

Taking (53) and (54) into account finishes to prove (48). $\square$