Nonparametric multiple regression by projection on non-compactly supported bases

Abstract

We study the nonparametric regression estimation problem with a random design in ${\mathbb{R}}^{p}$ with $p\ge 2$. We do so by using a projection estimator obtained by least squares minimization. Our contribution is to consider non-compact estimation domains in ${\mathbb {R}}^{p}$, on which we recover the function, and to provide a theoretical study of the risk of the estimator relative to a norm weighted by the distribution of the design. We propose a model selection procedure in which the model collection is random and takes into account the discrepancy between the empirical norm and the norm associated with the distribution of design. We prove that the resulting estimator automatically optimizes the bias-variance trade-off in both norms, and we illustrate the numerical performance of our procedure on simulated data.

Appendices

A Linear algebra

Lemma 8

Let E be a Euclidean vector space and let $\ell :E\rightarrow {\mathbb {R}}^n$ be an injective linear map. For $y\in {\mathbb {R}}^n$, the solution of the problem:

$$\begin{aligned} {\hat{a}} {:}{=} {{\,\mathrm{arg\, min}\,}}_{a\in E} \,\left| \left| y - \ell (a)\right| \right| _{{\mathbb {R}}^n}^2 \end{aligned}$$

is given by:

$$\begin{aligned} {\hat{a}} = \left[ (\ell ^* \circ \ell )^{-1} \circ \ell ^*\right] (y), \end{aligned}$$

where $\ell ^*:{\mathbb {R}}^n\rightarrow E$ is characterized by the relation ${\langle }{ y, \ell (a) }{\rangle }_{{\mathbb {R}}^n} = {\langle }{ \ell ^*(y), a }{\rangle }_{E}$.

Lemma 9

Let $\textbf{A}$, $\textbf{B}$ be square matrices. If $\textbf{A}$ is invertible and $\left| \left| \textbf{A}^{-1}\textbf{B} \right| \right| _{\textrm{op}}<1$, then $\textbf{A} +\textbf{B}$ is invertible and it holds:

$$\begin{aligned} \left| \left| (\textbf{A} + {\textbf{B}})^{-1} - \textbf{A}^{-1}\right| \right| _{\textrm{op}} \le \frac{\left| \left| {\textbf{A}}^{-1}\right| \right| _{\textrm{op}}^2 \left| \left| {\textbf{B}}\right| \right| _{\textrm{op}}}{1 - \left| \left| {\textbf{A}}^{-1}{\textbf{B}}\right| \right| _{\textrm{op}}}. \end{aligned}$$

B Concentration inequalities

You can find the proofs of the following bounds in Tropp (2012) and Gittens and Tropp (2011).

Theorem 5

(Matrix Chernoff bound) Let ${\textbf{Z}}_1, \dotsc , {\textbf{Z}}_n$ be independent random self-adjoint positive semi-definite matrices with dimension d, such that $\sup _k \lambda _{\max }({\textbf{Z}}_k) \le R$ a.s. If we define:

$$\begin{aligned} \mu _{\min } {:}{=} \lambda _{\min }\!\left( \sum _{k=1}^{n} {\mathbb {E}}[{\textbf{Z}}_k] \right) , \end{aligned}$$

then we have:

$$\begin{aligned}{} & {} \forall \delta \in (0,1),\quad {\mathbb {P}}\left[ \lambda _{\min }\!\left( \sum _{k=1}^{n} {\textbf{Z}}_k \right) \le (1-\delta ) \mu _{\min } \right] \le d\times \left( \frac{\textrm{e}^{-\delta }}{(1-\delta )^{(1-\delta )}} \right) ^{\mu _{\min } / R}, \end{aligned}$$

(29)

$$\begin{aligned}{} & {} \forall \delta >0,\quad {\mathbb {P}}\left[ \lambda _{\min }\!\left( \sum _{k=1}^{n} {\textbf{Z}}_k \right) \ge (1+\delta ) \mu _{\min } \right] \le \left( \frac{\textrm{e}^{\delta }}{(1+\delta )^{(1+\delta )}} \right) ^{\mu _{\min } / R}. \end{aligned}$$

(30)

Theorem 6

(Matrix Bernstein bound) Let ${\textbf{Z}}_1, \dotsc , {\textbf{Z}}_n$ be independent random self-adjoint positive semi-definite matrices with dimension d, such that ${\mathbb {E}}[{\textbf{Z}}_k] = \textbf{0}$ and that $\sup _k \lambda _{\max }({\textbf{Z}}_k) \le R$ a.s. If $v>0$ is such that:

$$\begin{aligned} \left| \left| \sum _{k=1}^{n} {\mathbb {E}}\left[ {\textbf{Z}}_k^2\right] \right| \right| _\textrm{op}\le v, \end{aligned}$$

then for all $x>0$ we have:

$$\begin{aligned} {\mathbb {P}}\left[ \lambda _{\max } \!\left( \sum _{k=1}^{n} {\textbf{Z}}_i \right) \ge x \right] \le d\times \exp \left( \frac{-x^2/2}{v + \frac{R}{3}x} \right) . \end{aligned}$$

C Combinatorics

Proposition 4

For $n\ge 1$ and $p\ge 2$ we have:

$$\begin{aligned} {{\,\textrm{Card}\,}}{\lbrace }{ {\varvec{m}}\in {\mathbb {N}}_+^p \,\vert \, m_1\cdots m_p \le n }{\rbrace } \le n\, H_n^{p-1}, \end{aligned}$$

where $H_n {:}{=} \sum _{k=1}^{n} \frac{1}{k}$ is the n-th harmonic number.

Proof

We compute:

$$\begin{aligned} {{\,\textrm{Card}\,}}{\lbrace }{ {\varvec{m}}\in {\mathbb {N}}_+^p \,\vert \, D_{{\varvec{m}}} \le n }{\rbrace }&= \sum _{m_1=1}^{n} \cdots \sum _{m_p=1}^{n} \textbf{1}_{m_1\cdots m_p \le n} \\&= \sum _{m_1=1}^{n} \cdots \sum _{m_p=1}^{n} \textbf{1}_{m_p \le \frac{n}{m_1\cdots m_{p-1}}} \\&= \sum _{m_1=1}^{n} \cdots \sum _{m_{p-1}=1}^{n} \left\lceil {\frac{n}{m_1\cdots m_{p-1}}}\right\rceil \\&\le \sum _{m_1=1}^{n} \cdots \sum _{m_{p-1}=1}^{n} \frac{n}{m_1\cdots m_{p-1}} = n\, H_n^{p-1}. \end{aligned}$$

$\square$

Theorem 7

(Divisor bound) Let $N\in {\mathbb {N}}_+$ and let $\textrm{div}(N)$ be the set of divisors of N. We have for all $\epsilon >0$:

$$\begin{aligned} {{\,\textrm{Card}\,}}\!\big (\textrm{div}(N)\big ) = {{\,\mathrm{\textrm{o}}\,}}(N^\epsilon ). \end{aligned}$$

As a consequence, we have for all $\epsilon >0$:

$$\begin{aligned} {{\,\textrm{Card}\,}}{\lbrace }{ m\in {\mathbb {N}}_+^p \,\vert \, m_1\cdots m_p = N }{\rbrace } \le {{\,\textrm{Card}\,}}\!\big (\textrm{div}(N)\big )^p = {{\,\mathrm{\textrm{o}}\,}}(N^\epsilon ). \end{aligned}$$

A proof of this result can be found in Tao (2008).