Learning General Sparse Additive Models from Point Queries in High Dimensions

Hemant Tyagi; Jan Vybiral

doi:10.1007/s00365-019-09461-6

Constructive Approximation

December 2019, Volume 50, Issue 3, pp 403–455 | Cite as

Learning General Sparse Additive Models from Point Queries in High Dimensions

Authors
Authors and affiliations

Hemant Tyagi
Jan Vybiral

Article

First Online: 03 May 2019

66 Downloads
1 Citations

Abstract

We consider the problem of learning a d-variate function f defined on the cube $[-1,1]^d\subset \mathbb {R}^d$, where the algorithm is assumed to have black box access to samples of f within this domain. Let ${\mathcal {S}}_r \subset {[d] \atopwithdelims ()r}; r=1,\dots ,r_0$ be sets consisting of unknown r-wise interactions amongst the coordinate variables. We then focus on the setting where f has an additive structure; i.e., it can be represented as

$$\begin{aligned} f = \sum _{{\mathbf {j}}\in {\mathcal {S}}_1} \phi _{{\mathbf {j}}} + \sum _{{\mathbf {j}}\in {\mathcal {S}}_2} \phi _{{\mathbf {j}}} + \dots + \sum _{{\mathbf {j}}\in {\mathcal {S}}_{r_0}} \phi _{{\mathbf {j}}}, \end{aligned}$$

where each $\phi _{{\mathbf {j}}}$; ${\mathbf {j}}\in {\mathcal {S}}_r$ is at most r-variate for $1 \le r \le r_0$. We derive randomized algorithms that query f at a carefully constructed set of points and exactly recover each ${\mathcal {S}}_r$ with high probability. In contrast to previous work, our analysis does not rely on numerical approximation of derivatives by finite order differences.

Keywords

Sparse additive models Sampling Hash functions Sparse recovery

Mathematics Subject Classification

41A25 41A63 65D15

This is a preview of subscription content, to check access.

Notes

Appendix A: Proofs for Section 2

Proof of Proposition 1

For $U\subseteq [d]$, we define

$$\begin{aligned} (P_Uf)({\mathbf {x}}_U)=f\Bigl (\sum _{j\in U}x_j{\mathbf {e}}_j\Bigr )\quad \text {and}\quad f_U({\mathbf {x}}_U)=\sum _{V\subseteq U}(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V). \end{aligned}$$

Here, $\{{\mathbf {e}}_1,\dots ,{\mathbf {e}}_d\}$ is the canonical basis of $\mathbb {R}^d$.

By its definition, $f_U$ is a continuous function. Furthermore,

$$\begin{aligned} \sum _{U\subseteq [d]}f_U({\mathbf {x}}_U)&=\sum _{U\subseteq [d]}\sum _{V\subseteq U}(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V)\\&=\sum _{V\subseteq [d]}(P_V f)({\mathbf {x}}_V)\sum _{U\supseteq V}(-1)^{|U|-|V|}\\&=\sum _{V\subseteq [d]}(P_V f)({\mathbf {x}}_V)\sum _{W\subseteq [d]{\setminus } V}(-1)^{|W|}. \end{aligned}$$

The last sum can be rewritten as

$$\begin{aligned} \sum _{W\subseteq [d]{\setminus } V}(-1)^{|W|}=\sum _{k=0}^{d-|V|}(-1)^k{d-|V|\atopwithdelims ()k}, \end{aligned}$$

which is equal to $(1-1)^{d-|V|}=0$ for all $V\subseteq [d]$ except $V=[d]$. This leads to (2.3).

If $x_{j}=0$ for some $j\in U$, we get

$$\begin{aligned} f_U({\mathbf {x}}_U)&=\sum _{V\subseteq U}(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V)\\&=\sum _{V\subseteq U{\setminus }\{j\}}\Bigl [(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V)+(-1)^{|U|-|V\cup \{j\}|}(P_{V\cup \{j\}} f)({\mathbf {x}}_{V\cup \{j\}})\Bigr ]=0, \end{aligned}$$

as all the terms in the last sum are equal to zero.

Finally, the uniqueness follows by induction. The statement is obvious for $d=1$. Let now $d>1$ and let us assume that a given function $f\in C([-1,1]^d)$ allows a decomposition

$$\begin{aligned} f({\mathbf {x}})=\sum _{U\subseteq [d]}f_U({\mathbf {x}}_U),\quad {\mathbf {x}}\in [-1,1]^d, \end{aligned}$$

which satisfies the properties (a)–(c) of Proposition 1.

Let W be a proper subset of [d], and put for ${\mathbf {x}}_W\in [-1,1]^{|W|}$:

$$\begin{aligned} g_W({\mathbf {x}}_W)=f\Bigl (\sum _{j\in W}x_j{\mathbf {e}}_j\Bigr ). \end{aligned}$$

Then $g_W\in C([-1,1]^{|W|})$, and using c) of Proposition 1, we obtain

$$\begin{aligned} g_W({\mathbf {x}}_W)=\sum _{U\subseteq [d]}f_U\Bigl (\Bigl (\sum _{j\in W}x_j{\mathbf {e}}_j\Bigr )_U\Bigr ) =\sum _{U\subseteq [d]}f_U\Bigl (\sum _{j\in W}x_j({\mathbf {e}}_j)_U\Bigr )=\sum _{U\subseteq W}f_U({\mathbf {x}}_U). \end{aligned}$$

This decomposition of $g_W$ satisfies (a)–(c) of Proposition 1 with $|W|\le d-1$. By the induction assumption, this decomposition is therefore unique. We conclude that $f_U$ is uniquely determined for all $U\subset [d]$. Finally, also

$$\begin{aligned} f_{[d]}({\mathbf {x}})=f({\mathbf {x}})-\sum _{U\subset [d]}f_U({\mathbf {x}}_U) \end{aligned}$$

is uniquely determined. $\square $

Proof of Proposition 2

Let f be given by (4.1), and let us assume that it satisfies all the assumptions of Proposition 2. We show that (4.1) coincides with its Anchored-ANOVA decomposition as described in Proposition 1, and (4.1) is therefore unique.

Let $U\subseteq [d]$ with $|U|\ge 3$. Then

$$\begin{aligned} f_U({\mathbf {x}}_U)&=\sum _{V\subseteq U}(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V)\\&=\sum _{V\subseteq U}(-1)^{|U|-|V|}\Bigl \{\mu +\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}\phi _j((x_V)_j)\\&\quad +\sum _{{\mathbf {j}}=(j_1,j_2)\in {\mathcal {S}}_2}\phi _{\mathbf {j}}((x_V)_{j_1},(x_V)_{j_2})\Bigr \}\\&=\mu \sum _{V\subseteq U}(-1)^{|U|-|V|}+\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}\phi _j(x_j)\sum _{\begin{array}{c} V\subseteq U\\ j\in V \end{array}}(-1)^{|U|-|V|}\\&\quad +\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}\phi _j(0)\sum _{\begin{array}{c} V\subseteq U\\ j\not \in V \end{array}}(-1)^{|U|-|V|}\\&\quad +\sum _{{\mathbf {j}}=(j_1,j_2)\in {\mathcal {S}}_2}\sum _{V\subseteq U}(-1)^{|U|-|V|}\phi _{\mathbf {j}}((x_V)_{j_1},(x_V)_{j_2}). \end{aligned}$$

It is easy to see that the first three terms are zero. Finally, the last term can be split into a sum of four terms depending on whether $j_1\in V$ or $j_2\in V$, i.e., terms of the kind

$$\begin{aligned} \sum _{{\mathbf {j}}=(j_1,j_2)\in {{\mathcal {S}}_2}}\phi _{\mathbf {j}}(x_{j_1},x_{j_2})\sum _{\begin{array}{c} V\subseteq U\\ j_1,j_2\in V \end{array}}(-1)^{|U|-|V|}, \end{aligned}$$

which also vanish. Therefore, $f_U({\mathbf {x}}_U)=0.$

If $U=\{j_1,j_2\}$ with $1\le j_1<j_2\le d$, then $\emptyset , \{j_1\}, \{j_2\}$ and $\{j_1,j_2\}$ are the only subsets of U, and we obtain

$$\begin{aligned} f_U({\mathbf {x}}_U)&=\sum _{V\subseteq U}(-1)^{|U|-|V|}(P_V f)({\mathbf {x}}_V)\\&=(P_\emptyset f)(0)-(P_{j_1} f)(x_{j_1})-(P_{j_2} f)(x_{j_2})+(P_{\{j_1,j_2\}} f)(x_{j_1},x_{j_2})\\&=f(0)-f(x_{j_1}{\mathbf {e}}_{j_1})-f(x_{j_2}{\mathbf {e}}_{j_2})+f(x_{j_1}{\mathbf {e}}_{j_1}+x_{j_2}{\mathbf {e}}_{j_2})\\&=\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}[\phi _j(0)-\phi _j((x_{j_1}{\mathbf {e}}_{j_1})_j)-\phi _j((x_{j_2}{\mathbf {e}}_{j_2})_j)+\phi _j((x_{j_1}{\mathbf {e}}_{j_1}+x_{j_2}{\mathbf {e}}_{j_2})_j)]\\&\quad +\sum _{{\mathbf {j}}\in {\mathcal {S}}_2}[\phi _{\mathbf {j}}(0)-\phi _{\mathbf {j}}((x_{j_1}{\mathbf {e}}_{j_1})_{\mathbf {j}})-\phi _{\mathbf {j}}((x_{j_2}{\mathbf {e}}_{j_2})_{\mathbf {j}})+\phi _{\mathbf {j}}((x_{j_1}{\mathbf {e}}_{j_1}+x_{j_2}{\mathbf {e}}_{j_2})_{\mathbf {j}})]. \end{aligned}$$

The first sum vanishes for all $j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}$, which can be easily observed by considering the options $j\not \in \{j_1,j_2\}, j=j_1$, or $j=j_2.$ If $(j_1,j_2)\not \in {\mathcal {S}}_2$, then also the second sum vanishes. If $(j_1,j_2)\in {\mathcal {S}}_2$, then

$$\begin{aligned} f_{j_1,j_2}(x_{j_1},x_{j_2})= & {} \phi _{j_1,j_2}(x_{j_1},x_{j_2}) -\phi _{j_1,j_2}(x_{j_1},0)-\phi _{j_1,j_2}(0,x_{j_2})\\&+\phi _{j_1,j_2}(0,0)=\phi _{j_1,j_2}(x_{j_1},x_{j_2}). \end{aligned}$$

Finally, if $l\in [d]$, then $\emptyset $ and $\{l\}$ are the only subsets of $\{l\}$. If furthermore $l\not \in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}$, we get

$$\begin{aligned} f_{\{l\}}(x_l)&=-f(0)+f(x_l{\mathbf {e}}_l)\\&=-\Bigl (\mu +\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}\phi _j(0)+\sum _{{\mathbf {j}}\in {\mathcal {S}}_2}\phi _{\mathbf {j}}(0)\Bigr )\\&\quad + \Bigl (\mu +\sum _{j\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}}\phi _j(0)+\sum _{{\mathbf {j}}\in {\mathcal {S}}_2} \phi _{\mathbf {j}}(0)\Bigr )=0. \end{aligned}$$

Similarly, $f_{\{l\}}(x_l)=\phi _l(x_l)$ if $l\in {\mathcal {S}}_1\cup {\mathcal {S}}_2^\mathrm{{var}}$. $\square $

Appendix B: Some Standard Concentration Results

First, we recall that the sub-Gaussian norm of a random variable X is defined as

$$\begin{aligned} \Vert {X}\Vert _{\psi _2}=\sup _{p\ge 1}p^{-1/2}(\mathbb {E}|X|^p)^{1/p}, \end{aligned}$$

and X is called a sub-Gaussian random variable if $\Vert {X}\Vert _{\psi _2}$ is finite. Similarly, the sub-exponential norm of a random variable is the quantity

$$\begin{aligned} \Vert {X}\Vert _{\psi _1}=\sup _{p\ge 1}p^{-1}(\mathbb {E}|X|^p)^{1/p}, \end{aligned}$$

and X is called sub-exponential if $\Vert {X}\Vert _{\psi _1}$ is finite. Next, we recall the following concentration results for sums of i.i.d. sub-Gaussian and sub-exponential random variables.

Proposition 4

[44, Proposition 5.16] Let $X_1,\dots ,X_N$ be independent centered sub-exponential random variables, and $K = \max _i \Vert {X_i}\Vert _{\psi _1}$. Then for every ${\mathbf {a}}= (a_1,\dots ,a_N) \in \mathbb {R}^N$ and every $t \ge 0$, we have

$$\begin{aligned} \mathbb {P}\left( \Bigl |\sum _i a_i X_i\Bigr | \ge t \right) \le 2 \exp \left[ -c\min \left\{ {\frac{t^2}{K^2\Vert {{\mathbf {a}}}\Vert _2^2},\frac{t}{K \Vert {{\mathbf {a}}}\Vert _{\infty }}}\right\} \right] ,\end{aligned}$$

where $c > 0$ is an absolute constant.

Proposition 5

[44, Proposition 5.10] Let $X_1,\dots ,X_N$ be independent centered sub-Gaussian random variables, and $K = \max _i \Vert {X_i}\Vert _{\psi _2}$. Then for every ${\mathbf {a}}= (a_1,\dots ,a_N) \in \mathbb {R}^N$ and every $t \ge 0$, we have

$$\begin{aligned} \mathbb {P}\left( \Bigl |\sum _i a_i X_i\Bigr | \ge t \right) \le e \cdot \exp \left[ - \frac{c t^2}{K^2\Vert {{\mathbf {a}}}\Vert _2^2}\right] , \end{aligned}$$

where $c > 0$ is an absolute constant.

Let $\varvec{\eta }= (\eta _1 \ \eta _2 \ \dots \ \eta _n)^T$, where $\eta _i \sim {\mathcal {N}}(0,\sigma ^2)$ are i.i.d. for each i. The Proposition below is a standard concentration result, stating that $\Vert {\varvec{\eta }}\Vert _2 = \Theta (\sigma \sqrt{n})$ and $\Vert {\varvec{\eta }}\Vert _1 = \Theta (\sigma n)$, with high probability. We provide proofs for completeness.

Proposition 6

Let $\varvec{\eta }= (\eta _1 \ \eta _2 \ \dots \ \eta _n)^T$ where $\eta _i \sim {\mathcal {N}}(0,\sigma ^2)$ i.i.d for each i. Then, there exist constants $c_1, c_2 > 0$ so that for any $\epsilon \in (0,1)$, we have:

1.
$\mathbb {P}\left( \Vert {\varvec{\eta }}\Vert _2 \in \left[ (1-\epsilon )\sigma \sqrt{n}, (1+\epsilon )\sigma \sqrt{n} \ \right] \right) \ge 1-2\exp \left( - c_1 \epsilon ^2 n \right) $, and
2.
$\mathbb {P}\left( \Vert {\varvec{\eta }}\Vert _1 \in \left[ (1-\epsilon ) n \sigma \sqrt{\frac{2}{\pi }}, (1+\epsilon )n \sigma \sqrt{\frac{2}{\pi }}\right] \right) \ge 1- e \cdot \exp \left( -\frac{2 c_2 \epsilon ^2 n}{\pi }\right) $.

Proof

1.

Note that $\eta _i$ are i.i.d. sub-Gaussian random variables with $\Vert {\eta _i}\Vert _{\psi _2} \le C_1 \sigma $. Hence $\eta _i^2$ are i.i.d. sub-exponential⁴ with
$$\begin{aligned} \Vert {\eta _i^2}\Vert _{\psi _1} \le 2 \Vert {\eta _i}\Vert _{\psi _2}^2 \le C_2 \sigma ^2 \end{aligned}$$
for some constant $C_2 > 0$. This implies that $\eta _i^2 - \mathbb {E}[\eta _i^2]$ are i.i.d. sub-exponential with
$$\begin{aligned} \Vert {\eta _i^2 - \mathbb {E}[\eta _i^2]}\Vert _{\psi _1} \le 2\Vert {\eta _i^2}\Vert _{\psi _1} \le C_3\sigma ^2 \end{aligned}$$
for some constant $C_3 > 0$. Using Proposition 4 with $t=n\epsilon \sigma ^2$ for $0<\epsilon <1$, we obtain for some constant $c_1 > 0$,
$$\begin{aligned} \mathbb {P}\left( \Bigl |\sum _{i=1}^{n} (\eta _i^2 - \mathbb {E}[\eta _i^2])\Bigr | \le n \epsilon \sigma ^2 \right) \ge 1-2\exp \left[ -c_1 \min \left\{ {\epsilon ^2, \epsilon }\right\} n \right] . \end{aligned}$$
Together with some standard manipulations this completes the proof.
2.

Note that $|{\eta _i}| - \mathbb {E}[|{\eta _i}|]$ is sub-Gaussian with
$$\begin{aligned} \Vert {|{\eta _i}| - \mathbb {E}[|{\eta _i}|]}\Vert _{\psi _2} \le 2 \Vert {\eta _i}\Vert _{\psi _2} \le C \sigma \end{aligned}$$
for some constant $C > 0$. Using Proposition 5 with $t = n \epsilon \mathbb {E}[|{\eta _1}|]$ for $\epsilon > 0$, we hence obtain
$$\begin{aligned} \mathbb {P}\left( \Bigl |\sum _{i=1}^{n} (|{\eta _i}| - \mathbb {E}[|{\eta _i}|])\Bigr | \le n \epsilon \mathbb {E}[|{\eta _1}|] \right) \ge 1-e \cdot \exp \left[ - \frac{c_2 n \epsilon ^2 (\mathbb {E}[|{\eta _1}|])^2}{\sigma ^2} \right] \end{aligned}$$
for some constant $c_2 > 0$. Finally, observing that $\mathbb {E}[|{\eta _1}|] = \sigma \sqrt{\frac{2}{\pi }}$ (cf., [48]) completes the proof.

$\square $

Appendix C: Proof of Theorem 2

The proof of part (1) follows in the same manner as [9, Proposition 1], with minor differences in calculation at certain parts. For completeness, we outline the main steps below.

Invoking the Hanson–Wright inequality for quadratic forms [21], we get for some constant $c > 0$ and all $t>0$,

$$\begin{aligned} \mathbb {P}(|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }- \mathbb {E}[\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }]}| > t) \le 2\exp \biggl [-c \min \left\{ {\frac{t^2}{K^4 \Vert {{\mathbf {A}}}\Vert _F^2}, \frac{t}{K^2\Vert {{\mathbf {A}}}\Vert }}\right\} \biggr ], \end{aligned}$$

(C.1)

where $\Vert {\beta _i}\Vert _{\psi _2} \le K$. For a Rademacher random variable $\beta _i$, $K = 1$ and

$$\begin{aligned} \mathbb {E}[\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }] =\mathbb {E}\sum _{i\not =j}\beta _i\beta _jA_{ij}=0. \end{aligned}$$

Using $\Vert {{\mathbf {A}}}\Vert \le \Vert {{\mathbf {A}}}\Vert _F$, (C.1) implies for every $t>0$,

$$\begin{aligned} \mathbb {P}(|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}| > t) \le 2\exp \biggl [-c \min \left\{ {\frac{t^2}{\Vert {{\mathbf {A}}}\Vert _F^2}, \frac{t}{\Vert {{\mathbf {A}}}\Vert _F}}\right\} \biggr ]. \end{aligned}$$

(C.2)

We will now find upper and lower bounds on $\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|]$. The upper bound is easy since (C.2) implies

$$\begin{aligned} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|]=\int _0^\infty \mathbb {P}(|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|>t) \mathrm{d}t\le c'\Vert {{\mathbf {A}}}\Vert _F. \end{aligned}$$

(C.3)

In order to find the lower bound, we have via repeated application of the Cauchy Schwartz inequality

$$\begin{aligned} \bigl (\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^2]\bigr )^2\le & {} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|]\cdot \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^3]\\\le & {} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|]\cdot \bigl (\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^2]\bigr )^{1/2} \cdot \bigl (\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^4]\bigr )^{1/2} \end{aligned}$$

and

$$\begin{aligned} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|] \ge \sqrt{\frac{(\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^2])^3}{\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^4]}}. \end{aligned}$$

Since $\varvec{\beta }$ consists of i.i.d. Rademacher random variables, we obtain

$$\begin{aligned} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^2] = 2\Vert {{\mathbf {A}}}\Vert _F^2 = \Vert {{\mathbf {a}}}\Vert _2^2. \end{aligned}$$

Moreover, an argument similar to (C.3) gives $\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|^4]\le c''\Vert {{\mathbf {A}}}\Vert _F^4$. Hence,

$$\begin{aligned} \mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|] \ge \sqrt{\frac{8\Vert {{\mathbf {A}}}\Vert _F^6}{c'' \Vert {{\mathbf {A}}}\Vert _F^4}} = {\tilde{c}} \Vert {{\mathbf {A}}}\Vert _F. \end{aligned}$$

(C.4)

Eqs. (C.3), (C.4) give us upper and lower bounds for $\mathbb {E}[|{\varvec{\beta }^T{\mathbf {A}}\varvec{\beta }}|]$. As a last step, we consider the zero mean random variables $X_1, \dots , X_n$, where $X_i = |{\varvec{\beta }_i^T{\mathbf {A}}\varvec{\beta }_i}| - \mathbb {E}{|{\varvec{\beta }_i^T{\mathbf {A}}\varvec{\beta }_i}|}$. A simple modification of (C.3) shows that they are sub-exponential with $\Vert {X_i}\Vert _{\psi _1}\le c\Vert {{\mathbf {A}}}\Vert _F.$ We can therefore apply a standard concentration bound (see [44, Proposition 5.16]) to bound the deviation

$$\begin{aligned} \biggl |\frac{1}{n}\Vert {{\mathbf {B}}{\mathbf {a}}}\Vert _1 - \frac{1}{n}\mathbb {E}[\Vert {{\mathbf {B}}{\mathbf {a}}}\Vert _1]\biggr |=\frac{1}{n}\biggl |\sum _{i=1}^nX_i\biggr |. \end{aligned}$$

A straightforward calculation then yields the statement of part (1) of the Theorem.

Part (2) follows from standard arguments based on $\epsilon $-nets, detailed for instance in [2].

The proof of part (3) copies that of [9, Theorem 3], which again is inspired by [7]. We sketch the main steps below for completeness. Setting $\widehat{{\mathbf {a}}} = {\mathbf {a}}+ {\mathbf {h}}$, we have by feasibility of ${\mathbf {a}}$ that $\frac{1}{n}\Vert {{\mathbf {B}}{\mathbf {h}}}\Vert _1 \le \frac{2\nu }{n}$. Defining $\Omega _0$ to be the set of indices corresponding to the $k$ largest entries of ${\mathbf {a}}$, we get ${{\mathbf {a}}} = {\mathbf {a}}_{\Omega _0} + {\mathbf {a}}_{\Omega _0^c}$. For a suitable positive integer K, we define $\Omega _1$ as the set of indices of the K largest entries of ${\mathbf {h}}_{\Omega _0^{c}}$, $\Omega _2$ as the set of indices of the K largest entries of ${\mathbf {h}}$ on $(\Omega _0 \cup \Omega _1)^{c}$, and so on. Following this argument of [7] gives the proof.

The proof of part (4) follows by choosing $K = 4 \bigl (\frac{4 c_2}{c_1}\bigr )^2k$. Indeed, (6.8) gives

$$\begin{aligned} \frac{1-\gamma ^{\mathrm {lb}}_{k+K}}{\sqrt{2}} - (1+\gamma ^{\mathrm {ub}}_{K})\sqrt{\frac{k}{K}}\ge \frac{c_1}{2\sqrt{2}}-2c_2\cdot \frac{c_1}{8c_2}=\frac{(\sqrt{2}-1)c_1}{4}=\beta >0 \end{aligned}$$

if $n>c_3'(k+K)\log (d^2/(k+K))$, with probability at least $1-e^{-C_4 n}$ for some constant $C_4 > 0$.

References

1.
Alizadeh, F., Goldfarb, D.D.: Second-order cone programming. Math. Program. Ser. B 95(1), 3–51 (2003)MathSciNetCrossRefGoogle Scholar
2.
Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constr. Approx. 28(3), 253–263 (2008)MathSciNetCrossRefGoogle Scholar
3.
Bien, J., Taylor, J., Tibshirani, R.: A Lasso for hierarchical interactions. Ann. Stat. 41(3), 1111–1141 (2013)MathSciNetCrossRefGoogle Scholar
4.
Blöchl, P.E.: Generalized separable potentials for electronic-structure calculations. Phys. Rev. B 41, 5414–5416 (1990)CrossRefGoogle Scholar
5.
Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harm. Anal. 27(3), 265–274 (2009)MathSciNetCrossRefGoogle Scholar
6.
Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Select. Top. Signal Proc. 4(2), 298–309 (2010)CrossRefGoogle Scholar
7.
Candès, E.J.: The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique 346(9–10), 589–592 (2008)MathSciNetCrossRefGoogle Scholar
8.
Candès, E.J., Romberg, J.: $\ell _{1}$-magic: recovery of sparse signals via convex programming (2005). Available at http://www.acm.caltech.edu/l1magic/
9.
Chen, Y., Chi, Y., Goldsmith, A.J.: Exact and stable covariance estimation from quadratic sampling via convex programming. IEEE Trans. Inf. Theory 61(7), 4034–4059 (2015)MathSciNetCrossRefGoogle Scholar
10.
Choi, N.H., Li, W., Zhu, J.: Variable selection with the strong heredity constraint and its oracle property. J. Am. Stat. Assoc. 105(489), 354–364 (2010)MathSciNetCrossRefGoogle Scholar
11.
Cohen, A., DeVore, R., Schwab, C.: Convergence rates of best n-term Galerkin approximations for a class of elliptic spdes. Found. Comput. Math. 10(6), 615–646 (2010)MathSciNetCrossRefGoogle Scholar
12.
Comminges, L., Dalalyan, A.S.: Tight conditions for consistency of variable selection in the context of high dimensionality. Ann. Stat. 40(5), 2667–2696 (2012)MathSciNetCrossRefGoogle Scholar
13.
Comminges, L., Dalalyan, A.S.: Tight conditions for consistent variable selection in high dimensional nonparametric regression. J. Mach. Learn. Res. 19, 187–206 (2012)zbMATHGoogle Scholar
14.
Dalalyan, A., Ingster, Y., Tsybakov, A.B.: Statistical inference in compound functional models. Probab. Theory Relat. Fields 158(3–4), 513–532 (2014)MathSciNetCrossRefGoogle Scholar
15.
DeVore, R., Petrova, G., Wojtaszczyk, P.: Approximation of functions of few variables in high dimensions. Constr. Approx. 33, 125–143 (2011)MathSciNetCrossRefGoogle Scholar
16.
Dibangoye, S.J., Amato, C., Buffet, O., Charpillet, F.: Exploiting separability in multiagent planning with continuous-state mdps. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS ’14, pp. 1281–1288. International Foundation for Autonomous Agents and Multiagent Systems (2014)Google Scholar
17.
Foucart, S., Rauhut, H.: A Mathematical Introduction to Compressive Sensing. Birkhäuser/Springer, New York (2013)CrossRefGoogle Scholar
18.
Ghiringhelli, L.M., Vybíral, J., Levchenko, S.V., Draxl, C., Scheffler, M.: Big data of materials science: critical role of the descriptor. Phys. Rev. Lett. 114(10), 105503 (2015)CrossRefGoogle Scholar
19.
Goel, G., Chou, I.-C., Voit, E.O.: System estimation from metabolic time-series data. Bioinformatics 24(21), 2505–2511 (2008)CrossRefGoogle Scholar
20.
Griewank, A., Toint, P.L.: On the unconstrained optimization of partially separable functions. In Nonlinear Optimization 1981, pp. 301–312. Academic Press (1982)Google Scholar
21.
Hanson, D.L., Wright, F.T.: A bound on tail probabilities for quadratic forms in independent random variables. Ann. Math. Stat. 42(3), 1079–1083 (1971)MathSciNetCrossRefGoogle Scholar
22.
Holtz, M.: Sparse Grid Quadrature in High Dimensions with Applications in Finance and Insurance, vol. 77. Springer, New York (2010)zbMATHGoogle Scholar
23.
Huang, J., Horowitz, J.L., Wei, F.: Variable selection in nonparametric additive models. Ann. Stat. 38(4), 2282–2313 (2010)MathSciNetCrossRefGoogle Scholar
24.
Kekatos, V., Giannakis, G.B.: Sparse volterra and polynomial regression models: recoverability and estimation. Trans. Sig. Proc. 59(12), 5907–5920 (2011)MathSciNetCrossRefGoogle Scholar
25.
Koltchinskii, V., Yuan, M.: Sparse recovery in large ensembles of kernel machines. In: 21st Annual Conference on Learning Theory (COLT), pp. 229–238 (2008)Google Scholar
26.
Koltchinskii, V., Yuan, M.: Sparsity in multiple kernel learning. Ann. Stat. 38(6), 3660–3695 (2010)MathSciNetCrossRefGoogle Scholar
27.
Lin, Y., Zhang, H.H.: Component selection and smoothing in multivariate nonparametric regression. Ann. Stat. 34(5), 2272–2297 (2006)MathSciNetCrossRefGoogle Scholar
28.
Meier, L., Van de Geer, S., Bühlmann, P.: High-dimensional additive modeling. Ann. Stat. 37(6B), 3779–3821 (2009)MathSciNetCrossRefGoogle Scholar
29.
Mossel, E., O’Donnell, R., Servedio, R.: Learning juntas. In: 35th Annual ACM Symposium on Theory of Computing (STOC), pp. 206–212 (2003)Google Scholar
30.
Nazer, B., Nowak, R.D.: Sparse interactions: Identifying high-dimensional multilinear systems via compressed sensing. In: 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1589–1596 (2010)Google Scholar
31.
Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. Society for Industrial and Applied Mathematics (1994)Google Scholar
32.
Novak, E., Triebel, H.: Function spaces in lipschitz domains and optimal rates of convergence for sampling. Constr. Approx. 23(3), 325–350 (2006)MathSciNetCrossRefGoogle Scholar
33.
Novak, E., Woźniakowski, H.: Approximation of infinitely differentiable multivariate functions is intractable. J. Compl. 25, 398–404 (2009)MathSciNetCrossRefGoogle Scholar
34.
Radchenko, P., James, G.M.: Variable selection using adaptive nonlinear interaction structures in high dimensions. J. Am. Stat. Assoc. 105, 1541–1553 (2010)MathSciNetCrossRefGoogle Scholar
35.
Raskutti, G., Wainwright, M.J., Yu, B.: Minimax-optimal rates for sparse additive models over kernel classes via convex programming. J. Mach. Learn. Res. 13(1), 389–427 (2012)MathSciNetzbMATHGoogle Scholar
36.
Rauhut, H.: Compressive sensing and structured random matrices. Theor. Found. Numer. Methods Sparse Recovery 9, 1–92 (2010)MathSciNetzbMATHGoogle Scholar
37.
Ravikumar, P., Lafferty, J., Liu, H., Wasserman, L.: Sparse additive models. J. R. Stat. Soc.: Ser. B (Stat. Methodol.) 71(5), 1009–1030 (2009)MathSciNetCrossRefGoogle Scholar
38.
Schnass, K., Vybíral, J.: Compressed learning of high-dimensional sparse functions. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3924–3927 (2011)Google Scholar
39.
Shan, S., Wang, G.G.: Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct. Multidiscip. Optim. 41(2), 219–241 (2010)MathSciNetCrossRefGoogle Scholar
40.
Storlie, C.B., Bondell, H.D., Reich, B.J., Zhang, H.H.: Surface estimation, variable selection, and the nonparametric oracle property. Statistica Sinica 21(2), 679–705 (2011)MathSciNetCrossRefGoogle Scholar
41.
Tyagi, H., Krause, A., Gärtner, B.: Efficient sampling for learning sparse additive models in high dimensions. Adv. Neural Inf. Process. Syst. (NIPS) 27, 514–522 (2014)Google Scholar
42.
Tyagi, H., Kyrillidis, A., Gärtner, B., Krause, A.: Learning sparse additive models with interactions in high dimensions. In: 19th International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 111–120 (2016)Google Scholar
43.
Tyagi, H., Kyrillidis, A., Gärtner, B., Krause, A.: Algorithms for learning sparse additive models with interactions in high dimensions. Inf. Inference: J. IMA, iax008 (2017)Google Scholar
44.
Vershynin, R.: Introduction to the non-asymptotic analysis of random matrices. In: Compressed Sensing: Theory and Applications, pp. 210–268. Cambridge University Press (2012)Google Scholar
45.
Vybíral, J.: Sampling numbers and function spaces. J. Compl. 23(4–6), 773–792 (2007)MathSciNetCrossRefGoogle Scholar
46.
Vybíral, J.: Widths of embeddings in function spaces. J. Compl. 24(4), 545–570 (2008)MathSciNetCrossRefGoogle Scholar
47.
Wahl, M.: Variable selection in high-dimensional additive models based on norms of projections (2015). ArXiv e-prints, arXiv:1406.0052
48.
Winkelbauer, A.: Moments and absolute moments of the normal distribution (2014). ArXiv e-prints, arXiv:1209.4340v2
49.
Yang, Y., Tokdar, S.T.: Minimax-optimal nonparametric regression in high dimensions. Ann. Stat. 43(2), 652–674 (2015)MathSciNetCrossRefGoogle Scholar
50.
Zhu, P., Morelli, J., Ferrari, S.: Value function approximation for the control of multiscale dynamical systems. In: 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 5471–5477 (2016)Google Scholar

Copyright information

Authors and Affiliations

Hemant Tyagi
- 1
Jan Vybiral
- 2
Email author

1.INRIA Lille - Nord EuropeVilleneuve d’AscqFrance
2.Department of Mathematics FNSPECzech Technical University in PraguePragueCzech Republic

Learning General Sparse Additive Models from Point Queries in High Dimensions

Abstract

Keywords

Mathematics Subject Classification

Notes

References

Copyright information

Authors and Affiliations

Personalised recommendations

Cite article