Sample average approximation with heavier tails II: localization in stochastic convex optimization and persistence results for the Lasso

Abstract

“Localization” has proven to be a valuable tool in the Statistical Learning literature as it allows sharp risk bounds in terms of the problem geometry. Localized bounds seem to be much less exploited in the stochastic optimization literature. In addition, there is an obvious interest in both communities in obtaining risk bounds that require weak moment assumptions or “heavier-tails”. In this work we use a localization toolbox to derive risk bounds in two specific applications. The first is in portfolio risk minimization with conditional value-at-risk constraints. We consider a setting where, among all assets with high returns, there is a portion of dimension g, unknown to the investor, that has significant less risk than the other remaining portion. Our rates for the SAA problem show that “risk inflation”, caused by a multiplicative factor, affects the statistical rate only via a term proportional to g. As the “normalized risk” increases, the contribution in the rate from the extrinsic dimension diminishes while the dependence on g is kept fixed. Localization is a key tool to show this property. As a second application of our localization toolbox, we obtain sharp oracle inequalities for least-squares estimators with a Lasso-type constraint under weak moment assumptions. One main consequence of these inequalities is to obtain persistence, as posed by Greenshtein and Ritov, with covariates having heavier tails. This gives improvements in prior work of Bartlett, Mendelson and Neeman.

Appendix

of Proposition 1

Given admissible sequences $\{{\mathcal {A}}_{1,j}\}_{j\ge 0}$ and $\{{\mathcal {A}}_{2,j}\}_{j\ge 0}$ for ${\mathcal {M}}_1$ and ${\mathcal {M}}_2$, one may define an admissible sequence $\{{\mathcal {C}}_j\}_{j\ge 0}$ for ${\mathcal {M}}$ via:

$$\begin{aligned}{\mathcal {C}}_0:=\{{\mathcal {M}}\}\hbox { and }{\mathcal {C}}_j:= {\mathcal {A}}_{1,j-1}\times {\mathcal {A}}_{2,j-1}\,(j\ge 1).\end{aligned}$$

It is easy to see that this is indeed admissible and moreover

$$\begin{aligned}\text {\textsf{diam}}({\mathcal {C}}_0) = \text {\textsf{diam}}({\mathcal {M}})= \text {\textsf{diam}}({\mathcal {M}}_1) + \text {\textsf{diam}}({\mathcal {M}}_2),\\\text {\textsf{diam}}({\mathcal {C}}_j)\le \text {\textsf{diam}}({\mathcal {A}}_{1,j-1}) + \text {\textsf{diam}}({\mathcal {A}}_{2,j-1}).\end{aligned}$$

Therefore,

$$\begin{aligned}\gamma _2^{(\alpha )}({\mathcal {M}})\le \text {\textsf{diam}}({\mathcal {M}})^\alpha + \sum _{j\ge 1}2^{j/2}(\text {\textsf{diam}}({\mathcal {A}}_{1,j-1})^\alpha + \text {\textsf{diam}}({\mathcal {A}}_{2,j-1})^\alpha )\end{aligned}$$

or equivalently

$$\begin{aligned}\gamma _2^{(\alpha )}({\mathcal {M}})\le & {} \text {\textsf{diam}}({\mathcal {M}}_1)^\alpha + \text {\textsf{diam}}({\mathcal {M}}_2)^\alpha \\ {}{} & {} + \sqrt{2}\left( \sum _{j\ge 0}2^{j/2}\text {\textsf{diam}}({\mathcal {A}}_{1,j-1})^\alpha \right) +\sqrt{2}\left( \sum _{j\ge 0}2^{j/2}\text {\textsf{diam}}({\mathcal {A}}_{2,j-1})^\alpha \right) .\end{aligned}$$

The proof finishes when we note that $\text {\textsf{diam}}({\mathcal {M}})^{\alpha }\le \gamma ^{(\alpha )}_2({\mathcal {M}})$ and take the infimum over admissible sequences. $\square$

We recall the following fundamental result due to Panchenko. It establishes a sub-Gaussian tail for the deviation of an heavy-tailed empirical process around its mean after a proper self-normalization by a random quantity ${{\widehat{V}}}$.

Theorem 4

(Panchenko’s inequality [39]) Let ${\mathcal {F}}$ be a finite family of measurable functions $g:\Xi \rightarrow {\mathbb {R}}$ such that ${\textbf{P}}g^2(\cdot )<\infty$. Let also $\{\xi _j\}_{j=1}^N$ and $\{\eta _j\}_{j=1}^N$ be both i.i.d. samples drawn from a distribution ${\textbf{P}}$ over $\Xi$ which are independent of each other. Define

$$\begin{aligned} {\textsf{S}}:=\sup _{g\in {\mathcal {F}}}\sum _{j=1}^Ng(\xi _j),\quad \quad \hbox {and}\quad \quad {\widehat{V}}:={\mathbb {E}}\left\{ \sup _{g\in {\mathcal {F}}}\sum _{j=1}^N\left[ g(\xi _j)-g(\eta _j)\right] ^2\Bigg |\sigma (\xi _1,\ldots ,\xi _N)\right\} . \end{aligned}$$

Then, for all $t>0$,

$$\begin{aligned} {\mathbb {P}}\left\{ {\textsf{S}}-{\mathbb {E}}[{\textsf{S}}]\ge \sqrt{\frac{2(1+t)}{N}{\widehat{V}}}\right\} \bigvee {\mathbb {P}}\left\{ {\textsf{S}}-{\mathbb {E}}[{\textsf{S}}]\le -\sqrt{\frac{2(1+t)}{N}{\widehat{V}}}\right\} \le 2e^{-t}. \end{aligned}$$

The following result is a direct consequence of Theorem 4 applied to the unitary class ${\mathcal {F}}:=\{g\}$. It provides a sub-Gaussian tail for any random variable with finite 2nd moment in terms its variance and empirical variance.

Lemma 8

(Sub-Gaussian tail for self-normalized sums) Suppose $\{\xi _j\}_{j=1}^N$ is i.i.d. sample of a distribution ${\textbf{P}}$ over $\Xi$ and denote by ${{\widehat{{\textbf{P}}}}}$ the correspondent empirical distribution. Then for any measurable function $g:\Xi \rightarrow {\mathbb {R}}$ satisfying ${\textbf{P}}g(\cdot )^2<\infty$ and, for any $t>0$,

$$\begin{aligned}{} & {} {\mathbb {P}}\left\{ ({{\widehat{{\textbf{P}}}}}-{\textbf{P}})g(\cdot )\ge \sqrt{\frac{2(1+t)}{N}\left( {{\widehat{{\textbf{P}}}}}+{\textbf{P}}\right) \left[ g(\cdot )-{\textbf{P}}g(\cdot )\right] ^2}\right\} \le 2e^{-t},\\{} & {} {\mathbb {P}}\left\{ ({{\widehat{{\textbf{P}}}}}-{\textbf{P}})g(\cdot )\le -\sqrt{\frac{2(1+t)}{N}\left( {{\widehat{{\textbf{P}}}}}+{\textbf{P}}\right) \left[ g(\cdot )-{\textbf{P}}g(\cdot )\right] ^2}\right\} \le 2e^{-t}. \end{aligned}$$

Finally, we present the sub-gaussian tail of nonnegative random variables.

Lemma 9

(Sub-Gaussian lower tail for nonnegative random variables) Let $\{Z_j\}_{j=1}^N$ be i.i.d. nonnegative random variables. Assume $a\in (1,2]$ and $0<{\mathbb {E}}[Z_1^a]<\infty$. Then, for all $\epsilon >0$,

$$\begin{aligned} {\mathbb {P}}\left\{ \frac{1}{N}\sum _{j=1}^NZ_j\le (1-\epsilon ){\mathbb {E}}[Z_1]\right\} \le \exp \left\{ -\left( \frac{a-1}{a}\right) \epsilon ^{\frac{a-1}{a}}\left\{ \frac{({\mathbb {E}}[Z_1])^a}{{\mathbb {E}}[Z_1^a]}\right\} ^{\frac{1}{a-1}}N\right\} . \end{aligned}$$

Proof

Let $\theta ,\epsilon >0$. By the usual “Bernstein trick”, we get

$$\begin{aligned} {\mathbb {P}}\left\{ \frac{1}{N}\sum _{j=1}^NZ_j\le (1-\epsilon ){\mathbb {E}}[Z_1]\right\}\le & {} {\mathbb {P}}\left\{ \sum _{j=1}^N({\mathbb {E}}[Z_i]-Z_i)\ge \epsilon {\mathbb {E}}[Z_1]N\right\} \nonumber \\\le & {} {\mathbb {P}}\left\{ e^{\theta \sum _{j=1}^N({\mathbb {E}}[Z_i]-Z_i)}\ge e^{\theta \epsilon {\mathbb {E}}[Z_1]N}\right\} \nonumber \\\le & {} e^{-\theta \epsilon {\mathbb {E}}[Z_1]N}{\mathbb {E}}\left[ e^{\theta \sum _{j=1}^N({\mathbb {E}}[Z_i]-Z_i)}\right] \nonumber \\= & {} e^{-\theta \epsilon {\mathbb {E}}[Z_1]N}{\mathbb {E}}\left[ e^{\theta ({\mathbb {E}}[Z_1]-Z_1)}\right] ^N. \end{aligned}$$

(57)

It is a simple calculus exercise to show that $\forall x\ge 0,e^{-x}\le 1-x+\frac{x^a}{a}.$ Applying this with $x:=\theta Z_1$, we obtain

$$\begin{aligned} {\mathbb {E}}\left[ e^{\theta ({\mathbb {E}}[Z_1]-Z_1)}\right]\le & {} e^{\theta {\mathbb {E}}[Z_1]}\left( 1-{\mathbb {E}}[\theta Z_1]+\frac{{\mathbb {E}}[(\theta Z_1)^a]}{a}\right) \\\le & {} e^{\theta {\mathbb {E}}[Z_1]}e^{-\theta {\mathbb {E}}[Z_1]+\frac{{\mathbb {E}}[(\theta Z_1)^a]}{a}}=e^{\frac{{\mathbb {E}}[(\theta Z_1)^a]}{a}}, \end{aligned}$$

where the second inequality follows from the relation $1+x\le e^x$ for all $x\in {\mathbb {R}}$. We plug this back into (57) and get, for all $\theta >0$,

$$\begin{aligned} {\mathbb {P}}\left\{ \frac{1}{N}\sum _{j=1}^NZ_j\le (1-\epsilon ){\mathbb {E}}[Z_1]\right\}\le & {} e^{\left( -\theta \epsilon {\mathbb {E}}[Z_1]+\theta ^a\frac{{\mathbb {E}}[Z_1^a]}{a}\right) N}. \end{aligned}$$

(58)

Since $a\in (1,2]$, we may actually minimize the above bound over $\theta >0$. The minimum is attained at $\theta _*:=\left( \frac{\epsilon {\mathbb {E}}[Z_1]}{{\mathbb {E}}[Z_1^a]}\right) ^{\frac{1}{a-1}}.$ To finish the proof, we plug this in (58) and notice that

$$\begin{aligned} -\theta _*\epsilon {\mathbb {E}}[Z_1]+\theta _*^a\frac{{\mathbb {E}}[Z_1^a]}{a}=\left( -1+\frac{1}{a}\right) \frac{(\epsilon {\mathbb {E}}[Z_1])^{\frac{a}{a-1}}}{{\mathbb {E}}[Z_1^a]^{\frac{1}{a-1}}}, \end{aligned}$$

using that $1+\frac{1}{a-1}=\frac{a}{a-1}$. $\square$