Sample average approximation with heavier tails I: non-asymptotic bounds with weak assumptions and stochastic constraints

Abstract

We derive new and improved non-asymptotic deviation inequalities for the sample average approximation (SAA) of an optimization problem. Our results give strong error probability bounds that are “sub-Gaussian” even when the randomness of the problem is fairly heavy tailed. Additionally, we obtain good (often optimal) dependence on the sample size and geometrical parameters of the problem. Finally, we allow for random constraints on the SAA and unbounded feasible sets, which also do not seem to have been considered before in the non-asymptotic literature. Our proofs combine different ideas of potential independent interest: an adaptation of Talagrand’s “generic chaining” bound for sub-Gaussian processes; “localization” ideas from the Statistical Learning literature; and the use of standard conditions in Optimization (metric regularity, Slater-type conditions) to control fluctuations of the feasible set.

Appendix

Proof of Lemma 1

The second statement in the Lemma is a direct consequence of the first. Therefore, we will only prove the first statement.

Assume that $Z'_1,\dots ,Z'_n$ are independent copies of the $Z_1,\dots ,Z_n$. Also let $Z=(Z_1,\dots ,Z_n)^T$. What we want to prove is that, for any $t\ge 0$,

$$\begin{aligned} \mathbf{Want:}\;{\mathbb {P}}\left\{ {\mathbb {E}}\left[ \sum _{k=1}^N (Z_k - Z'_k)\mid Z\right] \ge \sqrt{2(1+t)\,\sum _{k=1}^N{\mathbb {E}}[(Z_k-Z'_k)^2\mid Z]} \right\} \le e^{-t}. \end{aligned}$$

By [31, Corollary 1], it suffices to prove that, for any $t\ge 0$,

$$\begin{aligned} \mathbf{Sufficient:}\;{\mathbb {P}}\left\{ \sum _{k=1}^N (Z_k - Z'_k)\ge \sqrt{2t\,\sum _{k=1}^N(Z_k-Z'_k)^2} \right\} \le e^{-t}. \end{aligned}$$

We will prove that the above inequality holds almost surely conditionally on values $|Z_k-Z'_k|=a_k$, $1\le k\le N$. Notice that, conditionally on these values,

$$\begin{aligned} Z_k - Z'_k = u_k\,a_k \end{aligned}$$

where the $u_k$ are i.i.d. unbiased random signs. So what we must show is that:

$$\begin{aligned} \forall t\ge 0\,:\, {\mathbb {P}}\left\{ \sum _{k=1}^N u_ia_i\ge \sqrt{2t \sum _{k=1}^Na_k^2}\right\} \le e^{-t}, \end{aligned}$$

for any choice of $a_k$, $1\le k\le N$. This follows easily from the standard inequalities:

$$\begin{aligned} \forall \theta >0\,:\,{\mathbb {E}}[e^{\theta \sum _{k=1}^N u_ia_i}] = \prod _{k=1}^N\cosh (\theta a_k)\le e^{\frac{\theta ^2\sum _{k=1}^Na_k^2}{2}}, \end{aligned}$$

and Bernstein’s trick:

$$\begin{aligned} {\mathbb {P}}\left\{ \sum _{k=1}^N u_ia_i\ge \sqrt{2t \sum _{k=1}^Na_k^2}\right\} \le \inf _{\theta >0}{\mathbb {E}}[e^{\theta \sum _{k=1}^N u_ia_i}]e^{-\theta \sqrt{2t \sum _{k=1}^Na_k^2}}\le e^{-t}. \end{aligned}$$

$\square$

Proof of Proposition 1

We will need the following Lemma.

Lemma 5

There exists a constant ${\mathbf{c}}_{{\mathsf {bdg}}}$ such that, for all $p\ge 2$ and all i.i.d. random variables $Z_1,\dots ,Z_N\in L^p$ with ${\mathbb {E}}[Z_i]=0$,

$$\begin{aligned} {\left\| \,\frac{Z_1 + \dots + Z_N}{N}\, \right\| }_{p}\le {\mathbf{c}}_{{\mathsf {bdg}}}\sqrt{\frac{p}{N}}\,{\left\| \,Z_1\, \right\| }_{p}, \end{aligned}$$

Proof of the Lemma

By the Burkholder–Davis–Gundy inequality and the subaditivity of the $L^{p/2}$ norm:

$$\begin{aligned} {\left\| \,Z_1 + \dots + Z_N\, \right\| }_{p}\le {\mathbf{c}}_{{\mathsf {bdg}}}\sqrt{p}\,{\left\| \,Z^2 _1+\dots +Z^2_N\, \right\| }_{p/2}^{1/2}\le {\mathbf{c}}_{{\mathsf {bdg}}}\sqrt{p\sum _{i=1}^N{\left\| \,Z^2_i\, \right\| }_{p/2}} \end{aligned}$$

and the proof finishes when we note ${\left\| \,Z^2_i\, \right\| }_{p/2} ={\left\| \,Z_1\, \right\| }_{p}^2$ for each index i.$\square$

Now note that the random variables

$$\begin{aligned} H_k:=\frac{h(\xi _k) - {\mathbf {E}}h(\cdot )}{\sigma ^2}\,\,(1\le k\le N) \end{aligned}$$

are i.i.d. and satisfy ${\mathbb {E}}[H_k]=0$, ${\left\| \,H_k\, \right\| }_{p}\le \kappa _p$. Markov’s inequality implies:

$$\begin{aligned} {\mathbb {P}}\left\{ {\widehat{{\mathbf {E}}}}h(\cdot )> 2\sigma ^2\right\} \le {\mathbb {P}}\left\{ \frac{1}{N}\sum _{k=1}^NH_k>1\right\} \le {\left\| \,\frac{1}{N}\sum _{k=1}^NH_k\, \right\| }_{p}^p. \end{aligned}$$

Now use Lemma 5 to bound the RHS.$\square$