Bounding the Expectation of the Supremum of Empirical Processes Indexed by Hölder Classes

Abstract

In this note, we provide upper bounds on the expectation of the supremum of empirical processes indexed by Hölder classes of any smoothness and for any distribution supported on a bounded set in $\mathbb{R}^{d}$. These results can alternatively be seen as non-asymptotic risk bounds, when the unknown distribution is estimated by its empirical counterpart, based on $n$ independent observations, and the error of estimation is quantified by integral probability metrics (IPM). In particular, IPM indexed by Hölder classes are considered and the corresponding rates are derived. These results interpolate between two well-known extreme cases: the rate $n^{-1/d}$ corresponding to the Wassertein-1 distance (the least smooth case) and the fast rate $n^{-1/2}$ corresponding to very smooth functions (for instance, functions from a RKHS defined by a bounded kernel).

APPENDIX

PROOFS

This section contains the proofs of the main results, Theorems 3 and 4, stated in the main body of the note.

A.1. Proof of Theorem 3

The proof of Theorem 3 can be found in [16]. We add it here for completeness.

Let $\gamma_{0}=S_{n}(\mathcal{F})=\sup_{f\in\mathcal{F}}||f||_{L_{2}(P_{n})}$. Define $\gamma_{j}=2^{-j}\gamma_{0}$, for every integer $j\in\mathbb{N}$, and let $T_{j}$ be a minimal $\gamma_{j}$-cover of $\mathcal{F}$ with respect to $L_{2}(P_{n})$. For any function $f\in\mathcal{F}$, we denote by $\widehat{f}_{j}$ an element of $T_{j}$ which is an $\gamma_{j}$ approximation of $f$. For any positive integer $N$ we can decompose the function $f$ as

$$f=f-\widehat{f}_{N}+\sum_{j=1}^{N}(\widehat{f}_{j}-\widehat{f}_{j-1}),$$

where $\widehat{f}_{0}=0\in\mathcal{F}$. Hence, for any positive integer $N$, we have

$$\widehat{R}_{n}(\mathcal{F})=\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}\left(f(X_{i})-\widehat{f}_{N}(X_{i})+\sum_{j=1}^{N}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right)\right]$$

$${}\leqslant\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(f(X_{i})-\widehat{f}_{N}(X_{i}))\right]+\sum_{j=1}^{N}\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right]$$

$${}\leqslant\frac{1}{n}\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}|(f(X_{i})-\widehat{f}_{N}(X_{i}))|+\sum_{j=1}^{N}\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right]$$

$${}=\sup_{f\in\mathcal{F}}||f-\widehat{f}_{N}||_{L_{2}(P_{n})}+\sum_{j=1}^{N}\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right]$$

$${}\leqslant\gamma_{N}+\sum_{j=1}^{N}\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right].$$

For any positive integer $j$, the triangle inequality gives

$$||\widehat{f}_{j}-\widehat{f}_{j-1}||_{L_{2}(P_{n})}\leqslant||\widehat{f}_{j}-f||_{L_{2}(P_{n})}+||f-\widehat{f}_{j-1}||_{L_{2}(P_{n})}\leqslant\gamma_{j}+\gamma_{j-1}=3\gamma_{j}.$$

(A.1)

We need the following classic lemma which controls the expectation of a Rademacher average over a finite set.Footnote

We refer the reader to https://ttic.uchicago.edu/t̃ewari/lectures/lecture10.pdf for a simple proof of this lemma.

We refer the reader to https://ttic.uchicago.edu/t̃ewari/lectures/lecture10.pdf for a simple proof of this lemma.

Lemma A.1 (Massart’s finite class lemma). Let $\mathcal{X}$ be a finite subset of $\mathbb{R}^{n}$ and let $\sigma_{1},\dots,\sigma_{n}$ be independent Rademacher random variables. Denote the radius of $\mathcal{X}$ by $R=\sup_{x\in\mathcal{X}}||x||$. Then, we have,

$$\mathbb{E}\left[\sup_{x\in\mathcal{X}}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}x_{i}\right]\leqslant R\frac{\sqrt{2\log|\mathcal{X}|}}{n}.$$

Applying this lemma to $\mathcal{X}_{j}=\left\{(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))_{i=1}^{n}\in\mathbb{R}^{n}:f\in\mathcal{F}\right\}$ for any $j=1,\dots,n$ and using (3), we get

$$\sum_{j=1}^{N}\frac{1}{n}\mathbb{E}_{\sigma}\left[\sup_{f\in\mathcal{F}}\sum_{i=1}^{n}\sigma_{i}(\widehat{f}_{j}(X_{i})-\widehat{f}_{j-1}(X_{i}))\right]\leqslant\sum_{j=1}^{N}3\gamma_{j}\frac{\sqrt{2\log(|T_{j}|\cdot|T_{j-1}|)}}{n}$$

Therefore we have

$$\widehat{R}_{n}(\mathcal{F})\leqslant\gamma_{N}+\sum_{j=1}^{N}3\gamma_{j}\frac{\sqrt{2\log(|T_{j}|\cdot|T_{j-1}|)}}{n}\leqslant\gamma_{N}+\frac{6}{n}\sum_{j=1}^{N}\gamma_{j}\sqrt{\log|T_{j}|}$$

$${}=\gamma_{N}+\frac{12}{n}\sum_{j=1}^{N}(\gamma_{j}-\gamma_{j+1})\sqrt{\log|T_{j}|}=\gamma_{N}+\frac{12}{n}\sum_{j=1}^{N}(\gamma_{j}-\gamma_{j+1})\sqrt{\log\mathcal{N}(\mathcal{F},L_{2}(P_{n}),\gamma_{j})}$$

$${}\leqslant\gamma_{N}+\frac{12}{n}\int\limits_{\gamma_{N+1}}^{\gamma_{0}}\sqrt{\log\mathcal{N}(\mathcal{F},L_{2}(P_{n}),\varepsilon)}d\varepsilon.$$

For any $\tau>0$, pick $N=\sup\{j:\gamma_{j}>2\tau\}$. Then $\gamma_{N}=2\gamma_{N+1}\leqslant 4\tau$ and $\gamma_{N+1}=\gamma_{N}/2\geqslant\tau$. Hence, we conclude that

$$\widehat{R}_{n}(\mathcal{F})\leqslant 4\tau+\frac{12}{\sqrt{n}}\int\limits_{\tau}^{\gamma_{0}}\sqrt{\log\mathcal{N}(\mathcal{F},L_{2}(P_{n}),\varepsilon)}d\varepsilon.$$

Since $\tau$ can take any positive value we can take the infimum over all positive $\tau$ and this concludes the proof.

A.2. Proof of Theorem 4

Without loss of generality, we prove the theorem in the case $L=1$. The general case will follow by homogeneity. For simplicity we write $\mathcal{H}^{\alpha}=\mathcal{H}^{\alpha}(1)$, $Ph=\int_{\mathcal{X}}hdP$ and $P_{n}h=\int_{\mathcal{X}}hdP_{n}$. A symmetrization argument (Lemma 1) gives

$$\mathbb{E}\bigg{[}\sup_{h\in\mathcal{H}^{\alpha}}|Ph-P_{n}h|\bigg{]}\leqslant 2\mathbb{E}\big{[}\widehat{R}_{n}(\mathcal{H}^{\alpha})\big{]},$$

where the empirical Rademacher process $\widehat{R}_{n}(\mathcal{H}^{\alpha})$ is given by

$$\widehat{R}_{n}(\mathcal{H}^{\alpha})=\frac{1}{n}\mathbb{E}\left[\sup_{h\in\mathcal{H}^{\alpha}}\sum_{i=1}^{n}\sigma_{i}h(X_{i})\bigg{|}X_{1},\ldots,X_{n}\right].$$

Noting that, for any $h\in\mathcal{H}^{\alpha}$,

$$P_{n}h^{2}:=\frac{1}{n}\sum_{i=1}^{n}h^{2}(X_{i})\leqslant||h^{2}||_{\infty}\leqslant 1,$$

the improved Dudley bound (Theorem 3) coupled with Lemma 2 yields,

$$\mathbb{E}\bigg{[}\sup_{h\in\mathcal{H}^{\alpha}}|P_{n}h-Ph|\bigg{]}\leqslant\inf_{\tau>0}\left(4\tau+\frac{12}{\sqrt{n}}\int\limits_{\tau}^{1}\sqrt{\log\mathcal{N}(\mathcal{H}^{\alpha},||\cdot||_{\infty},\varepsilon)}d\varepsilon\right)$$

$${}\leqslant\inf_{\tau>0}\left(4\tau+\frac{12\sqrt{K\lambda_{d}(\mathcal{X}^{1})}}{\sqrt{n}}\int\limits_{\tau}^{1}\varepsilon^{-d/2\alpha}d\varepsilon\right).$$

Applying Lemma A.2 with $\beta=\frac{d}{2\alpha}$ and $a=3\sqrt{\frac{K\lambda}{n}}$ where $K=K_{\alpha,d}$ is the constant depending only on $\alpha$ and $d$ borrowed from Theorem 1 and $\lambda:=\lambda_{d}(\mathcal{X}^{1})$, we get

$$\mathbb{E}\bigg{[}\sup_{h\in\mathcal{H}^{\alpha}}|P_{n}h-Ph|\bigg{]}\leqslant 12\begin{cases}\left(\frac{K\lambda}{n}\right)^{{\alpha}/{d}}\left[\frac{d}{d-2\alpha}\wedge(1+0.5\log(\frac{n}{9K\lambda}))\right]\quad\text{if $\alpha<{d}/{2}$}\\ \left(\frac{K\lambda}{n}\right)^{{1}/{2}}\left[\frac{2\alpha}{2\alpha-d}\wedge(1+\frac{\alpha}{d}\log(\frac{n}{9K\lambda}))\right]\quad\text{if $\alpha\geqslant{d}/{2}$}.\end{cases}$$

(A.2)

The proof is finished since the upper bound stated in Theorem 4 is a direct consequence of (A.2)

A.3. Additional Lemma

The following lemma enables to obtain an upper bound on Dudley’s refined bound (Theorem 3) for any bounded class whose entropy grows polynomially in $1/\varepsilon$.

Lemma A.2. For any real positive numbers $a$ and $\beta$, it holds

$$\min_{0\leqslant\tau\leqslant 1}\left(\tau+a\int\limits_{\tau}^{1}\varepsilon^{-\beta}d\varepsilon\right)\leqslant(a^{{1}/{\beta}}\vee a)\left[\left(\frac{\beta\vee 1}{|\beta-1|}\right))\wedge\left(1+\frac{\log({1}/{a})}{\beta\vee 1}\right)\right].$$

Proof. Let $a$ and $\beta$ be real positive numbers. Define the function

$$f\colon[0,1]\to\mathbb{R},$$

$$\tau\mapsto\tau+a\int\limits_{\tau}^{1}\varepsilon^{-\beta}d\varepsilon.$$

One can easily check that

$$f^{*}:=\min_{0\leqslant\tau\leqslant 1}f(\tau)=\begin{cases}1\quad\text{if $a>1$}\\ a^{{1}/{\beta}}+\frac{a}{1-\beta}(1-a^{{1}/{\beta}-1})\quad\text{if $a<1$ }.\end{cases}$$

(A.3)

In the case $a<1$, using the fact that $1-x^{\alpha}\leqslant\log(x^{-\alpha})$ for any $\alpha>0$ and $x\in(0,1]$, we have

$$f^{*}\leqslant(a^{{1}/{\beta}}\vee a)\left[\left(\frac{\beta\vee 1}{|\beta-1|}\right))\wedge\left(1+\frac{\log({1}/{a})}{\beta\vee 1}\right)\right].$$

(3)

Finally, since the RHS of (A.3) is greater than $1$ for any $a>1$, (A.3) holds for any positive real $a$ and this concludes the proof. $\Box$

ACKNOWLEDGMENTS

The author thanks Arnak Dalalyan for his diligent proofreading of this note, Yannick Guyonvarch for interesting references and Alexander Tsybakov for suggesting to present an extension of the main result.