Empirical Processes Indexed by Classes of Functions

Abstract

In this chapter, we give new uniform central limit theorems for general empirical processes indexed by classes of sets or classes of functions. In Sect. 8.2, we consider convex sets of functions embedded in spaces of regular functions.

Exercises

(1) Use Exercise 4 in Chap. 1 to prove that the map $f \rightarrow \Vert f \Vert _{2,\beta }$, which is defined in (8.21), is a norm on $L_{2,\beta } (P)$.

Problem In this problem, we will prove the uniform central limit theorem of Pollard (1982) for classes of functions satisfying a universal entropy condition, in a particular case.

Let $(X_i)_{i>0}$ be a sequence of independent random variables with values in some measure space $(\mathcal{X} , \mathcal{E})$, with common law P. Let $\mathcal{A} ( \mathcal{X})$ be the set of probability measures on $\mathcal X$ with finite support. For any Q in $\mathcal{A} ( \mathcal{X})$, we denote by $d_Q$ the pseudodistance associated to the usual norm in $L^2 (Q)$. Let $\mathcal F$ be a class of measurable functions from $\mathcal X$ into $[-1, 1]$. We set

$$ H (x , \mathcal{F} ) = \sup _{Q\in \mathcal{A} ( \mathcal{X})} H (x, \mathcal{F}, d_Q) , {(1)} $$

where $H (x, \mathcal{F} , d_Q)$ is defined as in Definition 8.4. The function $x\rightarrow H (x , \mathcal{F} )$ is called universal Koltchinskii–Pollard entropy. The universal entropy of $\mathcal{F}$ is said to be integrable if

$$ \int _0^1 \sqrt{ H (x, \mathcal{F} ) } \, dx < \infty . {(2)} $$

The class $\mathcal F$ is said to fulfill the measurability condition (M) if there exists a countably generated and locally compact space $(K, \mathcal{B} (K))$ equipped with its Borel $\sigma$-field and a surjective map T from K onto $\mathcal F$ such that the map $(x, y) \rightarrow T(y) (x)$ is measurable with respect to the $\sigma$-fields $(\mathcal{X} \times K, \mathcal{E} \otimes \mathcal{B} (K) )$ and $\mathcal{B} (\mathrm{I\! R})$.

I. A symmetrization inequality.

Here $\mathcal G$ is a class of measurable functions from $\mathcal X$ into $[-1, 1]$, satisfying condition (M).

(1) Prove that the map

$$ (x_1, \ldots , x_p, y_1, \ldots , y_q) \rightarrow \sup \{ g(x_1) + \cdots + g(x_p) - g(y_1) - \cdots - g(y_q) : g\in \mathcal{G} \} $$

is universally measurable in the sense of Definition E.1, Annex E.

(2) Let $(X'_i)_{i>0}$ be an independent copy of $(X_i)_{i>0}$. Let $P_n$ be the empirical measure associated to $X_1, \ldots , X_n$, as defined in (1.37), and let $P'_n$ denote the empirical measure associated to $X'_1, \ldots , X'_n$. Prove that the variables in (3) are measurable and that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } |P_n (g) - P(g)|\Bigr ) \le \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } |P_n (g) - P'_n(g)|\Bigr ) . {(3)} $$

Hint: apply Jensen’s inequality conditionally to $X_1, \ldots , X_n$.

Let $({\varepsilon }_i)_{i>0}$ be a sequence of symmetric independent signs, independent of the $\sigma$-field generated by $(X_i)_{i>0}$ and $(X'_i)_{i>0}$. Let $(X_i^s , X_i^{\prime s} )$ be defined by $(X_i^s , X_i^{\prime s}) = (X_i , X'_i)$ if ${\varepsilon }_i = 1$ and $(X_i^s , X_i^{\prime s}) = (X'_i , X_i)$ if ${\varepsilon }_i = -1$.

(3) Prove that the sequence $(X_i^s , X_i^{\prime s})_i$ is a sequence of independent random variables with common law $P \otimes P$.

(4) Starting from (3), prove that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } |P_n (g) - P'_n(g)|\Bigr ) = n^{-1} \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } \Big | \sum _{i=1}^n {\varepsilon }_i (g(X_i) - g(X'_i))|\Bigr ) . $$

(5) Prove that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } |P_n (g) - P(g)|\Bigr ) \le 2n^{-1} \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } \Big | \sum _{i=1}^n {\varepsilon }_i g(X_i) \Big | \Bigr ) . {(4)} $$

II. Stochastic equicontinuity of the symmetrized empirical process

Throughout Part II, we fix $(x_1, \ldots , x_n)$ in $\mathcal{X}^n$. We assume that the class of functions $\mathcal G$ satisfies the universal entropy condition (2). We set

$$ \varphi (\sigma , \mathcal{G}) = \int _0^\sigma \sqrt{H(x, \mathcal{G}) }\, dx . {(5)} $$

Let $Q_n = n^{-1} (\delta _{x_1} + \cdots + \delta _{x_n})$ denote the empirical measure associated to $(x_1, \ldots , x_n)$. We define the empirical maximal variance V by $V = V (x_1, \ldots , x_n) = \sup \{ Q_n (g^2) : g\in \mathcal{G} \}$. Let $\delta$ be any real in ]0, 1].

(1) Let I be a finite subset of $\mathcal G$ with cardinality $\exp (H) \ge 2$. Prove that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in I} \Big | \sum _{i=1}^n {\varepsilon }_i g(x_i) \Big | \Bigr ) \le 2 \sqrt{H} \sup _{g\in I} \Bigl (\sum _{i=1}^n g^2 (x_i) \Bigr )^{1/2} . {(6)} $$

(2) Prove that, for any nonnegative integer k, there exists a finite subset $\mathcal{G}_k$ of $\mathcal{G}$ with cardinality at most $\exp (H (2^{-k}\delta ))$ and such that there exists some map $\Pi _k$ from $\mathcal{G}$ into $\mathcal{G}_k$ satisfying the condition below:

$$ d_{Q_n} (g , \Pi _k g) \le 2^{-k} \delta \hbox { for any } g \in \mathcal{G} . $$

(3) Prove that, for any function g in $\mathcal G$ and any integer l,

$$ \Big | \sum _{i=1}^n {\varepsilon }_i (g-\Pi _l g) (x_i) \Big | \le n 2^{-l} \delta . $$

Infer from this inequality that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } \Big | \sum _{i=1}^n {\varepsilon }_i g(x_i) \Big | \Bigr ) = \lim _{l\rightarrow \infty } \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G}_l } \Big | \sum _{i=1}^n {\varepsilon }_i g(x_i) \Big | \Bigr ) . $$

(4) Let $\delta _k = 2^{-k}\delta$. Prove that, for any g in $\mathcal{G}_l$, there exists a collection of functions $g_0, \ldots , g_l$ satisfying $g_l = g$ and $g_k = \Pi _k g_{k+1}$ for any integer k in [0, l[. Infer that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G}_l } \Big | \sum _{i=1}^n {\varepsilon }_i g(x_i) \Big | \Bigr ) \le 2 \sum _{k=1}^l \delta _{k-1} \sqrt{nH (\delta _k)} + 2\sqrt{n H(\delta ) V}\, . $$

(5) Prove that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } \Big | \sum _{i=1}^n {\varepsilon }_i g(x_i) \Big | \Bigr ) \le 8\sqrt{n} \varphi (\delta /2, \mathcal{G} ) + 2\sqrt{n H(\delta ) V}\, . $$

Infer from the above inequality that

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } \Big | \sum _{i=1}^n {\varepsilon }_i g(X_i) \Big | \Bigr ) \le 8\sqrt{n} \varphi (\delta /2 , \mathcal{G} ) + 2\sqrt{n H(\delta ) \mathrm{I\! E}( V (X_1, \ldots , X_n) ) }\, . {(7)} $$

III. Modulus of continuity of the normalized empirical process

Let H be a nonincreasing entropy function such that

$$ \int _0^1 \sqrt{H (x) } dx < \infty \ \hbox {and let } \varphi (\sigma ) = \int _0^\sigma \sqrt{H (x) } dx . $$

Let $\mathcal{E} (\delta , P , H)$ be the set of classes of functions $\mathcal G$ from $\mathcal X$ into $[-1, 1]$, satisfying the measurability condition (M), such that $H ( x , \mathcal{G}) \le H (x)$ and $\sup \{ P (g^2) : g\in \mathcal{G} \} \le \delta ^2$. We set

$$ w (\delta ) = \sup _{\mathcal{G} \in \mathcal{E} (\delta , P , H)} \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G} } |Z_n (g)| \Bigr ) . $$

(1) Let $\mathcal G$ be any class of functions in $\mathcal{E} (\delta , P , H)$. Prove that the class $\{ g^2/2 : g \in \mathcal{G} \}$ still belongs to $\mathcal{E} (\delta , P , H)$. Infer that

$$ w(\delta ) \le 16 \varphi (\delta /2) + 4\sqrt{H (\delta ) }\, \sqrt{ \delta ^2+ 2 n^{-1/2}w(\delta ) } \, . {(8)} $$

Starting from (8), prove that

$$ w(\delta ) \le 16 \varphi (\delta ) + 4w(\delta ) \sqrt{ H(\delta ) / (n\delta ^2) } . $$

(2) Prove that $w(\delta ) \le 32 \varphi (\delta )$ for any positive $\delta$ satisfying $2^6 H(\delta ) \le n\delta ^2$.

(3) Prove that the class

$$ \mathcal{G}_\delta = \{ (f - g)/2 : (f, g) \in \mathcal{F} \times \mathcal{F} , d_P (f, g) \le \delta \} $$

belongs to $\mathcal{E} (\delta , P , H)$ for $H = H ( . , \mathcal{F})$. Infer that, if $n\delta ^2 \ge 2^6 H(\delta )$, then

$$ \mathrm{I\! E}\Bigl ( \sup _{g\in \mathcal{G}_\delta } |Z_n (2g)| \Bigr ) \le 64 \int _0^\delta \sqrt{ H (x, \mathcal{F} )}\, dx. {(9)} $$

Apply Theorem 7.1 to conclude.