Sharp oracle inequalities for low-complexity priors

Abstract

In this paper, we consider a high-dimensional statistical estimation problem in which the number of parameters is comparable or larger than the sample size. We present a unified analysis of the performance guarantees of exponential weighted aggregation and penalized estimators with a general class of data losses and priors which encourage objects which conform to some notion of simplicity/complexity. More precisely, we show that these two estimators satisfy sharp oracle inequalities for prediction ensuring their good theoretical performances. We also highlight the differences between them. When the noise is random, we provide oracle inequalities in probability using concentration inequalities. These results are then applied to several instances including the Lasso, the group Lasso, their analysis-type counterparts, the $\ell _\infty$ and the nuclear norm penalties. All our estimators can be efficiently implemented using proximal splitting algorithms.

Appendices

Prerequisites from convex analysis

We here collect some ingredients from convex analysis that are essential to our exposition.

Monotone conjugate

Lemma 2

Let g be a non-decreasing function on $\mathbb {R}_+$ that vanishes at 0. Then the following holds:

• (i) $g^+$ is a proper closed convex and non-decreasing function on $\mathbb {R}_+$ that vanishes at 0.

• (ii) If g is also closed and convex, then $g^{++}=g$.

• (iii) Let $f: t \in \mathbb {R}\mapsto g(|t|)$ such that f is differentiable on $\mathbb {R}$, where g is finite-valued, strictly convex and strongly coercive. Then $g^+$ is likewise finite-valued, strictly convex, strongly coercive, and $f^*=g^+ \circ |\cdot |$ is differentiable on $\mathbb {R}$. In particular, both g and $g^+$ are strictly increasing on $\mathbb {R}_+$.

Proof

• (i) By Bauschke and Combettes (2011, Proposition 13.11), $g^+$ is a closed convex function. We have $\inf _{t \ge 0} g(t)=-\sup _{t \ge 0} t \cdot 0 - g(t) = -g^+(0)$. Since g is non-decreasing and $g(0)=0$, then $g^+(0)=-\inf _{t \ge 0} g(t)=-g(0)=0$. In addition, by (5), we have $g^+(a) \ge a \cdot 0 - g(0)=0$, $\forall a \in \mathbb {R}_+$. This shows that $g^+$ is nonnegative and ${{\mathrm{dom}}}(g^+) \ne \emptyset$, and in turn, it is also proper.

  Let a, b in $\mathbb {R}_+$ such that $a < b$. Then

  $$\begin{aligned} g^+(a)-g^+(b)= & {} (\sup _{t \ge 0} t a - g(t)) - (\sup _{t' \ge 0} t' b - g(t')) \\\le & {} \sup _{t \ge 0} (t a - g(t) - t b + g(t)) \\= & {} \sup _{t \ge 0} t(a-b) = 0. \end{aligned}$$

  That is, $g^+$ is non-decreasing on $\mathbb {R}_+$.

• (ii) This follows from Rockafellar (1996, Theorem 12.4).

• (iii) By definition of f, f is a finite-valued function on $\mathbb {R}$, strictly convex, differentiable and strongly coercive. It then follows from Hiriart-Urruty and Lemaréchal (2001, Corollary X.4.1.4) that $f^*$ enjoys the same properties. In turn, using the fact that both f and $f^*$ are even, we have $g^+$ is strongly coercive, and strict convexity of f (resp. $f^*$) is equivalent to that of g (resp. $g^+$). Altogether, this shows the first claim. We now prove that g vanishes only at 0 (and similarly for $g^+$). As g is non-decreasing and strictly convex, we have, for any $\rho \in ]0,1[$ and a, b in $\mathbb {R}_+$ such that $a < b$,

  $$\begin{aligned} g(a)\le & {} g(\rho a + (1-\rho ) b) < \rho g(a) + (1-\rho ) g(b)\\\le & {} \rho g(b) + (1-\rho ) g(b) = g(b) . \end{aligned}$$

$\square$

Support function The support function of ${\mathcal C}\subset \mathbb {R}^p$ is

$$\begin{aligned} \sigma _{{\mathcal C}}({\varvec{\omega }})=\sup _{{\varvec{\theta }}\in {\mathcal C}} \langle {\varvec{\omega }},{\varvec{\theta }}\rangle . \end{aligned}$$

We recall the following properties whose proofs can be found in, e.g., Rockafellar (1996) and Hiriart-Urruty and Lemaréchal (2001).

Lemma 3

Let ${\mathcal C}$ be a non-empty set.

• (i) $\sigma _{\mathcal C}$ is proper lower semicontinuous (lsc) and sublinear.

• (ii) $\sigma _{\mathcal C}$ is finite-valued if and only if ${\mathcal C}$ is bounded.

• (iii) If $0 \in {\mathcal C}$, then $\sigma _{\mathcal C}$ is nonnegative.

• (iv) If ${\mathcal C}$ is convex and compact with $0 \in {{\mathrm{int}}}({\mathcal C})$, then $\sigma _{\mathcal C}$ is finite-valued and coercive.

Gauges and polars

Definition 3

(Polar set) Let ${\mathcal C}$ be a non-empty convex set. The set ${\mathcal C}^\circ$ given by

$$\begin{aligned} {\mathcal C}^\circ = \big \{ {\varvec{\eta }}\in \mathbb {R}^p \;:\; \langle {\varvec{\eta }},{\varvec{\theta }}\rangle \le 1 \quad \forall {\varvec{\theta }}\in {\mathcal C} \big \} \end{aligned}$$

is called the polar of ${\mathcal C}$.

The set ${\mathcal C}^\circ$ is closed convex and contains the origin. When ${\mathcal C}$ is also closed and contains the origin, then it coincides with its bipolar, i.e., ${\mathcal C}^{\circ \circ }={\mathcal C}$.

Let ${\mathcal C}\subseteq \mathbb {R}^p$ be a non-empty closed convex set containing the origin. The gauge of ${\mathcal C}$ is the function $\gamma _{\mathcal C}$ defined on $\mathbb {R}^p$ by

$$\begin{aligned} \gamma _{\mathcal C}({\varvec{\theta }}) = \inf \big \{ \lambda > 0 \;:\; {\varvec{\theta }}\in \lambda {\mathcal C} \big \} . \end{aligned}$$

As usual, $\gamma _{\mathcal C}({\varvec{\theta }}) = + \infty$ if the infimum is not attained.

Lemma 4 hereafter recaps the main properties of a gauge that we need. In particular, (ii) is a fundamental result of convex analysis that states that there is a one-to-one correspondence between gauge functions and closed convex sets containing the origin. This allows to identify sets from their gauges, and vice versa.

Lemma 4

• (i) $\gamma _{\mathcal C}$ is a nonnegative, lsc and sublinear function.

• (ii) ${\mathcal C}$ is the unique closed convex set containing the origin such that

  $$\begin{aligned} {\mathcal C}= \big \{ {\varvec{\theta }}\in \mathbb {R}^p \;:\; \gamma _{\mathcal C}({\varvec{\theta }}) \le 1 \big \} . \end{aligned}$$

• (iii) $\gamma _{\mathcal C}$ is finite-valued if, and only if, $0 \in {{\mathrm{int}}}({\mathcal C})$, in which case $\gamma _{\mathcal C}$ is Lipschitz continuous.

• (iv) $\gamma _{\mathcal C}$ is finite-valued and coercive if, and only if, ${\mathcal C}$ is compact and $0 \in {{\mathrm{int}}}({\mathcal C})$.

See Vaiter et al. (2015a) for the proof.

Observe that thanks to sublinearity, local Lipschitz continuity valid for any finite-valued convex function is strengthened to global Lipschitz continuity. Moreover, $\gamma _{\mathcal C}$ is a norm, having ${\mathcal C}$ as its unit ball, if and only if ${\mathcal C}$ is bounded with non-empty interior and symmetric.

We now define the polar gauge.

Definition 4

(Polar Gauge) The polar of a gauge $\gamma _{\mathcal C}$ is the function $\gamma _{\mathcal C}^\circ$ defined by

$$\begin{aligned} \gamma _{\mathcal C}^\circ ({\varvec{\omega }}) = \inf \big \{ \mu \ge 0 \;:\; \langle {\varvec{\theta }},{\varvec{\omega }}\rangle \le \mu \gamma _{\mathcal C}({\varvec{\theta }}), \forall {\varvec{\theta }} \big \} . \end{aligned}$$

An immediate consequence is that gauges polar to each other have the property

$$\begin{aligned} \langle {\varvec{\theta }},{\varvec{u}}\rangle \le \gamma _{\mathcal C}({\varvec{\theta }}) \gamma _{\mathcal C}^\circ ({\varvec{u}}) \quad \forall ({\varvec{\theta }},{\varvec{u}}) \in {{\mathrm{dom}}}(\gamma _{\mathcal C}) \times {{\mathrm{dom}}}(\gamma _{\mathcal C}^\circ ) , \end{aligned}$$

(42)

just as dual norms satisfy a duality inequality. In fact, polar pairs of gauges correspond to the best inequalities of this type.

Lemma 5

Let ${\mathcal C}\subseteq \mathbb {R}^p$ be a closed convex set containing the origin. Then,

• (ii) $\gamma _{\mathcal C}^\circ$ is a gauge function and $\gamma _{\mathcal C}^{\circ \circ }=\gamma _{\mathcal C}$.

• (iii) $\gamma _{\mathcal C}^\circ =\gamma _{{\mathcal C}^\circ }$, or equivalently

  $$\begin{aligned} {\mathcal C}^\circ = \big \{ {\varvec{\theta }}\in \mathbb {R}^p \;:\; \gamma _{\mathcal C}^\circ ({\varvec{\theta }}) \le 1 \big \} . \end{aligned}$$

• (iv) The gauge of ${\mathcal C}$ and the support function of ${\mathcal C}$ are mutually polar, i.e.,

  $$\begin{aligned} \gamma _{\mathcal C}= \sigma _{{\mathcal C}^\circ } \quad \text {and} \quad \gamma _{{\mathcal C}^\circ } = \sigma _{\mathcal C}~. \end{aligned}$$

See Rockafellar (1996), Hiriart-Urruty and Lemaréchal (2001) and Vaiter et al. (2015a) for the proof.

Expectation of the inner product

We start with some definitions and notations that will be used in the proof. For a non-empty closed convex set ${\mathcal C}\in \mathbb {R}^p$, we denote ${\big ({\mathcal C}\big )}^0$ its minimal selection, i.e., the element of minimal norm in ${\mathcal C}$. This element is of course unique. For a proper lsc and convex function f and $\gamma > 0$, its Moreau envelope (or Moreau–Yosida regularization) is defined by

$$\begin{aligned} {}^{f}\gamma ({\varvec{\theta }})&{\mathop {=}\limits ^{\text { def}}}\min _{\overline{{\varvec{\theta }}}\in \mathbb {R}^p} \frac{1}{2\gamma }\big \Vert \overline{{\varvec{\theta }}}- {\varvec{\theta }}\big \Vert _{2}^2 + f(\overline{{\varvec{\theta }}}). \end{aligned}$$

The Moreau envelope enjoys several important properties that we collect in the following lemma.

Lemma 6

Let f be a finite-valued and convex function. Then

• (i) ${\left( {}^{f}\gamma ({\varvec{\theta }})\right) }_{\gamma > 0}$ is a decreasing net, and $\forall {\varvec{\theta }}\in \mathbb {R}^p$, ${}^{f}\gamma ({\varvec{\theta }}) \nearrow f({\varvec{\theta }})$ as $\gamma \searrow 0$.

• (ii) ${}^{f}\gamma \in C^1(\mathbb {R}^p)$ with $\gamma ^{-1}$-Lipschitz continuous gradient.

• (iii) $\forall {\varvec{\theta }}\in \mathbb {R}^p$, $\nabla {}^{f}\gamma ({\varvec{\theta }}) \rightarrow {\big (\partial f({\varvec{\theta }})\big )}^0$ and $\big \Vert \nabla {}^{f}\gamma ({\varvec{\theta }})\big \Vert _{2} \nearrow \big \Vert {\big (\partial f({\varvec{\theta }})\big )}^0\big \Vert _{2}$ as $\gamma \searrow 0$.

Proof

(i) Bauschke and Combettes (2011, Proposition 12.32). (ii) Bauschke and Combettes (2011, Proposition 12.29). (iii) Since f is finite-valued and convex, it is subdifferentiable everywhere and its subdifferential is a maximal monotone operator with full domain $\mathbb {R}^p$, and the result follows from Bauschke and Combettes (2011, Corollary 23.46(i)). $\square$

We are now equipped to prove the following important result. It will be proved here using Moreau–Yosida regularization. Yet another alternative proof could be based on mollifiers for approximating subdifferentials. Our result hereafter turns out to be instrumental to study EWA in the low-temperature regime for general penalties.

Proposition 5

Let the density $\mu _n$ in (2), where

• (a) F satisfies Assumptions (H.1)–(H.2);

• (b) J is a finite-valued lower-bounded convex function, and $\exists R > 0$ and $\rho \ge 0$, such that $\forall {\varvec{\theta }}\in \mathbb {R}^p$, $\big \Vert {\big (\partial J({\varvec{\theta }})\big )}^0\big \Vert _{2} \le R \left\| {\varvec{\theta }}\right\| _{2}^\rho$;

• (c) and $V_n$ is coercive.

Then, $\forall \overline{{\varvec{\theta }}}\in \mathbb {R}^p$,

$$\begin{aligned} \mathbb {E}_{\mu _n}\left[ \langle {\big (\partial V_n({\varvec{\theta }})\big )}^0,\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \right] = -p\beta . \end{aligned}$$

This result covers of course the situation where J fulfills (H.3). In this case, since $\partial J({\varvec{\theta }}) \subset {\mathcal C}^\circ$ by Theorem 1(i), we have $\rho =0$ and $R={{\mathrm{diam}}}({\mathcal C}^\circ )$, the diameter of the convex compact set ${\mathcal C}^\circ$ containing the origin. It can be shown that, when $F(\cdot ,\varvec{y})$ is strongly coercive, the coercivity assumption (c) can be equivalently stated as $J_{\infty }({\varvec{\theta }}) > 0$, $\forall {\varvec{\theta }}\in \ker (\varvec{X}) \setminus \left\{ 0 \right\}$, where $J_\infty$ is the recession/asymptotic function of J, see e.g., Rockafellar and Wets (1998).

Proof

Let $V^\gamma _n({\varvec{\theta }}) {\mathop {=}\limits ^{\text { def}}}\tfrac{1}{n}F(\varvec{X}{\varvec{\theta }},\varvec{y})+\lambda _n {}^{J}\gamma ({\varvec{\theta }})$ and define $\mu ^{\gamma }_n({\varvec{\theta }}) {\mathop {=}\limits ^{\text { def}}}\exp {\left( -V^\gamma _n({\varvec{\theta }})/\beta \right) }/Z$, where $0< Z < +\infty$ is the normalizing constant of the density $\mu _n$. Assumption (H.1) and Lemma 6(ii)–(iii) tell us that $V^\gamma _n \in C^1(\mathbb {R}^p)$ and $\nabla V^\gamma _n({\varvec{\theta }}) \rightarrow {\big (\partial V_n({\varvec{\theta }})\big )}^0$ as $\gamma \rightarrow 0$. Thus

$$\begin{aligned} \mathbb {E}_{\mu _n}\left[ \langle {\big (\partial V_n({\varvec{\theta }})\big )}^0,\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \right]&= \int _{\mathbb {R}^p} \lim _{\gamma \rightarrow 0} \langle \mu ^\gamma _n({\varvec{\theta }})\nabla V^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \mathrm{d}{\varvec{\theta }}. \end{aligned}$$

We now check that $\langle \mu ^\gamma _n({\varvec{\theta }})\nabla V^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle$ is dominated by an integrable function. From the definition of the Moreau envelope, we have

$$\begin{aligned} V^\gamma _n({\varvec{\theta }}) = \min _{\overline{{\varvec{\theta }}}\in \mathbb {R}^p} \tfrac{1}{n}F(\varvec{X}{\varvec{\theta }},\varvec{y}) + \lambda _n{\big (J({\varvec{\theta }}- \overline{{\varvec{\theta }}}) + \frac{1}{2\gamma }\big \Vert \overline{{\varvec{\theta }}}\big \Vert _{2}^2\big )} . \end{aligned}$$

From coercivity of $V_n$, the objective in the $\min$ is also coercive in $({\varvec{\theta }},\overline{{\varvec{\theta }}})$ by Rockafellar and Wets (1998, Exercise 3.29(b)). It then follows from Rockafellar and Wets (1998, Theorem 3.31) that $V^\gamma _n$ is also coercive. In turn, Rockafellar and Wets (1998, Theorem 11.8(c) and 3.26(a)) allow to assert that for some $a \in ]0,+\infty [$, $\exists b \in ]-\infty ,+\infty [$ such that for all $\gamma > 0$ and ${\varvec{\theta }}\in \mathbb {R}^p$

$$\begin{aligned} \mu ^{\gamma }_n({\varvec{\theta }}) \le \exp {\left( -a\left\| {\varvec{\theta }}\right\| _{2}-b\right) }/Z . \end{aligned}$$

(43)

Lemma 6-(iii) and assumption (b) on J entail that for any ${\varvec{\theta }}\in \mathbb {R}^p$,

$$\begin{aligned} \big \Vert \nabla {}^{J}\gamma ({\varvec{\theta }})\big \Vert _{2} \le \big \Vert {\big (\partial J({\varvec{\theta }})\big )}^0\big \Vert _{2} \le R \left\| {\varvec{\theta }}\right\| _{2}^\rho . \end{aligned}$$

Altogether, we have

$$\begin{aligned}&\big |\langle \mu ^\gamma _n({\varvec{\theta }})\nabla V^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \big |\\&\quad \le \mu ^\gamma _n({\varvec{\theta }}) {\left( \big |\langle \varvec{X}^\top \tfrac{1}{n}\nabla F(\varvec{X}{\varvec{\theta }},\varvec{y}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \big |+\lambda _n\big \Vert \nabla {}^{J}\gamma ({\varvec{\theta }})\big \Vert _{2}\big \Vert \overline{{\varvec{\theta }}}-{\varvec{\theta }}\big \Vert _{2}\right) }\\&\quad \le C Z^{-1}\exp {\left( -F(\varvec{X}{\varvec{\theta }},\varvec{y})/(n\beta )\right) } \big |\langle \tfrac{1}{n}\nabla F(\varvec{X}{\varvec{\theta }},\varvec{y}),\varvec{X}(\overline{{\varvec{\theta }}}-{\varvec{\theta }})\rangle \big | \\&\qquad + (Z\exp {b})^{-1} \lambda _n R \exp {\left( -a\left\| {\varvec{\theta }}\right\| _{2}\right) } \big \Vert {\varvec{\theta }}\big \Vert _{2}^\rho \big \Vert \overline{{\varvec{\theta }}}-{\varvec{\theta }}\big \Vert _{2} , \end{aligned}$$

where the constant $C > 0$ reflects the lower boundedness of J. It is easy to see that the function in this upper bound is integrable, where we also use (H.2). Hence, we can apply the dominated convergence theorem to get

$$\begin{aligned} \mathbb {E}_{\mu _n}\left[ \langle {\big (\partial V_n({\varvec{\theta }})\big )}^0,\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \right]&= \lim _{\gamma \rightarrow 0} \int _{\mathbb {R}^p} \langle \mu ^\gamma _n({\varvec{\theta }})\nabla V^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \mathrm{d}{\varvec{\theta }}. \end{aligned}$$

Now, by simple differential calculus (chain and product rules), we have

$$\begin{aligned} \langle \mu ^\gamma _n({\varvec{\theta }})\nabla V^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle&= -\beta \langle \nabla \mu ^\gamma _n({\varvec{\theta }}),\overline{{\varvec{\theta }}}-{\varvec{\theta }}\rangle \\&= -\beta \sum _{i=1}^p \frac{\partial }{\partial {\varvec{\theta }}_i}{\left( \mu ^\gamma _n({\varvec{\theta }})(\overline{{\varvec{\theta }}}_i-{\varvec{\theta }}_i)\right) } - p\beta \mu ^\gamma _n({\varvec{\theta }}). \end{aligned}$$

Integrating the first term, we get by Fubini theorem and the Newton–Leibniz formula

$$\begin{aligned}&\int _{\mathbb {R}^{p-1}} {\left( \int _{\mathbb {R}} \frac{\partial }{\partial {\varvec{\theta }}_i}{\left( \mu ^\gamma _n({\varvec{\theta }})(\overline{{\varvec{\theta }}}_i-{\varvec{\theta }}_i)\right) } \mathrm{d}{\varvec{\theta }}_i\right) } \mathrm{d}{\varvec{\theta }}_1 \cdots \mathrm{d}{\varvec{\theta }}_{i-1} \mathrm{d}{\varvec{\theta }}_{i+1}\cdots \mathrm{d}{\varvec{\theta }}_p \\&\quad = \int _{\mathbb {R}^{p-1}} {\left[ \mu ^\gamma _n({\varvec{\theta }})(\overline{{\varvec{\theta }}}_i-{\varvec{\theta }}_i)\right] }_{\mathbb {R}} \mathrm{d}{\varvec{\theta }}_1 \cdots \mathrm{d}{\varvec{\theta }}_{i-1} \mathrm{d}{\varvec{\theta }}_{i+1}\cdots \mathrm{d}{\varvec{\theta }}_p = 0 , \end{aligned}$$

where we used coercivity of $V^\gamma _n$ (see (43)) to conclude that $\lim _{|{\varvec{\theta }}_i| \rightarrow +\infty } \mu ^\gamma _n({\varvec{\theta }})(\overline{{\varvec{\theta }}}_i-{\varvec{\theta }}_i) = 0$. For the second term, we have from Lemma 6(i) that $\mu ^\gamma _n \rightarrow \mu _n$ as $\gamma \rightarrow 0$. Thus, arguing again as in (43), we can apply the dominated convergence theorem to conclude that

$$\begin{aligned} \lim _{\gamma \rightarrow 0} \int _{\mathbb {R}^p} \mu ^\gamma _n({\varvec{\theta }}) \mathrm{d}{\varvec{\theta }}= \int _{\mathbb {R}^p} \mu _n({\varvec{\theta }}) \mathrm{d}{\varvec{\theta }}= 1. \end{aligned}$$

This concludes the proof. $\square$