The degrees of freedom of partly smooth regularizers

Abstract

We study regularized regression problems where the regularizer is a proper, lower-semicontinuous, convex and partly smooth function relative to a Riemannian submanifold. This encompasses several popular examples including the Lasso, the group Lasso, the max and nuclear norms, as well as their composition with linear operators (e.g., total variation or fused Lasso). Our main sensitivity analysis result shows that the predictor moves locally stably along the same active submanifold as the observations undergo small perturbations. This plays a pivotal role in getting a closed-form expression for the divergence of the predictor w.r.t. observations. We also show that, for many regularizers, including polyhedral ones or the analysis group Lasso, this divergence formula holds Lebesgue a.e. When the perturbation is random (with an appropriate continuous distribution), this allows us to derive an unbiased estimator of the degrees of freedom and the prediction risk. Our results unify and go beyond those already known in the literature.

Basic properties of o-minimal structures

In the following results, we collect some important stability properties of o-minimal structures. To be self-contained, we also provide proofs. To the best of our knowledge, these proofs, although simple, are not reported in the literature or some of them are left as exercises in the authoritative references (van den Dries 1998; Coste 1999). Moreover, in most proofs, to show that a subset is definable, we could just write the appropriate first-order formula (see Coste 1999, Page 12; van den Dries 1998, Section Ch1.1.2), and conclude using (Coste 1999, Theorem 1.13). Here, for the sake of clarity and to avoid cryptic statements for the non-specialist, we will translate the first-order formula into operations on the involved subsets, in particular projections, and invoke the above stability axioms of o-minimal structures. In the following, n denotes an arbitrary (finite) dimension which is not necessarily the number of observations used previously the paper.

Lemma 5

(Addition and multiplication) Let $f : \Omega \subset \mathbb {R}^n \rightarrow \mathbb {R}^p$ and $g : \Omega \subset \mathbb {R}^n \subset \mathbb {R}^p$ be definable functions. Then their pointwise addition and multiplication is also definable.

Proof

Let $h=f+g$, and

$$\begin{aligned} B=(\Omega \times \mathbb {R}\times \Omega \times \mathbb {R}\times \Omega \times \mathbb {R}) \cap (\Omega \times \mathbb {R}\times {{\mathrm{gph}}}(f) \times {{\mathrm{gph}}}(h)) \cap S, \end{aligned}$$

where $S= \{ (x,u,y,v,z,w) \;:\; x=y=z, u=v+w \}$ is obviously an algebraic (in fact linear) subset, hence definable by axiom 2. Axiom 1 and 2 then imply that B is also definable. Let $\varPi _{3n+3p,n+p}: \mathbb {R}^{3n+3p} \rightarrow \mathbb {R}^{n+p}$ be the projection on the first $n+p$ coordinates. We then have

$$\begin{aligned} {{\mathrm{gph}}}(h) = \varPi _{3n+3p,n+p}(B) \end{aligned}$$

whence we deduce that h is definable by applying $3n+3p$ times axiom 4. Definability of the pointwise multiplication follows the same proof taking $u=v \cdot w$ in S. $\square$

Lemma 6

(Inequalities in definable sets) Let $f : \Omega \subset \mathbb {R}^n \rightarrow \mathbb {R}$ be a definable function. Then $\{ x \in \Omega \;:\; f(x) > 0 \}$, is definable. The same holds when replacing > with <.

Clearly, inequalities involving definable functions are accepted when defining definable sets.

There are many possible proofs of this statement.

Proof

(1) Let $B= \{ (x,y) \in \mathbb {R}\times \mathbb {R} \;:\; f(x)=y \} \cap (\Omega \times (0,+\infty )$, which is definable thanks to axioms 1 and 3, and that the level sets of a definable function are also definable. Thus

$$\begin{aligned} \{ x \in \Omega \;:\; f(x)> 0 \} = \{ x \in \Omega \;:\; \exists y, f(x) = y, y > 0 \} = \varPi _{n+1,n}(B) ~, \end{aligned}$$

and we conclude using again axiom 4. $\square$

Yet another (simpler) proof.

Proof

(2) It is sufficient to remark that $\{ x \in \Omega \;:\; f(x) > 0 \}$ is the projection of the set $\{ (x,t) \in \Omega \times \mathbb {R} \;:\; t^2f(x)-1 = 0 \}$, where the latter is definable owing to Lemma 5. $\square$

Lemma 7

(Derivative) Let $f: I \rightarrow \mathbb {R}$ be a definable differentiable function on an open interval I of $\mathbb {R}$. Then its derivative $f': I \rightarrow \mathbb {R}$ is also definable.

Proof

Let $g: (x,t) \in I \times \mathbb {R}\mapsto g(x,t) = f(x+t)-f(x)$. Note that g is definable function on $I \times \mathbb {R}$ by Lemma 5. We now write the graph of $f'$ as

$$\begin{aligned} {{\mathrm{gph}}}(f') \!=\! \{ (x,y) \!\in \!s I \times \mathbb {R} \;:\; \forall \varepsilon> 0, \exists \delta > 0, \forall t \in \mathbb {R}, | t | \!<\! \delta , | g(x,t) - yt | < \varepsilon |t| \} . \end{aligned}$$

Let $C= \{ (x,y,v,t,\varepsilon ,\delta ) \in I \times \mathbb {R}^5 \;:\; ((x,t),v) \in {{\mathrm{gph}}}(g) \}$, which is definable since g is definable and using axiom 3. Let

$$\begin{aligned} B = \{ (x,y,v,t,\varepsilon ,\delta ) \;:\; t^2< \delta ^2, (v-ty)^2 < \varepsilon ^2t^2 \} \cap C. \end{aligned}$$

The first part in B is semi-algebraic, hence definable thanks to axiom 2. Thus B is also definable using axiom 1. We can now write

$$\begin{aligned} {{\mathrm{gph}}}(f') = \mathbb {R}^3 {\setminus } \left( \varPi _{5,3}\left( \mathbb {R}^5 {\setminus } \varPi _{6,5}(B)\right) \right) \cap (I \times \mathbb {R}), \end{aligned}$$

where the projectors and completions translate the actions of the existential and universal quantifiers. Using again axioms 4 and 1, we conclude. $\square$

With such a result at hand, this proposition follows immediately.

Proposition 2

(Differential and Jacobian) Let $f=(f_1,\ldots ,f_p): \Omega \rightarrow \mathbb {R}^p$ be a differentiable function on an open subset $\Omega$ of $\mathbb {R}^n$. If f is definable, then so its differential mapping and its Jacobian. In particular, for each $i=1,\ldots ,n$ and $j=1,\ldots ,p$, the partial derivative $\partial f_i/\partial x_j: \Omega \rightarrow \mathbb {R}$ is definable.

We provide below some results concerning the subdifferential.

Proposition 3

(Subdifferential) Suppose that f is a finite-valued convex definable function. Then for any $x \in \mathbb {R}^n$, the subdifferential $\partial f(x)$ is definable.

Proof

For every $x \in \mathbb {R}^n$, the subdifferential $\partial f(x)$ reads

$$\begin{aligned} \partial f(x) = \{ \eta \in \mathbb {R}^n \;:\; f(x') \ge f(x) + \langle \eta ,\,x'-x\rangle \quad \forall x' \in \mathbb {R}^n \} . \end{aligned}$$

Let $K = \{ (\eta ,x') \in \mathbb {R}^n \times \mathbb {R}^n \;:\; f(x') < f(x) + \langle \eta ,\,x'-x\rangle \}$. Hence, $\partial f(x) = \mathbb {R}^n {\setminus } \varPi _{2n,n}(K)$. Since f is definable, the set K is also definable using Lemmas 5 and 6, whence definability of $\partial f(x)$ follows using axiom 4. $\square$

Lemma 8

Suppose that f is a finite-valued convex definable function. Then, the set

$$\begin{aligned} \{ (x,\eta ) \;:\; \eta \in {{\mathrm{ri}}}\partial f(x) \} \end{aligned}$$

is definable.

Proof

Denote $C = \{ (x,\eta ) \;:\; \eta \in {{\mathrm{ri}}}\partial f(x) \}$. Using the characterization of the relative interior of a convex set (Rockafellar 1996, Theorem 6.4), we rewrite C in the more convenient form

$$\begin{aligned} C = \{(x,\eta ) \, : \,&\forall u \in \mathbb {R}^n, \forall z \in \mathbb {R}^n, f(z) - f(x) \ge \langle u,\,z-x\rangle ,\\&\exists t > 1, \forall x' \in \mathbb {R}^n, f(x') - f(x) \ge \langle (1-t) u + t \eta ,\,x'-x\rangle \} . \end{aligned}$$

Let $D = \mathbb {R}^n \times \mathbb {R}^n \times \mathbb {R}^n \times \mathbb {R}^n \times (1,+\infty ) \times \mathbb {R}^n$ and K defined as

$$\begin{aligned} K= & {} \{ (x,\eta ,u,z,t,x') \!\in \! D \;:\; f(z) - f(x) \geqslant \langle u,\,z-x\rangle ), f(x') - f(x)\nonumber \\&\geqslant \langle (1-t) u + t \eta ,\,x'-x\rangle \} . \end{aligned}$$

Thus,

$$\begin{aligned} C = \mathbb {R}^{2n} {\setminus } \varPi _{3n,2n} \left( \mathbb {R}^{3n} {\setminus } \varPi _{4n,3n} \left( \varPi _{4n+1,4n} \left( \mathbb {R}^{4n} \times (1,+\infty ) {\setminus } \varPi _{5n+1,4n+1} (K) \right) \right) \right) , \end{aligned}$$

where the projectors and completions translate the actions of the existential and universal quantifiers. Using again axioms 4 and 1, we conclude. $\square$