A fast unified algorithm for solving group-lasso penalize learning problems

Abstract

This paper concerns a class of group-lasso learning problems where the objective function is the sum of an empirical loss and the group-lasso penalty. For a class of loss function satisfying a quadratic majorization condition, we derive a unified algorithm called groupwise-majorization-descent (GMD) for efficiently computing the solution paths of the corresponding group-lasso penalized learning problem. GMD allows for general design matrices, without requiring the predictors to be group-wise orthonormal. As illustration examples, we develop concrete algorithms for solving the group-lasso penalized least squares and several group-lasso penalized large margin classifiers. These group-lasso models have been implemented in an R package gglasso publicly available from the Comprehensive R Archive Network (CRAN) at http://cran.r-project.org/web/packages/gglasso. On simulated and real data, gglasso consistently outperforms the existing software for computing the group-lasso that implements either the classical groupwise descent algorithm or Nesterov’s method.

Appendix: Proofs

Proof of Lemma 1

Part (1). For any $\varvec{\beta }$ and $\varvec{\beta }^*$, write $\varvec{\beta }-\varvec{\beta }^*=V$ and define $g(t)=L(\varvec{\beta }^*+tV \mid \mathbf{D})$ so that

$$\begin{aligned} g(0)=L(\varvec{\beta }^* \mid \mathbf{D}), \quad g(1)=L(\varvec{\beta }\mid \mathbf{D}). \end{aligned}$$

By the mean value theorem, $\exists \ a \in (0,1)$ such that

$$\begin{aligned} g(1)=g(0)+g^{\prime }(a)=g(0)+g^{\prime }(0)+[g^{\prime }(a)-g^{\prime }(0)]. \end{aligned}$$

(25)

Write $\Phi _f^{\prime }=\frac{\partial {\Phi (y,f)}}{\partial {f}}$. Note that

$$\begin{aligned} g^{\prime }(t)=\frac{1}{n}\sum ^n_{i=1}\tau _i\Phi _f^{\prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i (\varvec{\beta }^*+tV))(\mathbf{x}^{{ \mathsf T}}_i V). \end{aligned}$$

(26)

Thus $g^{\prime }(0)=(\varvec{\beta }-\varvec{\beta }^*)^{{ \mathsf T}}\nabla L(\varvec{\beta }^* | \mathbf{D}).$ Moreover, from (26) we have

$$\begin{aligned} |g^{\prime }(a)-g^{\prime }(0)|&= \vert \frac{1}{n}\sum ^n_{i=1}\tau _i[\Phi _f^{\prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i (\varvec{\beta }^*+aV))\nonumber \\&-\Phi _f^{\prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i \varvec{\beta }^*)](\mathbf{x}^{{ \mathsf T}}_i V) \vert \nonumber \\&\le \frac{1}{n}\sum ^n_{i=1} \tau _i|\Phi _f^{\prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i (\varvec{\beta }^*+aV))\nonumber \\&-\Phi _f^{\prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i \varvec{\beta }^*) | |\mathbf{x}^{{ \mathsf T}}_i V| \nonumber \\&\le \frac{1}{n}\sum ^n_{i=1} C\tau _i|\mathbf{x}^{{ \mathsf T}}_i aV | |\mathbf{x}^{{ \mathsf T}}_i V| \end{aligned}$$

(27)

$$\begin{aligned}&\le \frac{1}{n}\sum ^n_{i=1} C \tau _i \Vert \mathbf{x}^{{ \mathsf T}}_i V \Vert ^2_2 \nonumber \\&= \frac{C}{n}V^{{ \mathsf T}} [ \mathbf{X}^{{ \mathsf T}} \varvec{\Gamma }\mathbf{X}] V, \end{aligned}$$

(28)

where in (27) we have used the inequality $| \Phi ^{\prime }(y,f_1)-\Phi ^{\prime }(y,f_2)| \le C |f_1-f_2|$. Plugging (27) into (25) we have

$$\begin{aligned} L(\varvec{\beta }\mid \mathbf{D})&\le L(\varvec{\beta }^* \mid \mathbf{D})+(\varvec{\beta }-\varvec{\beta }^*)^{{ \mathsf T}}\nabla L(\varvec{\beta }^* | \mathbf{D})\\&+\frac{1}{2}(\varvec{\beta }-\varvec{\beta }^*)^{{ \mathsf T}} \mathbf{H}( \varvec{\beta }- \varvec{\beta }^*), \end{aligned}$$

with $\mathbf{H}=\frac{2C}{n}\mathbf{X}^{{ \mathsf T}} \varvec{\Gamma }\mathbf{X}.$

Part (2). Write $\Phi _f^{\prime \prime }=\frac{\partial {\Phi ^2(y,f)}}{\partial {f^2}}$. By Taylor’s expansion, $\exists \ b \in (0,1)$ such that

$$\begin{aligned} g(1)=g(0)+g^{\prime }(0)+g^{\prime \prime }(b). \end{aligned}$$

(29)

Note that

$$\begin{aligned} g^{\prime \prime }(b)&= \frac{1}{n}\sum ^n_{i=1}\tau _i \Phi _f^{\prime \prime }(y_i,\mathbf{x}^{{ \mathsf T}}_i (\varvec{\beta }^*+bV))(\mathbf{x}^{{ \mathsf T}}_i V)^2\nonumber \\&\le \frac{1}{n}\sum ^n_{i=1}C_2\tau _i (\mathbf{x}^{{ \mathsf T}}_i V)^2, \end{aligned}$$

(30)

where we have used the inequality $\Phi _f^{\prime \prime } \le C_2$. Plugging (30) into (29) we have

$$\begin{aligned} L(\varvec{\beta }\mid \mathbf{D})&\le L(\varvec{\beta }^* \mid \mathbf{D})+(\varvec{\beta }-\varvec{\beta }^*)^{{ \mathsf T}}\nabla L(\varvec{\beta }^* | \mathbf{D})\\&+\frac{1}{2}(\varvec{\beta }-\varvec{\beta }^*)^{{ \mathsf T}} \mathbf{H}( \varvec{\beta }- \varvec{\beta }^*), \end{aligned}$$

with $\mathbf{H}=\frac{C_2}{n}\mathbf{X}^{{ \mathsf T}} \varvec{\Gamma }\mathbf{X}.$ $\square$