Relating $\ell _p$ regularization and reweighted $\ell _1$ regularization

Abstract

We propose a general framework of iteratively reweighted $\ell _1$ algorithms for solving $\ell _p$ regularization problems. We show that all the limit points of the iterates generated by the proposed algorithms have the same sign. Moreover, for sufficiently large iterations, the iterates also have the same sign as the limit points, and the nonzero components are bounded away from zero. Therefore, the algorithm behaves like solving a smooth problem in the reduced space consisting of the nonzero components. We analyze the global convergence and the worst-case complexity for the reweighted algorithms. Besides, a smoothing parameter updating strategy is proposed which can automatically stop reducing the smoothing parameters corresponding to the zero components of the limit points. We show that the $\ell _p$ regularized regression problem is locally equivalent to a weighted $\ell _1$ regularization problem near a stationary point and every stationary point corresponds to a Maximum A Posterior estimation for independently and non-identically distributed Laplace prior parameters. Numerical experiments exhibit the behaviors and the efficiency of our proposed algorithms.

Appendix

We discuss two common examples to see that the iterates $\{x^k\}$ generated by Algorithm 1 generally has a unique limit point. By Theorem 6, we only have to verify whether for a stationary point $x^*$, $\nabla ^2 F( [x_{\mathcal{I}^*}^*; 0_{\mathcal{A}^*}])$ is invertible.

$\ell _p$ regularized linear regression The loss function is $f(x) = \tfrac{1}{2}\Vert Ax-b\Vert ^2_2$ with $A\in \mathbb {R}^{m\times n}$, $x\in \mathbb {R}^n$ and $b\in \mathbb {R}^m$. Therefore, $\nabla ^2 F( [x_{\mathcal{I}^*}^*; 0_{\mathcal{A}^*}]) = [A^TA]_{\mathcal{I}^*\mathcal{I}^*} + \lambda \nabla ^2 \Vert x_{\mathcal{I}^*}^*\Vert ^p_p$. In this case, the algorithm converges uniquely to the stationary point as long as $[A^TA]_{\mathcal{I}^*\mathcal{I}^*} + \lambda \nabla ^2 \Vert x_{\mathcal{I}^*}^*\Vert ^p_p$ is nonsingular.

$\ell _p$ regularized logistic regression The loss function is

$$\begin{aligned} f(x) = \sum \limits _{i=1}^m[y_i\ln \sigma (a_i^Tx) + (1-y_i)\ln (1-\sigma (a_i^Tx))],\end{aligned}$$

with $a_i\in \mathbb {R}^n, y_i\in \{0,1\}$ and $\sigma (s) = \frac{1}{1+e^{-s}}$. In this case, $[\nabla ^2 f(x^*)]_{\mathcal{I}^*\mathcal{I}^*} = [A^TD(x^*)A]_{\mathcal{I}^*\mathcal{I}^*}$, where $D(x^*) = \text {diag}(\sigma (a_i^Tx^*)(1-\sigma (a_i^Tx^*)), i\in \{1,\ldots ,m\})$ and $A = [a_1, \ldots , a_m]$. In this case, the algorithm converges uniquely to the stationary point as long as $[A^TD(x^*)A]_{\mathcal{I}^*\mathcal{I}^*} + \lambda \nabla ^2 \Vert x_{\mathcal{I}^*}^*\Vert ^p_p$ is nonsingular.