Conjugate gradient acceleration of iteratively re-weighted least squares methods

Abstract

Iteratively re-weighted least squares (IRLS) is a method for solving minimization problems involving non-quadratic cost functions, perhaps non-convex and non-smooth, which however can be described as the infimum over a family of quadratic functions. This transformation suggests an algorithmic scheme that solves a sequence of quadratic problems to be tackled efficiently by tools of numerical linear algebra. Its general scope and its usually simple implementation, transforming the initial non-convex and non-smooth minimization problem into a more familiar and easily solvable quadratic optimization problem, make it a versatile algorithm. However, despite its simplicity, versatility, and elegant analysis, the complexity of IRLS strongly depends on the way the solution of the successive quadratic optimizations is addressed. For the important special case of compressed sensing and sparse recovery problems in signal processing, we investigate theoretically and numerically how accurately one needs to solve the quadratic problems by means of the conjugate gradient (CG) method in each iteration in order to guarantee convergence. The use of the CG method may significantly speed-up the numerical solution of the quadratic subproblems, in particular, when fast matrix-vector multiplication (exploiting for instance the FFT) is available for the matrix involved. In addition, we study convergence rates. Our modified IRLS method outperforms state of the art first order methods such as Iterative Hard Thresholding (IHT) or Fast Iterative Soft-Thresholding Algorithm (FISTA) in many situations, especially in large dimensions. Moreover, IRLS is often able to recover sparse vectors from fewer measurements than required for IHT and FISTA.

Appendix: Proof of Lemma 10

“$\Rightarrow$” (in the case $0 < \tau \leqslant 1$)

Let $x = x^{{\varepsilon },1}$ or $x\in \mathcal {X}_{{\varepsilon },\tau }(y)$, and $\eta \in \mathcal {N}_{\varPhi }$ arbitrary. Consider the function

$$\begin{aligned} G_{{\varepsilon },\tau }(t) \mathrel {\mathop :}=f_{{\varepsilon },\tau }\left( x + t\eta \right) - f_{{\varepsilon },\tau }\left( x \right) \end{aligned}$$

with its first derivative

$$\begin{aligned} G^{\prime }_{{\varepsilon },\tau }(t) = \tau \sum \limits _{i=1}^{N}\frac{x_{i}\eta _{i} +t\eta _{i}^2}{\left[ |x_{i} + t\eta _{i}|^{2} + {\varepsilon }^{2}\right] ^{\frac{2-\tau }{2}}}. \end{aligned}$$

Now $G_{{\varepsilon },\tau }(0) = 0$ and from the minimization property of $f_{{\varepsilon },\tau }(x)$, $G_{{\varepsilon },\tau }(t) \ge 0$. Therefore,

$$\begin{aligned} 0 = G^{\prime }_{{\varepsilon },\tau }(0) = \sum \limits _{i=1}^{N}\frac{x_{i}\eta _{i}}{\left[ x_{i}^{2} + {\varepsilon }^{2}\right] ^{\frac{2-\tau }{2}}} = \left\langle x,\eta \right\rangle _{\hat{w}(x,{\varepsilon },\tau )}. \end{aligned}$$

“$\Leftarrow$” (only in the case $\tau =1$)

Now let $x\in \mathcal {F}_{\varPhi }(y)$ and $\left\langle x,\eta \right\rangle _{\hat{w}(x,{\varepsilon },1)} = 0$ for all $\eta \in \mathcal {N}_{\varPhi }$. We want to show that x is the minimizer of $f_{{\varepsilon },1}$ in $\mathcal {F}_{\varPhi }(y)$. Consider the convex univariate function $g(u)\mathrel {\mathop :}=[u^{2} + {\varepsilon }^{2}]^{1/2}$. For any point $u_{0}$ we have from convexity that

$$\begin{aligned} {[}u^{2} + {\varepsilon }^{2}{]}^{1/2} \geqslant [u_{0}^{2} + {\varepsilon }^{2}]^{1/2} + {[}u_{0}^{2} + {\varepsilon }^{2}{]}^{-1/2}u_{0}(u-u_{0}) \end{aligned}$$

because the right-hand-side is the linear function which is tangent to g at $u_{0}$. It follows, that for every point $v\in \mathcal {F}_{\varPhi }(y)$ we have

$$\begin{aligned} f_{{\varepsilon },1}(v)\geqslant & {} f_{{\varepsilon },1}(x) + \sum \limits _{i=1}^{N}{[x_{i}^{2} + {\varepsilon }^{2}]^{-1/2}x_{i}(v_{i} - x_{i})}\\= & {} f_{{\varepsilon },1}(x) + \left\langle x, v-x\right\rangle _{\hat{w}(x,{\varepsilon },1)} = f_{{\varepsilon },1}(x), \end{aligned}$$

where we have used the orthogonality condition and the fact that $(v - x) \in \mathcal {N}_{\varPhi }$. Since v was chosen arbitrarily, $x = x^{{\varepsilon },1}$ as claimed.