Newton-Type Methods with the Proximal Gradient Step for Sparse Estimation

Abstract

In this paper, we propose new methods to efficiently solve convex optimization problems encountered in sparse estimation. These methods include a new quasi-Newton method that avoids computing the Hessian matrix and improves efficiency, and we prove its fast convergence. We also prove the local convergence of the Newton method under the assumption of strong convexity. Our proposed methods offer a more efficient and effective approach, particularly for $L_1$ regularization and group regularization problems, as they incorporate variable selection with each update. Through numerical experiments, we demonstrate the efficiency of our methods in solving problems encountered in sparse estimation. Our contributions include theoretical guarantees and practical applications for various kinds of problems.

Appendices

Appendix A. Proof of Proposition 3

Lemma 1

Suppose $A,B \in {\mathbb R}^{n \times n}$ are symmetric and A is positive semidefinite. Then, any eigenvalue $\lambda$ of AB satisfies $\min \{\Vert A \Vert \lambda _{\min } (B),0\} \le \lambda \le \max \{ \Vert A\Vert \lambda _{\max } (B),0 \}$, where $\lambda _{\min } (B)$ and $\lambda _{\max } (B)$ are the minimum and maximum eigenvalues, respectively, of B.

Proof

Since the eigenvalues of AB are equivalent to the eigenvalues of BA, we consider the eigenvalues of BA. We let $\lambda \in {\mathbb R},x \in {\mathbb R}^n$ such that $BAx = \lambda x$ and $x\ne 0$. By multiplying $x^TA$ from the left, we obtain

$$\begin{aligned} x^TABA x= \lambda x^T A x. \end{aligned}$$

If $x^T A x = 0$, then $\lambda = 0$ since $Ax = 0$. Next, we consider the $x^TAx > 0$ case. Since A is a symmetric positive semidefinite matrix, $A^{\frac{1}{2}}$, and we obtain

$$\begin{aligned} \begin{aligned} \frac{x^TA B A x}{x^T A x}&= \lambda \\ \frac{x^TA^{\frac{1}{2}} A^{\frac{1}{2}}B A^{\frac{1}{2}} A^{\frac{1}{2}}x}{x^T A^{\frac{1}{2}} A^{\frac{1}{2}} x}&= \lambda . \end{aligned} \end{aligned}$$

(41)

We can rewrite (41) as

$$\begin{aligned} \frac{y^TA^{\frac{1}{2}} B A^{\frac{1}{2}} y}{y^T y} = \lambda , \end{aligned}$$

where $y = A^{\frac{1}{2}} x\ne 0$. Thus, since $0<\Vert A^{\frac{1}{2}} y \Vert _2 \le \Vert A\Vert ^{\frac{1}{2}} \Vert y\Vert _2$, we can obtain

$$\begin{aligned} \min \{\Vert A\Vert \lambda _{\min }(B), 0\} \le \frac{y^TA^{\frac{1}{2}} B A^{\frac{1}{2}} y}{y^T y} \le \max \{\Vert A\Vert \lambda _{\max }(B), 0\}. \end{aligned}$$

Therefore, if $x^T A x>0$, then $\min \{\Vert A\Vert \lambda _{\min }(B), 0\} \le \lambda \le \Vert A\Vert \max \{\Vert A\Vert \lambda _{\max }(B), 0\}$. Also using the result when $x^T A x = 0$, Lemma 4 holds.$\square$

Theorem 1

([6], Theorem 3.2). Suppose $g:{\mathbb R}^n \rightarrow \mathbb {R} \cup \{+ \infty \}$ is a closed convex function. Every $V\in \partial _B \textrm{prox}_{\nu g}(x)$ is a symmetric positive semidefinite matrix that satisfies $\Vert V\Vert \le 1$ for all $x\in {\mathbb R}^n$.

Proof of Proposition 3

By assumption, since $\lambda _{\min }(\nabla ^2 f(x)) \ge \mu$ for any $x\in {\mathbb R}^n$,

$$\begin{aligned} \lambda _{\max } (I- \nu \nabla ^2 f(x)) \le 1-\nu \mu . \end{aligned}$$

Since every $V\in \partial _B \textrm{prox}_{\nu g}(x)$ is a symmetric positive semidefinite matrix that satisfies $\Vert V\Vert \le 1$ for all $x\in {\mathbb R}^n$ by Theorem 4, from Lemma 4,

$$\begin{aligned} \lambda _{\max }\left( V \left( I-\nu \nabla ^2 f(x) \right) \right) \le \max \{1- \nu \mu , 0\}. \end{aligned}$$

Thus, every eigenvalue of $I - V \left( I-\nu \nabla ^2 f(x) \right)$ is a real number that is greater than or equal to $\min \{\nu \mu , 1\}$, and $I - V \left( I-\nu \nabla ^2 f(x) \right)$ is a nonsingular matrix.$\square$

Appendix B. Proof of Theorem 2

Lemma 2

([24], Lemma 2.3.2). Let $A, C \in {\mathbb R}^{n \times n}$, and we assume that A is nonsingular, with $\Vert A^{-1} \Vert \le \alpha$. If $\Vert A - C \Vert \le \beta$ and $\beta \alpha < 1$, then C is also nonsingular, and

$$\begin{aligned} \Vert C^{-1}\Vert \le \frac{\alpha }{(1-\alpha \beta )} \end{aligned}$$

Proof of Theorem 2

Let $V^{(k)} \in \partial _B \textrm{prox}_{\nu g}(x^{(k)} - \nu \nabla f(x^{(k)}))$, $U^{(k)}:= I - V^{(k)}\left( I-\nu \nabla ^2 f(x^{(k)}) \right) \in \partial F_\nu (x^{(k)})$ and $W^{(k)}:= I - V^{(k)}\left( I-\nu B^{(k)} \right) \in \hat{\partial }^{(k)}F_\nu (x^{(k)})$. From Proposition 3, every eigenvalue of $U^{(k)}$ is a real number that is greater than or equal to $\xi := \min \{\nu \mu , 1\}$, and

$$\begin{aligned} \left\| \left( U^{(k)}\right) ^{-1} \right\| \le \frac{\sqrt{n}}{\xi }. \end{aligned}$$

Let $\Delta = \frac{\xi }{5 \nu \sqrt{n}}$. Since $\partial F_\nu$ is the LNA of $F_\nu$ at $x^*$, there exists $\epsilon > 0$ such that

$$\begin{aligned} \Vert F_\nu (x) - F_\nu (x^*) - U(x-x^*)\Vert _2 \le \nu \Delta \Vert x-x^*\Vert _2 \end{aligned}$$

for any $x\in B(x^*, \epsilon ):= \{y\mid \Vert x^*-y\Vert _2 < \epsilon \},U\in \partial F_\nu (x)$. Since $W^{(k)} - U^{(k)} = \nu V^{(k)}\left( B^{(k)} - \nabla ^2 f(x^{(k)}) \right)$ and $\Vert B^{(k)} - \nabla ^2 f(x^{(k)})\Vert < \Delta$, we obtain $\Vert W^{(k)} - U^{(k)}\Vert \le \nu \Delta$. By Lemma 5, $W^{(k)}$ is nonsingular and

$$\begin{aligned} \left\| \left( W^{(k)} \right) ^{-1} \right\|&\le \frac{\sqrt{n}/\xi }{1-\sqrt{n}/\xi \times \nu \Delta }\\&= \frac{5}{4} \frac{\sqrt{n}}{\xi }. \end{aligned}$$

Thus, if $\Vert x^{(k)} - x^* \Vert _2< \epsilon$, then we have

$$\begin{aligned} \Vert x^{(k+1)}-x^*\Vert _2&= \Vert x^{(k)}-(W^{(k)})^{-1}F_\nu (x^{(k)})-x^*\Vert _2 \\&\le \Vert (W^{(k)})^{-1}\Vert \Vert F_\nu (x^{(k)})-F_\nu (x^*)-W^{(k)}(x^{(k)} - x^*)\Vert _2 \\&\le \Vert (W^{(k)})^{-1}\Vert \left[ \Vert F_\nu (x^{(k)})-F_\nu (x^*)-U^{(k)}(x^{(k)} - x^*)\Vert _2 + \Vert W^{(k)}-U^{(k)}\Vert \Vert x^{(k)}-x^*\Vert _2\right] \\&\le \frac{5}{4}\frac{\sqrt{n}}{\xi } (2\nu \Delta \Vert x^{(k)} - x^*\Vert _2)\\&= \frac{1}{2}\Vert x^{(k)} - x^*\Vert _2 \end{aligned}$$

Therefore, there exists $\epsilon , \Delta$ such that the sequence generated by Algorithm 2 locally linearly converges to $x^*$.$\square$

Appendix C. Proof of Theorem 3

Proof

Let $V^{(k)} \in \partial _B \textrm{prox}_{\nu g}(x^{(k)} - \nu \nabla f(x^{(k)}))$, $U^{(k)}:= I - V^{(k)}\left( I-\nu \nabla ^2 f(x^{(k)}) \right) \in \partial F_\nu (x^{(k)})$ and $W^{(k)}:= I - V^{(k)}\left( I-\nu B^{(k)} \right) \in \hat{\partial }^{(k)}F_\nu (x^{(k)})$. We let $e^{(k)} = x^{(k)}-x^*,s^{(k)}=x^{(k+1)}-x^{(k)}$. We note that $s^{(k)} = e^{(k+1)} - e^{(k)}$ and $\{e^{(k)}\}$ and $\{s^{(k)}\}$ converge to 0 since $\{x^{(k)}\}$ converges to $x^*$. From the update rule of Algorithm 2, we have

$$\begin{aligned} F_\nu (x^*)&= \left[ F_\nu (x^{(k)}) + W^{(k)}s^{(k)}\right] + \left[ \left( U^{(k)} - W^{(k)}\right) s^{(k)}\right] - \left[ F_\nu (x^{(k)}) - F_\nu (x^*) - U^{(k)} e^{(k)}\right] - U^{(k)}e^{(k+1)} \\&=\left[ \left( U^{(k)} - W^{(k)}\right) s^{(k)}\right] - \left[ F_\nu (x^{(k)}) - F_\nu (x^*) - U^{(k)} e^{(k)}\right] - U^{(k)}e^{(k+1)}. \end{aligned}$$

Since $F_\nu (x^*) = 0$ and $U^{(k)}$ is a nonsingular matrix,

$$\begin{aligned} U^{(k)}e^{(k+1)}&=\left[ \left( U^{(k)} - W^{(k)}\right) s^{(k)}\right] - \left[ F_\nu (x^{(k)}) - F_\nu (x^*) - U^{(k)} e^{(k)}\right] \\ e^{(k+1)}&= \left( U^{(k)} \right) ^{-1}\left[ \left( U^{(k)} - W^{(k)}\right) s^{(k)}\right] - \left( U^{(k)} \right) ^{-1} \left[ F_\nu (x^{(k)}) - F_\nu (x^*) - U^{(k)} e^{(k)}\right] . \end{aligned}$$

By assumption, since $\Vert (W^{(k)} - U^{(k)})s^{(k)}\Vert _2 = \Vert \nu V^{(k)} (\nabla ^2 f(x^{(k)}) - B^{(k)}) s^{(k)}\Vert _2$ and $\Vert \nabla ^2 f(x^{*}) - \nabla ^2 f(x^{(k)}) \Vert \rightarrow 0$ for $k \rightarrow \infty$, we have $\Vert (W^{(k)} - U^{(k)})s^{(k)}\Vert _2 = o( \Vert s^{(k)} \Vert _2 )$. Therefore,

$$\begin{aligned} \Vert e^{(k+1)}\Vert _2 = o(\Vert s^{(k)}\Vert _2) + o(\Vert e^{(k)}\Vert _2) = o(\Vert e^{(k+1)}\Vert _2) + o(\Vert e^{(k)} \Vert _2). \end{aligned}$$

Thus, we obtain $\Vert e^{(k+1)}\Vert _2 = o(\Vert e^{(k)}\Vert _2)$, and since $e^{(k)} = x^{(k)} - x^*$, the sequence $\{x^{(k)}\}$ generated by Algorithm 2 superlinearly converges to $x^*$.$\square$