A stochastic extra-step quasi-Newton method for nonsmooth nonconvex optimization

Abstract

In this paper, a novel stochastic extra-step quasi-Newton method is developed to solve a class of nonsmooth nonconvex composite optimization problems. We assume that the gradient of the smooth part of the objective function can only be approximated by stochastic oracles. The proposed method combines general stochastic higher order steps derived from an underlying proximal type fixed-point equation with additional stochastic proximal gradient steps to guarantee convergence. Based on suitable bounds on the step sizes, we establish global convergence to stationary points in expectation and an extension of the approach using variance reduction techniques is discussed. Motivated by large-scale and big data applications, we investigate a stochastic coordinate-type quasi-Newton scheme that allows to generate cheap and tractable stochastic higher order directions. Finally, numerical results on large-scale logistic regression and deep learning problems show that our proposed algorithm compares favorably with other state-of-the-art methods.

9 Appendix: proofs of auxiliary results

1.1 9.1 Proof of Lemma 3.2

Proof

As in [25, Lemma 3.2], applying the definition and characterization (2.3) of the proximal operator, it follows

$$\begin{aligned} {\bar{x}} = {\mathrm {prox}}^{}_{\theta \psi }(x) \,&\iff \, {\bar{x}} \in x - \theta [\nabla f({\bar{x}}) + \partial \varphi ({\bar{x}})] \\&\iff \, {\bar{x}}\in {\bar{x}} - \lambda _{+}[ \nabla f({\bar{x}}) + \theta ^{-1} ({\bar{x}} - x)] - \lambda _{+} \partial \varphi ({\bar{x}}) \\&\iff \, {\bar{x}} = {\mathrm {prox}}^{}_{\lambda _{+}\varphi }({\bar{x}} - \lambda _{+}\nabla f({\bar{x}}) - \lambda _{+}\theta ^{-1}[{\bar{x}} - x]). \end{aligned}$$

Now, setting $z := x + \beta d$ and $q := \alpha d + \lambda _{+}(\nabla f(x) - \nabla f(z))$, we have $p = \lambda _{+}[\nabla f({\bar{x}}) - \nabla f(x)] + q$ and $\Vert q\Vert \le (\alpha + L_f \beta \lambda _{+}) \Vert d\Vert$. Furthermore, using Young’s inequality, the nonexpansiveness of the proximity operator and the Lipschitz continuity of $\nabla f$, we obtain

$$\begin{aligned} \Vert p_+ - {\bar{x}} \Vert ^2&=\Vert {\mathrm {prox}}^{}_{\lambda _{+}\varphi }({\bar{x}} - \lambda _{+}\nabla f({\bar{x}}) - \lambda _{+}\theta ^{-1}[{\bar{x}} - x])- {\mathrm {prox}}^{}_{\lambda _+\varphi }(x+\alpha d - \lambda _+v_+)\Vert ^2\\&\le \Vert (1 - \lambda _{+}\theta ^{-1})[{\bar{x}} - x] - p + \lambda _{+}(v_+ - \nabla f(z))\Vert ^2 \\&= \Vert (1- \lambda _{+}\theta ^{-1})[{\bar{x}} - x] - \lambda _{+} [\nabla f({\bar{x}}) - \nabla f(x)]\Vert ^2 \\&\quad - 2\langle (1-\lambda _{+}\theta ^{-1})[{\bar{x}} - x], q \rangle - 2 \lambda _{+} \langle \nabla f({\bar{x}}) - \nabla f(x), q \rangle + \Vert q\Vert ^2 \\&\quad + 2 \lambda _{+} \langle (1-\lambda _{+}\theta ^{-1})[{\bar{x}} - x] -p, v_+ - \nabla f(z) \rangle + \lambda _{+}^2 \Vert \nabla f(z) - v_+\Vert ^2 \\&\le \left[ (1+\rho _1)(1 - {\lambda _{+}}{\theta }^{-1})^2 + 2\lambda _{+}(1 - {\lambda _{+}}{\theta }^{-1})L_f +(1+\rho _2)L_f^2\lambda _{+}^2 \right] \Vert {\bar{x}} - x\Vert ^2 \\&\quad + \left[ 1+\frac{1}{\rho _{1}}+\frac{1}{\rho _{2}} \right] \mu ^2\Vert d\Vert ^2+ \lambda _{+}^2 \Vert \nabla f(z) - v_+\Vert ^2 \\&\quad +2 \lambda _{+} \langle (1-\lambda _+\theta ^{-1})[{\bar{x}} - x] -p, v_+ - \nabla f(z) \rangle , \end{aligned}$$

where $\mu = \alpha + L_f \beta \lambda _+$. This establishes the statement in Lemma 3.2. $\square$

1.2 9.2 Proof of Corollary  4.1

Proof

We need to verify that the choice of $\lambda$ and $\lambda _+$ satisfies the constraints derived in Theorem 4.1 and in (4.4), respectively. Notice that we can set $\bar{\rho }= 0$ and we can work with $(\lambda _+^m)^{-1}$ instead of $(2\lambda _+^m)^{-1}$ in (4.4). Due to the definition of b and $b_+$, we have $\tau _m = \sqrt{2}$ for all m and thus, it follows

$$\begin{aligned} L_f - \frac{1}{\lambda _+} + \frac{K(K-1)}{2} \theta (\lambda _+)&\le L - \frac{1}{\lambda _+} + \frac{L^2 K^2}{2} \left[ \frac{1}{b_+} + \frac{1}{b} \right] \lambda _+ = 0. \end{aligned}$$

Furthermore, with the choice $\lambda _+ = \gamma L^{-1}$ it holds that

$$\begin{aligned} {\mathcal {L}}_k^m&= \left[ \frac{(\alpha _k^m)^2}{\lambda _+} + 2L_f \alpha _k^m \beta _k^m + L_f^2 (\beta ^m_k)^2 \lambda _+ + \frac{L^2(\beta _k^m)^2}{K} \lambda _+ \right] (\nu ^m_k)^2 + \frac{1}{\lambda _+} \\&\le \left[ \frac{1}{\gamma } + 2 + \gamma + \frac{\gamma }{K}\right] L \bar{\nu }^2 + \frac{L}{\gamma } \le \frac{L}{\gamma }(1+3\bar{\nu }^2) \end{aligned}$$

and we have ${\mathcal {L}}^m_k \le \frac{1}{\lambda }$ for all k and m. Hence, the estimate (4.7) follows from (4.5).

Since each iteration in the inner loop of Algorithm 2 requires $b+b_+$ gradient component evaluations (IFO), the total amount of evaluations in a single outer iteration is given by $N + K(b + b_+)$ and the corresponding total number of gradient component evaluations after M outer iterations is

$$\begin{aligned} MK \cdot \left[ \frac{N}{K} + b+b_+ \right] = MK \cdot \left[ \frac{N}{K} + 2K^2 \right] . \end{aligned}$$

Moreover, since the bound for $\mathbb {E}[\Vert F^1(\mathsf{X})\Vert ^2]$ in (4.7) is proportional to $(MK)^{-1}$, the IFO complexity of Algorithm 2 for reaching an $\varepsilon$-accurate stationary point with $\mathbb {E}[\Vert F^1(\mathsf{X})\Vert ^2] \le \varepsilon$ is ${\mathcal {O}}((N/K + 2K^2)/\varepsilon )$. Minimizing this expression with respect to K yields $K \sim N^{1/3}$ and the IFO complexity ${\mathcal {O}}(N^{2/3}/\varepsilon )$. $\square$