Efficient random coordinate descent algorithms for large-scale structured nonconvex optimization

Abstract

In this paper we analyze several new methods for solving nonconvex optimization problems with the objective function consisting of a sum of two terms: one is nonconvex and smooth, and another is convex but simple and its structure is known. Further, we consider both cases: unconstrained and linearly constrained nonconvex problems. For optimization problems of the above structure, we propose random coordinate descent algorithms and analyze their convergence properties. For the general case, when the objective function is nonconvex and composite we prove asymptotic convergence for the sequences generated by our algorithms to stationary points and sublinear rate of convergence in expectation for some optimality measure. Additionally, if the objective function satisfies an error bound condition we derive a local linear rate of convergence for the expected values of the objective function. We also present extensive numerical experiments for evaluating the performance of our algorithms in comparison with state-of-the-art methods.

Appendix

Proof of Lemma 4

We derive our proof based on the following remark (see also [5]), for given $u, v \in \mathbb {R}^n$ if $\langle v, u-v \rangle > 0$, then

$$\begin{aligned} \frac{||u||}{||v||} \le \frac{\langle u, u-v \rangle }{\langle v, u-v \rangle }. \end{aligned}$$

(27)

Let $\alpha > \beta > 0$. Taking $u = \text {prox}_{\alpha h}(x + \alpha d) -x$ and $v= \text {prox}_{\beta h}(x+ \beta d) - x$, we show first that inequality $\langle v, u-v \rangle > 0$ holds. Given a real constant $c>0$, from the optimality conditions corresponding to proximal operator we have:

$$\begin{aligned} x - \text {prox}_{c h}(x) \in \partial c h(\text {prox}_{c h}(x)). \end{aligned}$$

Therefore, from the convexity of $h$ we can derive that:

$$\begin{aligned} c h(z) \ge c h(\text {prox}_{c h}(y)) + \langle y - \text {prox}_{ch}(y), z - \text {prox}_{c h}(y) \rangle \quad \; \forall y, z \in \mathbb {R}^n. \end{aligned}$$

Taking $c=\alpha$, $z = \text {prox}_{\beta h}(x + \beta d)$ and $y = x + \alpha d$ we have:

$$\begin{aligned} \langle u, u - v \rangle \le \alpha \left( \langle d, u-v \rangle + h(\text {prox}_{\beta h} (x+\beta d)) - h(\text {prox}_{\alpha h}(x+ \alpha d)) \right) . \end{aligned}$$

(28)

Also, if $c=\beta$, $z= \text {prox}_{\alpha h}(x+ \alpha d)$ and $y = x+ \beta d$, then we have:

$$\begin{aligned} \langle v, u-v \rangle \ge \beta \left( \langle d, u-v \rangle + h(\text {prox}_h(x+\beta d)) - h(\text {prox}_h(x+ \alpha d)) \right) . \end{aligned}$$

(29)

Summing these two inequalities and taking in account that $\alpha > \beta$ we get:

$$\begin{aligned} \langle d, u -v \rangle + h(\text {prox}_h(x+\beta d)) - h(\text {prox}_h(x+ \alpha d))> 0. \end{aligned}$$

Therefore, replacing this expression into inequality (28) leads to $\langle v, u-v \rangle > 0$. Finally, from (27),(28) and (29) we get the inequality:

$$\begin{aligned} \frac{||u||}{||v||} \le \frac{\alpha \langle d, u-v \rangle }{\beta \langle d, u-v \rangle }, \end{aligned}$$

and then the statement of Lemma 4 can be easily derived. $\square$