Abstract
Recent years have witnessed the surge of asynchronous parallel (async-parallel) iterative algorithms due to problems involving very large-scale data and a large number of decision variables. Because of asynchrony, the iterates are computed with outdated information, and the age of the outdated information, which we call delay, is the number of times it has been updated since its creation. Almost all recent works prove convergence under the assumption of a finite maximum delay and set their stepsize parameters accordingly. However, the maximum delay is practically unknown. This paper presents convergence analysis of an async-parallel method from a probabilistic viewpoint, and it allows for large unbounded delays. An explicit formula of stepsize that guarantees convergence is given depending on delays’ statistics. With \(p+1\) identical processors, we empirically measured that delays closely follow the Poisson distribution with parameter p, matching our theoretical model, and thus, the stepsize can be set accordingly. Simulations on both convex and nonconvex optimization problems demonstrate the validness of our analysis and also show that the existing maximum-delay-induced stepsize is too conservative, often slows down the convergence of the algorithm.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
In the “big data” era, the size of the dataset and the number of decision variables involved in many areas such as health care, the Internet, economics, and engineering are becoming tremendously large [1]. It motivates the development of new computational approaches by efficiently utilizing modern multi-core computers or computing clusters.
In this paper, we consider the block-structured optimization problem
where \({\mathbf {x}}=({\mathbf {x}}_1,\ldots ,{\mathbf {x}}_m)\) is partitioned into m disjoint blocks, f has a Lipschitz continuous gradient (possibly nonconvex), and \(r_i\)’s are (possibly nondifferentiable) proper closed convex functions. Note that \(r_i\)’s can be extended-valued, and thus, (1.1) can have block constraints \({\mathbf {x}}_i\in X_i\) by incorporating the indicator function of \(X_i\) in \(r_i\) for all i.
Many applications can be formulated in the form of (1.1), and they include classic machine learning problems: support vector machine (squared hinge loss and its dual formulation) [2], least absolute shrinkage and selection operator (LASSO) [3], and logistic regression (linear or multilinear) [4], and also subspace learning problems: sparse principal component analysis [5], nonnegative matrix or tensor factorization [6], just to name a few.
Toward the solutions for these problems with extremely large-scale datasets and many variables, first-order methods and also stochastic methods become particularly popular because of their scalability to the problem size, such as fast iterative shrinkage-thresholding algorithm (FISTA) [7], stochastic approximation [8], randomized coordinate descent [9], and their combinations [10, 11]. Recently, lots of efforts have been made to the parallelization of these methods, and in particular, asynchronous parallel (async-parallel) methods attract more attention (e.g., [12, 13]) over their synchronous counterparts partly due to the better speedup performance.
This paper focuses on the async-parallel block coordinate update (async-BCU) method (see Algorithm 1) for solving (1.1). To the best of our knowledge, all works on async-BCU before 2013 consider a deterministic selection of blocks with an exception to [14], and thus, they require strong conditions (like a contraction) for convergence. Recent works, e.g., [12, 13, 15, 16], employ randomized block selection and significantly weaken the convergence requirement. However, all of them require bounded delays and/or are restricted to convex problems. The work [15] allows unbounded delays but requires convexity, and [17, 18] do not assume convexity but require bounded delays. We consider unbounded delays and deal with nonconvex problems.
1.1 Algorithm
We describe the async-BCU method as follows. Assume there are \(p+1\) processors, and the data and variable \({\mathbf {x}}\) are accessible to all processors. We let all processors continuously and asynchronously update the variable \({\mathbf {x}}\) in parallel. At each time k, one processor reads the variable \({\mathbf {x}}\) as \(\hat{{\mathbf {x}}}^k\) from the global memory, randomly picks a block \(i_k\in \{1,2,\ldots ,m\}\), and renews \({\mathbf {x}}_{i_k}\) by a prox-linear update while keeping all other blocks unchanged. The pseudocode is summarized in Algorithm 1, where the \({\mathbf {prox}}\) operator is defined in (1.3).
The algorithm first appeared in [12], where the age of \({\hat{{\mathbf {x}}}}^k\) relative to \({\mathbf {x}}^k\), which we call the delay of iteration k, was assumed to be bounded by a certain integer \(\tau \). For general convex problems, sublinear convergence was established, and for the strongly convex case, linear convergence was shown. However, its convergence for nonconvex problems and/or with unbounded delays was unknown. In addition, numerically, the stepsize is difficult to tune because it depends on \(\tau \), which is unknown before the algorithm completes.
1.2 Contributions
We summarize our contributions as follows:
-
We analyze the convergence of Algorithm 1 and allow for large unbounded delays following a certain distribution. We require the delays to have certain bounded expected quantities (e.g., expected delay, variance of delay). Our results are more general than those requiring bounded delays such as [12, 16].
-
Both nonconvex and convex problems are analyzed, and those problems include both smooth and nonsmooth functions. For nonconvex problems, we establish the global convergence in terms of first-order optimality conditions and show that any limit point of the iterates is a critical point almost surely. It appears to be the first result of an async-BCU method for general nonconvex problems and allowing unbounded delays. For weakly convex problems, we establish a sublinear convergence result, and for strongly convex problems, we show the linear convergence.
-
We show that if all \(p+1\) processors run at the same speed, the delay follows the Poisson distribution with parameter p. In this case, all the relevant expected quantities can be explicitly computed and are bounded. By setting appropriate stepsizes, we can reach a near-linear speedup if \(p=o(\sqrt{m})\) for smooth cases and \(p=o(\root 4 \of {m})\) for nonsmooth cases.
-
When the delay follows the Poisson distribution, we can explicitly set the stepsize based on the delay expectation (which equals p). We simulate the async-BCU method on one convex problem: LASSO, and one nonconvex problem: the nonnegative matrix factorization. The results demonstrate that async-BCU performs consistently better with a stepsize set based on the expected delay than on the maximum delay. The number of processors is known while the maximum delay is not. Hence, the setting based on expected delay is practically more useful.
Our algorithm updates one (block) coordinate of \({\mathbf {x}}\) in each step and is sharply different from stochastic gradient methods that sample one function in each step to update all coordinates of \({\mathbf {x}}\). While there are async-parallel algorithms in either classes and how to handle delays is important to both of their convergence, their basic lines of analysis are different with respect to how to absorb the delay-induced errors. The results of the two classes are in general not comparable. That is said, for problems with certain proper structures, it is possible to apply both coordinate-wise update and stochastic sampling (e.g., [11, 18,19,20]), and our results apply to the coordinate part.
1.3 Notation and Assumptions
Throughout the paper, bold lowercase letters \({\mathbf {x}}, {\mathbf {y}},\ldots ,\) are used for vectors. We denote \({\mathbf {x}}_i\) as the ith block of \({\mathbf {x}}\) and \(U_i\) as the ith sampling matrix, i.e., \(U_i {\mathbf {x}}\) is a vector with \({\mathbf {x}}_i\) as its ith block and \(\mathbf {0}\) for the remaining ones. \(\textit{E}_{i_k}\) denotes the expectation with respect to \(i_k\) conditionally on all previous history, and \([m]=\{1,\ldots ,m\}\).
We consider the Euclidean norm denoted by \(\Vert \cdot \Vert \), but all our results can be directly extended to problems with general primal and dual norms in a Hilbert space.
The projection to a convex set X is defined as
and the proximal mapping of a convex function h is defined as
Definition 1.1
(Critical point) A point \({\mathbf {x}}^*\) is a critical point of (1.1) if \(\mathbf {0}\in \nabla f({\mathbf {x}}^*)+\partial R({\mathbf {x}}^*),\) where \(\partial R({\mathbf {x}})\) denotes the subdifferential of R at \({\mathbf {x}}\) and
Throughout our analysis, we make the following three assumptions to problem (1.1) and Algorithm 1. Other assumed conditions will be specified if needed.
Assumption 1.1
The function F is lower bounded. The problem (1.1) has at least one solution, and the solution set is denoted as \(X^*\).
Assumption 1.2
\(\nabla f({\mathbf {x}})\) is Lipschitz continuous with constant \(L_f\), namely,
In addition, for each \(i\in [m]\), fixing all block coordinates but the ith one, \(\nabla f({\mathbf {x}})\) and \(\nabla _i f({\mathbf {x}})\) are Lipschitz continuous about \({\mathbf {x}}_i\) with \(L_r\) and \(L_c\), respectively, i.e., for any \({\mathbf {x}},~{\mathbf {y}}\), and i,
From (1.6), we have that for any \({\mathbf {x}},~{\mathbf {y}}\), and i,
We denote \(\kappa =\frac{L_r}{L_c}\) as the condition number.
Assumption 1.3
For each \(k\geqslant 1\), the reading \(\hat{{\mathbf {x}}}^k\) is consistent and delayed by \(j_k\), namely, \(\hat{{\mathbf {x}}}^k={\mathbf {x}}^{k-j_k}\). The delay \(j_k\) follows an identical distribution as a random variable \({\mathbf {j}}\)
and is independent of \(i_k\). We let
Remark 1.1
Although the delay always satisfies \(0\leqslant j_k\leqslant k\), the assumption in (1.8) is without loss of generality if we make negative iterates and regard \({\mathbf {x}}^k={\mathbf {x}}^0,\,\forall k<0\). For simplicity, we make the identical distribution assumption, which is the same as that in [14]. Our results can still hold for nonidentical distribution; see the analysis for the smooth nonconvex case in the arXiv version of the paper.
2 Related Works
We briefly review block coordinate update (BCU) and async-parallel computing methods.
The BCU method is closely related to the Gauss–Seidel method for solving linear equations, which can date back to 1823. In the literature of optimization, BCU method first appeared in [21] as the block coordinate descent method, or more precisely, block minimization (BM), for quadratic programming. The convergence of BM was established early for both convex and nonconvex problems, for example [22,23,24]. However, in general, its convergence rate result was only shown for strongly convex problems (e.g., [23]) until the recent work [25] shows sublinear convergence for weakly convex cases. Tseng and Yun [26] proposed a new version of BCU methods, called coordinate gradient descent method, which mimics proximal gradient descent but only updates a block coordinate every time. The block coordinate gradient or block prox-linear update (BPU) becomes popular since [9] proposed to randomly select a block to update. The convergence rate of the randomized BPU is easier to show than the deterministic BPU. It was firstly established for convex smooth problems (both unconstrained and constrained) in [9] and then generalized to nonsmooth cases in [27, 28]. Recently, Refs. [10, 11] incorporated stochastic approximation into the BPU framework to deal with stochastic programming, and both established sublinear convergence for convex problems and also global convergence for nonconvex problems.
The async-parallel computing method (also called chaotic relaxation) first appeared in [29] to solve linear equations arising in electrical network problems. Chazan and Miranker [30] first systematically analyzed (more general) asynchronous iterative methods for solving linear systems. Assuming bounded delays, it gave a necessary and sufficient condition for convergence. Bertsekas [31] proposed an asynchronous distributed iterative method for solving more general fixed-point problems and showed its convergence under a contraction assumption. Tseng et al. [32] weakened the contraction assumption to pseudo-nonexpansiveness but made other more assumptions. Frommer and Szyld [33] made a thorough review of asynchronous methods before 2000. It summarized convergence results under nested sets and synchronous convergence conditions, which are satisfied by P-contraction mappings and isotone mappings.
Since it was proposed in 1969, the async-parallel method has not attracted much attention until recent years when the size of data is increasing exponentially in many areas. Motivated by “big data” problems, Refs. [12, 16] proposed the async-parallel stochastic coordinate descent method (i.e., Algorithm 1) for solving problems in the form of (1.1). Their analysis focuses on convex problems and assumes bounded delays. Specifically, they established sublinear convergence for weakly convex problems and linear convergence for strongly convex problems. In addition, near-linear speedup was achieved if \(\tau =o(\sqrt{m})\) for unconstrained smooth convex problems and \(\tau =o(\root 4 \of {m})\) for constrained smooth or nonsmooth cases. For nonconvex problems, Davis [18] introduced an async-parallel coordinate descent method, whose convergence was established under iterate boundedness assumptions and appropriate stepsizes.
3 Convergence Results for the Smooth Case
Throughout this section, let \(r_i=0,\,\forall i\), i.e., we consider the smooth optimization problem
The general (possibly nonsmooth) case will be analyzed in the next section. The results for nonsmooth problems of course also hold for smooth ones. However, the smooth case requires weaker conditions for convergence than those required by the nonsmooth case, and their analysis techniques are different. Hence, we consider the two cases separately.
3.1 Convergence for the Nonconvex Case
In this subsection, we establish a subsequence convergence result for the general (possibly nonconvex) case. We begin with some technical lemmas. The first lemma deals with certain infinite sums that will appear later in our analysis.
Lemma 3.1
For any k and \(t\leqslant k\), let
Then
Proof
To bound \(\sum _{k=0}^\infty \gamma _k\), we bound the first term \(\sum _{d=1}^{k-1} (c_{k-d}-c_k) c_d\) in (3.2a). Specifically,
where the last equality holds since \(T:=\textit{E}[{\mathbf {j}}]=\sum _{t=1}^\infty t q_t=\sum _{t=1}^\infty \sum _{d=1}^t q_t=\sum _{d=1}^\infty \sum _{t=d}^\infty q_t=\sum _{d=1}^\infty c_d. \) We obtain (3.3) by combining these two equations.
To prove (3.4), we will use
The above inequality yields
where the last inequality follows from \(c_k\leqslant 1\).
The second lemma below bounds the cross term that appears in our analysis.
Lemma 3.2
(Cross term bound) For any \(k > 1\) and \(t\leqslant k\), it holds that
Proof
Define \(\Delta ^d:=\nabla f({\mathbf {x}}^d) - \nabla f({\mathbf {x}}^{d+1})\). Applying the Cauchy–Schwarz inequality with \(\nabla f({\mathbf {x}}^{k-t}) - \nabla f({\mathbf {x}}^{k}) = \sum _{d=k-t}^{k-1}\Delta ^d\) yields
Since \(\Vert \Delta ^d\Vert \leqslant L_r\Vert {\mathbf {x}}^{d+1}-{\mathbf {x}}^d\Vert = \eta L_r\Vert \nabla _{i_d}f({\hat{{\mathbf {x}}}}^{d})\Vert ,\) by applying Young’s inequality, we get
By taking expectation, we have
Now taking expectation on both sides of (3.7) and using the above equation, we get
Finally, (3.6) follows from
and
Using the above lemma, we show a result of running one iteration of the algorithm.
Theorem 3.1
(Fundamental bound) Set \(\gamma _k, \beta _k\) and \(C_{t,k}\) as in (3.2). For any \(k>1\), we have
Proof
Since \({\mathbf {x}}^{k+1}={\mathbf {x}}^k-\eta U_{i_k}\nabla f({\mathbf {x}}^{k-j_k})\), we have from (1.7) that
Taking conditional expectation on \((i_k, j_k)\) gives
For the first cross term in (3.11), we write each summand as
and we use Young’s inequality to bound the second cross term by
Now taking expectation over both sides of (3.11), plugging in (3.12) and (3.13), and using Lemma 3.2, we have the desired result.
We are now ready to show the main result in the following theorem.
Theorem 3.2
(Convergence for the nonconvex smooth case) Under Assumptions 1.1 through 1.3, let \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\) be generated from Algorithm 1. Assume \(T<\infty \). Take the stepsize as \(0<\eta <\frac{1/L_c}{1+2\kappa T/\sqrt{m}}.\) If \(q_0>0\) or \(\nabla f({\mathbf {x}})\) is bounded for all \({\mathbf {x}}\), then
and any limit point of \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\) is almost surely a critical point of (3.1).
Remark 3.1
If \(T=\textit{E}[{\mathbf {j}}]=o(\sqrt{m})\), then \(\eta \) only weakly depends on the delay. The conditions \(q_0>0\) or \(\nabla f({\mathbf {x}})\) being bounded can be dropped if \(S=\textit{E}[{\mathbf {j}}^2]\) is bounded; see Theorem 4.1.
Proof
Summing up (3.10) from \(k=0\) through K and using (A.3), we have
Note that \(\beta _K\rightarrow \left( \frac{\eta }{m}-\frac{\eta ^2 L_c}{2m}\right) q_0\) as \(K\rightarrow \infty \). If \(q_0>0\) or \(\Vert \nabla f({\mathbf {x}})\Vert \) is bounded, by letting \(K\rightarrow \infty \) in (3.15) and using the lower boundedness of f, we have from Lemma 3.1 that
Since \(\eta <\frac{1/L_c}{1+2\kappa T/\sqrt{m}}\), we have (3.14) from the above inequality.
From the Markov inequality, it follows that \(\Vert \nabla f({\mathbf {x}}^k)\Vert \) converges to zero with probability one. Let \(\bar{{\mathbf {x}}}\) be a limit point of \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\), i.e., there is a subsequence \(\{{\mathbf {x}}^k\}_{k\in {\mathcal {K}}}\) convergent to \(\bar{{\mathbf {x}}}\). Hence, \(\Vert \nabla f({\mathbf {x}}^k)\Vert \rightarrow 0\) almost surely as \({\mathcal {K}}\ni k\rightarrow \infty \). By [34, Theorem 3.4, p. 212], there is a sub-subsequence \(\{{\mathbf {x}}^k\}_{k\in {\mathcal {K}}'}\) such that \(\Vert \nabla f({\mathbf {x}}^k)\Vert \rightarrow 0\) almost surely as \({\mathcal {K}}'\ni k\rightarrow \infty \). This completes the proof.
3.2 Convergence Rate for the Convex Case
In this subsection, we assume the convexity of f and establish convergence rate results of Algorithm 1 for solving (3.1). Besides Assumptions 1.1 through 1.3, we make an additional assumption to the delay as follows. It means the delay follows a sub-exponential distribution.
Assumption 3.1
There is a constant \(\sigma >1\) such that
The condition in (3.16) is stronger than \(T<\infty \), and both of them hold if the delay \(j_k\) is uniformly bounded by some number \(\tau \) or follows the Poisson distribution; see the discussions in Sect. 5. Using this additional assumption and choosing an appropriate stepsize, we are able to control the gradient of f such that it changes not too fast.
Lemma 3.3
Under Assumptions 1.2 through 3.1, for any \(1<\rho \leqslant \sigma \), if the stepsize satisfies
with \(M_\rho \) defined in (3.16), then for all k, it holds that
The proof of Lemma 3.3 follows an argument similar to [12]. Since it is rather long, it is included in Appendix. Similar to Lemma 3.2, we can show the following result.
Lemma 3.4
For any k, it holds that
Proof
Following an argument similar to how (3.8) is obtained, we can show
Using the above inequalities, we complete the proof by noting (3.9),
and \(\textstyle c_k\sum _{d=0}^{k-1}\sum _{r=0}^{d-1} q_r \Vert \nabla f({\mathbf {x}}^{d-r})\Vert ^2=\sum _{t=1}^{k-1}\sum _{d=1}^t c_k q_{t-d}\Vert \nabla f({\mathbf {x}}^{k-t})\Vert ^2.\)
Using the above two lemmas, we establish sufficient objective decrease.
Theorem 3.3
(Sufficient progress) Under Assumptions 1.1 through 3.1, we let \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\) be the sequence generated from Algorithm 1. For a certain \(1<\rho < \sigma \), define
Take the stepsize such that (3.17) is satisfied and also
Let
Then,
Proof
First note that for any \(\rho <\sigma , t\rho ^t\) is dominated by \(\sigma ^t\) as t is sufficiently large. Hence, \(N_\rho <\infty \) from (3.16), and it is easy to see \(T<\infty \). Also note that
We write the cross terms in (3.11) to
Taking expectation on both sides of (3.11) and using (3.19), we have
The above inequality together with (3.18) implies
Note that \(\sum _{t=1}^{k-1}\sum _{d=1}^t c_dq_{t-d}\rho ^t\leqslant \sum _{t=1}^\infty \sum _{d=1}^t c_dq_{t-d}\rho ^t\), which by exchanging summations equals \(\sum _{d=1}^\infty c_d\rho ^d\sum _{t=d}^\infty q_{t-d}\rho ^{t-d}\overset{(3.25)}{\leqslant }N_\rho M_\rho \). Also note that
From these relations and (3.27), we obtain
which completes the proof.
Using (3.24) and the convexity of f, we establish the following convergence rate.
Theorem 3.4
(Convergence rate for the convex smooth case) Under the assumptions of Theorem 3.3, we have
-
1.
If f is convex and \(\Vert {\mathbf {x}}^k-{\mathcal {P}}_{X^*}({\mathbf {x}}^k)\Vert \leqslant B,\,\forall k\) for a certain constant B, then
$$\begin{aligned} \textit{E}[f({\mathbf {x}}^{k+1})-f^*] \leqslant \frac{1}{(f({\mathbf {x}}^0)-f^*)^{-1}+(k+1)DB^{-2}}, \end{aligned}$$(3.28)where \(f^*\) denotes the minimum value of (3.1) and D is given in (3.23).
-
2.
If f is strongly convex with constant \(\mu \), then
$$\begin{aligned} \textit{E}[f({\mathbf {x}}^{k+1})-f^*]\leqslant (1-2\mu D)\textit{E}[f({\mathbf {x}}^k)-f^*], \end{aligned}$$(3.29)where D is given in (3.23).
Remark 3.2
For the sublinear rate in (3.28), we assume the boundedness of the iterates. This assumption can be relaxed if we use potentially smaller stepsize; see Theorem 4.2.
For the linear convergence, the assumption on strongly convexity can be weakened to either essential or restrict strong convexity, see [12, 35].
Proof
If \(\Vert {\mathbf {x}}^k-{\mathcal {P}}_{X^*}({\mathbf {x}}^k)\Vert \leqslant B\), then from \(f({\mathbf {x}}^k)-f({\mathcal {P}}_{X^*}({\mathbf {x}}^k))\leqslant \langle \nabla f({\mathbf {x}}^k),{\mathbf {x}}^k-{\mathcal {P}}_{X^*}({\mathbf {x}}^k)\rangle \), we have
and thus
Substituting (3.30) into (3.24) yields
Hence,
and thus (3.28) holds.
If f is strongly convex with constant \(\mu \), then
We immediately have (3.29) from (3.24) and the above inequality. This completes the proof.
4 Convergence Results for the Nonsmooth Case
In this section, we analyze the convergence of Algorithm 1 for possibly nonsmooth cases. Throughout this section, we let
a virtual full-update iterate, where R is defined in (1.4), and denote
Due to more generality, we will make stronger assumptions on the delay than those made in the previous section. But all these assumptions are satisfied if the delay is uniformly bounded or follows the Poisson distribution, as shown in Sect. 5.
4.1 Convergence for the Nonconvex Case
We first establish the almost sure global convergence for possibly nonconvex cases starting with the following square summable result.
Lemma 4.1
(Square summability) Under Assumptions 1.1 through 1.3, we let \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\) be the sequence generated in Algorithm 1. Assume \(S<\infty \), and the stepsize is taken as \(0<\eta < \frac{1/L_c}{1+\kappa ^2S/(2m)}.\) Then
Proof
By the definition of \(\bar{{\mathbf {x}}}^{k+1}\), we have \(-\nabla f({\mathbf {x}}^{k-j_k})-\tfrac{1}{\eta }{\mathbf {d}}^k\in \partial R(\bar{{\mathbf {x}}}^{k+1})\), which together with the convexity of R implies that, for any \({\mathbf {x}}\),
By \({\mathbf {x}}^{k+1}={\mathbf {x}}^k+U_{i_k}{\mathbf {d}}^k\) and (1.7), we get \(F({\mathbf {x}}^{k+1})\leqslant f({\mathbf {x}}^k)+\langle \nabla _{i_k}f({\mathbf {x}}^k),{\mathbf {d}}^k_{i_k}\rangle +\tfrac{L_c}{2}\Vert {{\mathbf {d}}}^{k}_{i_k}\Vert ^2+R({\mathbf {x}}^{k+1}).\) To this inequality, take conditional expectation on \(i_k\):
To bound the right-hand side, we split the cross term as
and apply (4.2) with \({\mathbf {x}}={\mathbf {x}}^k\), arriving at
Following a similar argument in the proof of Lemma 3.2 and Young’s inequality, we get
Note that
Hence, taking expectation yields
Taking expectation on both sides of (4.3) and substituting (4.6) yield
From Lemma A.1, we have that for any \(K\geqslant 0\),
Summing up (4.7) from \(k=0\) through K and substituting (4.8) and (4.9), we have
Note that
Since F is lower bounded, we have (4.1) from (4.10) by letting \(K\rightarrow \infty \).
Since \(\left( \textit{E}[{\mathbf {j}}]\right) ^2\leqslant \textit{E}[{\mathbf {j}}^2]\), the condition \(S<\infty \) implies \(T<\infty \). Equation (4.1) indicates that \(\textit{E}\Vert {\mathbf {d}}^k\Vert \rightarrow 0\) as \(k\rightarrow \infty \). Together with \(S<\infty \), we are able to show \(\textit{E}\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert \) also approaches zero, as summarized in the following.
Lemma 4.2
Under the assumptions of Lemma 4.1, we have
Proof
Pick any \(\varepsilon >0\). From (4.1), there must exist an integer \(J>0\) such that
For the above J, there must exist an integer \(K>J\) such that, for any \(k\geqslant K\),
From Young’s inequality, it follows that \(\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert ^2\leqslant j_k\sum _{d=k-j_k}^{k-1}\Vert {\mathbf {x}}^{d+1}-{\mathbf {x}}^d\Vert ^2.\) Hence, for any \(k\geqslant K\), using (4.5) and (A.1), we have
which implies \(\textit{E}\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert ^2\leqslant \varepsilon \) under (4.11) and (4.12). We have \(\lim _{k\rightarrow \infty }\textit{E}\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert ^2=0\) as \(\varepsilon \) is arbitrary. Now note \(\textit{E}\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert \leqslant \sqrt{\textit{E}\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert ^2}\) to complete the proof.
Using Lemmas 4.1 and 4.2, we establish the almost sure global convergence of Algorithm 1.
Theorem 4.1
Under the assumptions of Lemma 4.1, any limit point \({\mathbf {x}}^*\) of \(\{{\mathbf {x}}^k\}\) is a critical point of (1.1) almost surely.
Before proving this theorem, we make two remarks as follows.
Remark 4.1
From the theorem, we see that if \(S=\textit{E}[{\mathbf {j}}^2]=o(m)\), then the stepsize required for convergence only weakly depends on the delay.
Remark 4.2
(Comparison of stepsize) The works [18] consider asynchronous coordinate descent for nonconvex problems. To have convergence to critical points, they assume delays bounded by a number \(\tau \). Also, they require the boundedness of iterates and the stepsize less than \(\frac{1/L_c}{1+2\kappa \tau /\sqrt{m}}\). Note that our stepsize in Theorem 4.1 is larger if \(\kappa ^2 S\leqslant 16m\), where \(S=\textit{E}[{\mathbf {j}}^2]<\tau ^2\), and that can lead to faster convergence.
Proof
Let \(\{{\mathbf {x}}^k\}_{k\in {\mathcal {K}}}\) be a subsequence that converges to \({\mathbf {x}}^*\). Since \(\textit{E}\Vert {\mathbf {d}}^k\Vert \rightarrow 0\) as \({\mathcal {K}}\ni k\rightarrow \infty \), from the Markov inequality, \(\Vert {\mathbf {d}}^k\Vert \) converges to zero in probability as \({\mathcal {K}}\ni k\rightarrow \infty \). By [34, Theorem 3.4, p. 212], there is a sub-subsequence \(\{{\mathbf {x}}^k\}_{k\in {\mathcal {K}}'}\) such that \(\Vert {\mathbf {d}}^k\Vert \) almost surely converges to zero as \({\mathcal {K}}'\ni k\rightarrow \infty \). Hence, \(\bar{{\mathbf {x}}}^{k+1}\) almost surely converges to \({\mathbf {x}}^*\) as \({\mathcal {K}}'\ni k\rightarrow \infty \).
Since \(-\nabla f({\mathbf {x}}^{k-j_k})-\frac{1}{\eta }{\mathbf {d}}^k\in \partial R(\bar{{\mathbf {x}}}^{k+1})\), we have
Using triangle inequality and the Lipschitz continuity of \(\nabla f\), and taking expectation give
From Lemmas 4.1 and 4.2, it follows that the right-hand side approaches to zero as \(k\rightarrow \infty \). Hence, \(\textit{E}\text {dist}\left( \mathbf {0},\partial F(\bar{{\mathbf {x}}}^{k+1})\right) \rightarrow 0\) as \(k\rightarrow \infty \). If necessary, passing to another subsequence, we use Markov inequality and [34, Theorem 3.4, p. 212] again to have \(\text {dist}\left( \mathbf {0},\partial F(\bar{{\mathbf {x}}}^{k+1})\right) \) almost surely converges to zero as \({\mathcal {K}}'\ni k\rightarrow \infty \). Now use the outer semicontinuity [36] of \(\text {dist}\left( \mathbf {0},\partial F({\mathbf {x}})\right) \) to obtain the desired result.
4.2 Convergence Rate for the Convex Case
In this subsection, we establish convergence rates of Algorithm 1 for nonsmooth convex cases. Similar to (3.18), we first show that choosing an appropriate stepsize, the iterate difference does not change too fast.
Lemma 4.3
(Fundamental bounds) Assume Assumptions 1.2 through 3.1. Then for any \(1<\rho <\sigma \), it holds that
In addition, if the stepsize is taken such that
then, for all \(k\geqslant 1\),
Proof
It is easy to show (4.13) by noting that \(t\rho ^t\) is dominated by \(\sigma ^t\) as t is sufficiently large. Next we show (4.15) by induction.
Using the inequality \(\Vert {\mathbf {u}}\Vert ^2-\Vert {\mathbf {v}}\Vert ^2\leqslant 2\Vert {\mathbf {u}}\Vert \cdot \Vert {\mathbf {v}}-{\mathbf {u}}\Vert \), we have
In addition, for all k,
Furthermore, from \({\mathbf {d}}^{k}-{\mathbf {d}}^{k-1}={\mathbf {x}}^k-{\mathbf {prox}}_{\eta R}\left( {\mathbf {x}}^k-\eta \nabla f({\mathbf {x}}^{k-j_k})\right) -{\mathbf {x}}^{k-1}+{\mathbf {prox}}_{\eta R}\left( {\mathbf {x}}^{k-1}-\eta \nabla f({\mathbf {x}}^{k-1-j_{k-1}})\right) \), the nonexpansiveness of \({\mathbf {prox}}_{\eta R}\), and the triangle inequality, we have
When \(k=1\), we have \(j_0=0\) and \(j_1\in \{0, 1\}\) because \(j_k\leqslant k,\,\forall k\). Hence, from (4.18),
which together with (4.16) and (4.17) implies
Hence,
Assume (4.15) holds for all \(k\leqslant K-1\). We show it holds for \(k=K\). First, for any \(d\leqslant K-1\),
Secondly, we have
Note that
By the triangle inequality and the Lipschitz of \(\nabla f\), it follows that, for any \(1\leqslant t\leqslant K\),
Since \(\Vert {\mathbf {d}}^{K-1}\Vert \) is independent of \(j_K\), we have from the above two equations that
Using (4.20), the definition of \(\gamma _{\rho ,1}\) in (4.13) and \(\sum _{d=K-t}^{K-1} \rho ^{\frac{K-1-d}{2}}=\frac{\rho ^{t/2}-1}{\rho ^{1/2}-1},\,\forall 1\leqslant t\leqslant K,\) we have
Also, using Young’s inequality and (4.22) with K replaced by \(K-1\) and \(t=j_{K-1}\), we have, for any \(\beta >0\),
Note that
Substituting this inequality into (4.24), noting \(\textit{E}\Vert {\mathbf {x}}^d-{\mathbf {x}}^{d+1}\Vert ^2=\frac{1}{m}\textit{E}\Vert {\mathbf {d}}^d\Vert ^2\), and applying (4.15) for all \(k\leqslant K-1\), we have
where \(C= \frac{L_r}{2\beta }+\frac{L_r\beta }{2m}\sum \nolimits _{t=1}^{K-2}q_t t\sum \nolimits _{d=K-1-t}^{K-2}\rho ^{K-1-d}+\frac{L_r\beta }{2m}c_{K-1}(K-1)\sum \nolimits _{d=0}^{K-2}\rho ^{K-1-d}\). Now let \(\beta ={\sqrt{m}}{\left( \sum _{t=1}^{K-2}q_tt\frac{\rho ^{t}-1}{1-\rho ^{-1}}+c_{K-1}(K-1)\frac{\rho ^{K-1}-1}{1-\rho ^{-1}}\right) ^{-1/2}}\) and recall the definition of \(\gamma _{\rho ,2}\) in (4.13). From the above inequality, we have
Substituting (4.23) and (4.25) into (4.21) gives
and thus
Therefore, by induction, it follows that (4.15) holds for all k, and we complete the proof.
By this lemma, we are able to establish the convergence rate result of Algorithm 1 for solving (1.1) if the problem is convex.
Theorem 4.2
(Convergence rate for the nonsmooth convex case) Under Assumptions 1.1 through 3.1, let \(\{{\mathbf {x}}^k\}_{k\geqslant 1}\) be the sequence generated from Algorithm 1 with stepsize satisfying (4.14) and also
where \(\gamma _{\rho ,1}\) and \(\gamma _{\rho ,2}\) are defined in (4.13). We have
-
1.
If the function F is convex, then
$$\begin{aligned} \textit{E}[F({\mathbf {x}}^{k})-F^*] \leqslant \frac{m\Phi ({\mathbf {x}}^0)}{2\eta (m+k)}, \end{aligned}$$(4.27)where
$$\begin{aligned} \Phi ({\mathbf {x}}^k)=\textit{E}\left\| {\mathbf {x}}^{k}-{\mathcal {P}}_{X^*}({\mathbf {x}}^{k})\right\| ^2+2\eta \textit{E}[F({\mathbf {x}}^{k})-F^*]. \end{aligned}$$ -
2.
If F is strongly convex with constant \(\mu \), then
$$\begin{aligned} \Phi ({\mathbf {x}}^k)\leqslant \left( 1-\frac{\eta \mu }{m(1+\eta \mu )}\right) ^{k} \Phi ({\mathbf {x}}^0). \end{aligned}$$(4.28)
Before proving this theorem, we make two remarks and present a few lemmas below.
Remark 4.3
Similar to (3.29), for the linear convergence result (4.28), the strong convexity assumption can be weakened to optimal strong convexity. The latter one is strictly weaker than the former one; see [16] for more discussions.
Remark 4.4
(Comparison of stepsize) For the special case that the delay is bounded by \(\tau =o(\root 4 \of {m})\), choosing \(\rho =O(1+\frac{1}{\tau })\), we have both \(\gamma _{\rho ,1}\) and \(\gamma _{\rho ,2}\) are \(O(\tau )\). Thus we can take stepsize almost \(\frac{1}{L_c}\), which is larger than the stepsize \(\frac{1}{2L_c}\) given in [16].
Lemma 4.4
Let \(\gamma _{\rho ,2}\) be defined in (4.13). We have
Proof
It is proved via the Cauchy–Schwarz inequality, the bound (4.25), and \(\textit{E}\langle \nabla _{i_k}f({\mathbf {x}}^{k-j_k})-\nabla _{i_k}f({\mathbf {x}}^{k}),{\mathbf {x}}_{i_k}^{k}-{\mathbf {x}}_{i_k}^{k+1}\rangle = \tfrac{1}{m}\textit{E}\langle \nabla f({\mathbf {x}}^{k-j_k})-\nabla f({\mathbf {x}}^{k}),{\mathbf {d}}^{k}\rangle \).
Lemma 4.5
It holds that
Proof
Equation (4.30) is a direct consequence of \(r_{i_k}(({\mathcal {P}}_{X^*}({\mathbf {x}}^{k}))_{i_k})-r_{i_k}({\mathbf {x}}_{i_k}^{k+1})=r_{i_k}(({\mathcal {P}}_{X^*}({\mathbf {x}}^{k}))_{i_k})-r_{i_k}({\mathbf {x}}_{i_k}^k)+R({\mathbf {x}}^{k})-R({\mathbf {x}}^{k+1}).\)
Lemma 4.6
Let \(\gamma _{\rho ,2}\) be defined in (4.13). It holds that
Proof
Since \(i_k\) is uniformly distributed and independent of \(j_k\), we have
We split the term and apply the convexity of f and Lipschitz continuity of \(\nabla f\) to get
Substituting (4.33) into (4.32) and taking expectation yield
Noting \(\Vert {\mathbf {x}}^k-{\mathbf {x}}^{k-j_k}\Vert ^2\leqslant j_k\sum _{d=k-j_k}^{k-1}\Vert {\mathbf {x}}^{d+1}-{\mathbf {x}}^d\Vert ^2\), applying (4.5) and (4.15) and using the definition of \(\gamma _{\rho ,2}\), we complete the proof of (4.31).
Lemma 4.7
Under the assumptions of Theorem 4.2, we have \(\textit{E}[F({\mathbf {x}}^{k+1})]\leqslant \textit{E}[F({\mathbf {x}}^k)],\,\forall k\).
Proof
Taking expectation on both sides of (4.3) and using (4.29) yield
which implies \(\textit{E}[F({\mathbf {x}}^{k+1})]\leqslant \textit{E}[F({\mathbf {x}}^k)]\) from the condition on \(\eta \) in (4.26).
Now we are ready to prove Theorem 4.2.
Proof of Theorem 4.2
From the update of \({\mathbf {x}}^{k+1}\), we have
and thus for any \({\mathbf {x}}_{i_k}\), it holds from the convexity of \(r_{i_k}\) that
Since \({\mathbf {x}}^{k+1}={\mathbf {x}}^{k}+U_{i_k}({\mathbf {x}}^{k+1}-{\mathbf {x}}^k)\), we have
From the definition of \({\mathcal {P}}_{X^*}\), it follows that \(\Vert {\mathbf {x}}^{k+1}-{\mathcal {P}}_{X^*}({\mathbf {x}}^{k+1})\Vert ^2\leqslant \Vert {\mathbf {x}}^{k+1}-{\mathcal {P}}_{X^*}({\mathbf {x}}^{k})\Vert ^2\). Then using (4.34) and (4.35), we have
We split the cross term to have
From (1.7), it follows that
Plugging the above two equations into (4.36) gives
Substituting (4.29) through (4.31) into (4.37) and rearranging terms yield
The above inequality together with (4.26) implies
and thus, with the monotonicity of \(\textit{E}[F({\mathbf {x}}^k)]\) in Lemma 4.7,
Hence, (4.27) follows.
When F is strongly convex with constant \(\mu \), we have
and thus from (4.38), it follows that
Therefore, (4.28) follows, and we complete the proof.
5 Poisson Distribution
We can treat the asynchronous reading and writing as a queueing system. Assume the \(p\,+\,1\) processors have the same computing power (i.e., the same speed of reading and writing). At any time k, suppose the update to \({\mathbf {x}}_{i_k}\) is performed by the \(p_k\)th processor, which can be treated as the server with speed (or service rate) one of reading and writing. All the other p processors can be treated as customers, each with speed (or arrival rate) one, where any update to \({\mathbf {x}}\) from the p processors can be regarded as one customer’s arrival. Under this setting, from the \(p_k\)th processor starts reading \({\mathbf {x}}\) until it finishes updating \({\mathbf {x}}_{i_k}\), there would be p customers in the queue in average, namely, the delay \(j_k\) follows the Poisson distribution with parameter p. Summarizing the above discussion, we have the following result.
Claim
Suppose Algorithm 1 runs on a system with \(p+ 1\) processors, which have the same speed of reading and writing during the iterations. Then the delay \(j_k\) follows the Poisson distribution with parameter p, i.e., for all k,
which implies no delay if \(p=0\).
In general, if the processors have different computing power, \(j_k\) would follow Poisson distribution with a parameter being the speed ratio of the other p processors to the \(p_k\)th one. However, in a multi-core workstation with shared memory, the processors are usually of the same style and can have the same computing ability. In the following, we assume the distribution in (1.8) to be Poisson distribution with parameter p and discuss the convergence results we obtained in the previous sections. First we give the values of the expected quantities we used before.
Proposition 5.1
Suppose there are \(p+1\) processors and (5.1) holds. Then for any \(\rho >1\), we have that for all k,
where \(\gamma _{\rho ,1}\) and \(\gamma _{\rho ,2}\) are defined in (4.13).
The proof of this proposition is standard. From the quantities in (5.2) and the theorems, which we established in the previous sections, we make the following observations:
-
1.
If \(p=o(\sqrt{m})\), we can guarantee the convergence of Algorithm 1 for both smooth and nonsmooth problems by setting \(\eta \lessapprox \frac{1}{L_c}\) (see Theorems 3.2 and 4.1), where \(\lessapprox \) means “less than but close to”;
-
2.
If \(2\mathrm{e}^2(p+1)+p=o(\sqrt{m})\), then choosing \(\rho =1+\frac{1}{p}\), we have the convergence rate of Algorithm 1 obtained in Theorem 3.4 by setting \(\eta \lessapprox \frac{2}{\mathrm{e}L_c}\). Then \(D\approx \frac{\eta }{m}\) in (3.23), and thus, near-linear speedup is achieved for solving convex smooth problems;
-
3.
If \(p=o(\root 4 \of {m})\), we can guarantee the convergence rate of Algorithm 1 in Theorem 4.2 by setting \(\eta \lessapprox \frac{1}{L_c}\) and thus a near-linear speedup for convex nonsmooth problems.
6 Numerical Experiments
In this section, we evaluate the numerical performance of Algorithm 1 on solving two problems: the LASSO problem and the nonnegative matrix factorization (NMF). The tests were carried out on a machine with 64GB memory and two Intel Xeon E5-2690 v2 processors (20 cores, 40 threads). All of the experiments were coded in C\(++\) and its threading library was used for parallelization. We use the Eigen library for numerical linear algebra operations. To measure the delay, we use an atomic variable to track the number of iterations as defined in the paper. The atomic variable will be incremented by one for each update. For each thread, the delay is calculated based on the difference of the iteration counters before and after the update. For LASSO, two different settings were used. The first one sets the stepsize by the expected delay according to the analysis of this paper, and the other one used the maximum delay from [12, 16] and is dubbed as AsySCD. We compared the async-BCU to the serial BCU, which can be regarded as a special case of Algorithm 1 with the delay \(j_k\equiv 0,\,\forall k\). For NMF, we set the stepszie by the expected delay and test its convergence behavior with different numbers of threads.
6.1 Parameter Settings
According to Theorem 4.1, the following two stepsizes were used:Footnote 1
where \(\tau \) equals the maximum number of the generated sequence of delays.
6.2 LASSO
We measure the performance of Algorithm 1 on the LASSO problem [3]
where \({\mathbf {A}}\in \mathbb {R}^{N\times n}, {\mathbf {b}}\in \mathbb {R}^N\), and \(\lambda \) is a parameter balancing the fitting term and the regularization term. We randomly generated \({\mathbf {A}}\) and \({\mathbf {b}}\) following the standard normal distribution. The size was fixed to \(n = 2N\) and \(N=10\,000\), and \(\lambda =\frac{1}{N}\) was used. The Lipschitz constant \(L_c = \max \{ \Vert ({\mathbf {A}}_i^\mathrm{T}{\mathbf {A}}_i)\Vert ^2, ~\forall i \}\), where \({\mathbf {A}}_i\) represents the ith column block of \({\mathbf {A}}\).
Delay distribution behaviors of Algorithm 1 for solving LASSO (6.2). The tested problem has 20 000 coordinates, and it was running with 5, 10, 20, and 40 threads. (Color figure online)
Figure 1 shows the delay distribution of Algorithm 1 with different numbers of threads. The blue bars are the normalized histogram so that the bar heights add to 1. Orange curve is the probability density function of Poisson distribution. By using 5 and 10 threads, we observe that the number of delays is concentrated on 4 and 9, respectively. When the number of threads is relatively large, the actual delay distribution closely matches with the theoretical distribution as we discussed in Sect. 5. For 20 threads, an interesting observation is that the actual probability density is higher than the theoretical probability density when the number of delays is around 9. We think this is due to the architecture of the testing environment, i.e., the average delay within a CPU is smaller than the average delay across two different CPUs. We observe a similar behavior when 40 threads are used.
Figure 2 plots the convergence behavior of Algorithm 1 running on 40 threads with different block sizes. We partition \({\mathbf {x}}\) into m equal-sized blocks with block sizes varying among \(\{10,~50,~100,~500\}\). The results of the serial randomized coordinate descent method is also plotted for comparison. Here, one epoch is equivalent to updating all coordinates once. Comparing to the serial method, we observe that the delay does affect the convergence speed, and the affect becomes weaker as m increases. Hence, Algorithm 1 can have nearly linear speedup when the number of blocks is large. In addition, we note that the stepsize setting of AsySCD is too conservative, and Algorithm 1 with stepsize set by the expected delay converges significantly faster. However, we observed that, in general, we could not take larger stepsize than that in (6.1a). Some divergence behaviors are observed when using stepsizes larger than that in (6.1a).
Convergence behaviors of Algorithm 1 for solving the LASSO problem (6.2) with the stepsize given in (6.1), and also the serial randomized coordinate descent method. The tested problem has \(10\,000\) samples and \(20\,000\) coordinates that are evenly partitioned into m blocks. It was simulated as running with 40 threads. We run 100 epochs for each experiments. (Color figure online)
Delay distribution behaviors of Algorithm 1 for solving NMF (6.3). It was running with 5, 10, 20, and 40 threads. (Color figure online)
6.3 Nonnegative Matrix Factorization (NMF)
This section presents the numerical results of applying Algorithm 1 for solving the NMF problem [37]
where \({\mathbf {Z}}\in \mathbb {R}^{M\times N}_+\) is a given nonnegative matrix. We generated \({\mathbf {Z}}={\mathbf {Z}}_L{\mathbf {Z}}_R^\top \) with the elements of \({\mathbf {Z}}_L\) and \({\mathbf {Z}}_R\) first drawn from the standard normal distribution and then projected into the nonnegative orthant. The size was fixed to \(M=N=10\,000\) and \(m=100\).
We treated one column of \({\mathbf {X}}\) or \({\mathbf {Y}}\) as one block coordinate, and during the iterations, every column of \({\mathbf {X}}\) was kept with unit norm. Therefore, the partial gradient Lipschitz constant equals one if one column of \({\mathbf {Y}}\) is selected to update and \(\Vert {\mathbf {y}}_{i_k}^k\Vert _2^2\) if the \(i_k\)th column of \({\mathbf {X}}\) is selected. Since \(\Vert {\mathbf {y}}_{i_k}^k\Vert _2^2\) could approach to zero, we set the Lipschitz constant to \(\max (0.001, \Vert {\mathbf {y}}_{i_k}^k\Vert _2^2)\). This modification can guarantee the whole sequence convergence of the coordinate descent method [38]. Due to nonconvexity, global optimality cannot be guaranteed. Thus, we set the starting point close to \({\mathbf {Z}}_L\) and \({\mathbf {Z}}_R\). Specifically, we let \({\mathbf {X}}^0={\mathbf {Z}}_L+0.5\varvec{\Xi }_L\) and \({\mathbf {Y}}^0={\mathbf {Z}}_R+0.5\varvec{\Xi }_R\) with the elements of \(\varvec{\Xi }_L\) and \(\varvec{\Xi }_R\) following the standard normal distribution. All methods used the same starting point.
Figure 3 shows the delay distribution behavior of Algorithm 1 for solving NMF. The observation is similar to Fig. 1. Figure 4 plots the convergence results of Algorithm 1 running with 1, 5, 10, 20 and 40 threads. From the results, we observe that Algorithm 1 scales up to 10 threads for the tested problem. Degenerated convergence is observed with 20 and 40 threads. This is mostly due to the following three reasons: (1) since the number of blocks is relatively small (\(m =200\)), as shown in (6.1a), using more threads leads to smaller stepsize, hence, slower convergence; (2) the gradient used for the current update is more staled when a relative large number of threads are used, which also leads to slow convergence; (3) high cache miss rates and false sharing also downgrade the speedup performance.
Convergence behaviors of Algorithm 1 for solving the NMF problem (6.3) with the stepsize set based on the expected delay. The size of the tested problem is \(M=N=10\,000\) and \(m=100\), i.e., 200 block coordinates, and the algorithm was tested with 1, 5, 10, 20, and 40 threads. (Color figure online)
7 Conclusions
We have analyzed the convergence of the async-BCU method for solving both convex and nonconvex problems in a probabilistic way. We showed that the algorithm is guaranteed to converge for smooth problems if the expected delay is finite and for nonsmooth problems if the variance of the delay is also finite. In addition, we established sublinear convergence of the method for weakly convex problems and linear convergence for strongly convex ones. The stepsize we obtained depends on certain expected quantities. Assuming the given \(p+1\) processors perform identically, we showed that the delay follows a Poisson distribution with parameter p and thus fully determined the stepsize. We have simulated the performance of the algorithm with our determined stepsize on solving LASSO and the nonnegative matrix factorization, and the numerical results validated our analysis.
8 Proofs of Lemmas
The following lemma is used in other proofs several times, and it is easy to verify.
Lemma A.1
For any scalar sequences \(\{a_{i,j}\}\) and \(\{b_i\}\), it holds that
8.1 Proof of Lemma 3.3
Proof
Following the proof of Theorem 1 in [12], we have
and
We first show the first inequality in (3.18). Note that (3.17) gives us
When \(t=0\), we have from (A.4) that \(\textstyle \Vert \nabla f({\mathbf {x}}^0)\Vert ^2-\textit{E}\Vert \nabla f({\mathbf {x}}^1)\Vert ^2\leqslant \frac{2 \eta L_r}{\sqrt{m}}\Vert \nabla f({\mathbf {x}}^0)\Vert ^2\leqslant (1+M_\rho )\frac{\eta L_r}{\sqrt{m}}\Vert \nabla f({\mathbf {x}}^0)\Vert ^2.\) Hence, \(\Vert \nabla f({\mathbf {x}}^0)\Vert ^2\leqslant \rho \textit{E}\Vert \nabla f({\mathbf {x}}^1)\Vert ^2\) from (A.6). Now we assume that \(\textit{E}\Vert \nabla f({\mathbf {x}}^t)\Vert ^2\leqslant \rho \textit{E}\Vert \nabla f({\mathbf {x}}^{t+1})\Vert ^2\) for all \(t\leqslant k-1\). For \(t=k\), it holds from (A.4) and the induction assumption that
Hence, we have \(\textit{E}\Vert \nabla f({\mathbf {x}}^k)\Vert ^2\leqslant \rho \textit{E}\Vert \nabla f({\mathbf {x}}^{k+1})\Vert ^2\) from (A.6). Therefore, we finish the induction step, and thus, the first inequality of (3.18) holds.
Next we show the second inequality of (3.18). Since (3.17) implies \(\textstyle \eta \leqslant \frac{\rho -1}{\frac{L_r}{\sqrt{m}}\left( 1+M_\rho +\frac{(\rho -1)M_\rho }{\rho (1+M_\rho )}\right) }\),
When \(t=0\), we have from (A.5) that
Hence, \(\textit{E}\Vert \nabla f({\mathbf {x}}^1)\Vert ^2\leqslant \rho \Vert \nabla f({\mathbf {x}}^0)\Vert ^2\) holds from (A.7). Assume \(\textit{E}\Vert \nabla f({\mathbf {x}}^{t+1})\Vert ^2\leqslant \rho \textit{E}\Vert \nabla f({\mathbf {x}}^t)\Vert ^2\) for all \(t\leqslant k-1\). It follows from (A.5) and the induction assumption that
Hence, from (A.7), \(\textit{E}\Vert \nabla f({\mathbf {x}}^{k+1})\Vert ^2\leqslant \rho \textit{E}\Vert \nabla f({\mathbf {x}}^k)\Vert ^2\) holds, and we complete the proof.
Change history
20 June 2018
A Correction to this paper has been published: https://doi.org/10.1007/s40305-018-0208-4
Notes
For the NMF problem, \(L_c\) cannot be determined in the beginning, so instead of using a uniform \(L_c\), we used the gradient Lipschitz constant adaptive to the iterate.
References
WhiteHouse: Big Data: Seizing Opportunities Preserving Values (2014)
Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)
Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B (Methodol.) 58, 267–288 (1996)
Zhou, H., Li, L., Zhu, H.: Tensor regression with applications in neuroimaging data analysis. J. Am. Stat. Assoc. 108(502), 540–552 (2013)
Zou, H., Hastie, T., Tibshirani, R.: Sparse principal component analysis. J. Comput. Graph. Stat. 15(2), 265–286 (2006)
Cichocki, A., Zdunek, R., Phan, A.H., Amari, Si: Nonnegative Matrix and Tensor Factorizations: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation. Wiley, London (2009)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)
Nemirovski, A., Juditsky, A., Lan, G., Shapiro, A.: Robust stochastic approximation approach to stochastic programming. SIAM J. Optim. 19(4), 1574–1609 (2009)
Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22(2), 341–362 (2012)
Dang, C.D., Lan, G.: Stochastic block mirror descent methods for nonsmooth and stochastic optimization. SIAM J. Optim. 25(2), 856–881 (2015)
Xu, Y., Yin, W.: Block stochastic gradient iteration for convex and nonconvex optimization. SIAM J. Optim. 25(3), 1686–1716 (2015)
Liu, J., Wright, S., Re, C., Bittorf, V., Sridhar, S.: An asynchronous parallel stochastic coordinate descent algorithm. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 469–477 (2014)
Peng, Z., Xu, Y., Yan, M., Yin, W.: Arock: an algorithmic framework for asynchronous parallel coordinate updates. SIAM J. Sci. Comput. 38(5), A2851–A2879 (2016)
Strikwerda, J.C.: A probabilistic analysis of asynchronous iteration. Linear Algebra Appl. 349(13), 125–154 (2002)
Hannah, R., Yin, W.: On unbounded delays in asynchronous parallel fixed-point algorithms. arXiv preprint arXiv:1609.04746 (2016)
Liu, J., Wright, S.J.: Asynchronous stochastic coordinate descent: parallelism and convergence properties. SIAM J. Optim. 25(1), 351–376 (2015)
Cannelli, L., Facchinei, F., Kungurtsev, V., Scutari, G.: Asynchronous parallel algorithms for nonconvex big-data optimization: model and convergence. arXiv preprint arXiv:1607.04818 (2016)
Davis, D.: The asynchronous PALM algorithm for nonsmooth nonconvex problems. arXiv preprint arXiv:1604.00526 (2016)
Mokhtai, A., Koppel, A., Ribeiro, A.: A class of parallel doubly stochastic algorithms for large-scale learning. arXiv preprint arXiv:1606.04991 (2016)
Recht, B., Re, C., Wright, S., Niu, F.: Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In: Advances in Neural Information Processing Systems, pp. 693–701 (2011)
Hildreth, C.: A quadratic programming procedure. Naval Res. Logist. Q. 4(1), 79–85 (1957)
Grippo, L., Sciandrone, M.: On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper. Res. Lett. 26(3), 127–136 (2000)
Luo, Z.Q., Tseng, P.: On the convergence of the coordinate descent method for convex differentiable minimization. J. Optim. Theory Appl. 72(1), 7–35 (1992)
Tseng, P.: Convergence of a block coordinate descent method for nondifferentiable minimization. J. Optim. Theory Appl. 109(3), 475–494 (2001)
Hong, M., Wang, X., Razaviyayn, M., Luo, Z.Q.: Iteration complexity analysis of block coordinate descent methods. Math. Program. 163(1–2), 85–114 (2017)
Tseng, P., Yun, S.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117(1–2), 387–423 (2009)
Lu, Z., Xiao, L.: On the complexity analysis of randomized block-coordinate descent methods. Math. Program. 152(1–2), 615–642 (2015)
Richtárik, P., Takáč, M.: Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function. Math. Program. 144(1–2), 1–38 (2014)
Rosenfeld, J.L.: A case study in programming for parallel-processors. Commun. ACM 12(12), 645–655 (1969)
Chazan, D., Miranker, W.: Chaotic relaxation. Linear Algebra Appl. 2(2), 199–222 (1969)
Bertsekas, D.P.: Distributed asynchronous computation of fixed points. Math. Program. 27(1), 107–120 (1983)
Tseng, P., Bertsekas, D.P., Tsitsiklis, J.N.: Partially asynchronous, parallel algorithms for network flow and other problems. SIAM J. Control Optim. 28(3), 678–710 (1990)
Frommer, A., Szyld, D.B.: On asynchronous iterations. J. Comput. Appl. Math. 123(1), 201–216 (2000)
Gut, A.: A Graduate Course: A Graduate Course. Springer, Berlin (2006)
Lai, M.J., Yin, W.: Augmented \(\ell _1\) and nuclear-norm models with a globally linearly convergent algorithm. SIAM J. Imaging Sci. 6(2), 1059–1091 (2013)
Rockafellar, R.T., Wets, R.J.B.: Variational Analysis, vol. 317. Springer, Berlin (2009)
Paatero, P., Tapper, U.: Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5(2), 111–126 (1994)
Xu, Y., Yin, W.: A globally convergent algorithm for nonconvex optimization based on block coordinate update. J. Sci. Comput. 72(2), 700–734 (2017)
Author information
Authors and Affiliations
Corresponding author
Additional information
The original version of this article was revised due to a retrospective Open Access order.
This paper is dedicated to Professor Yin-Yu Ye in celebration of his 70th birthday.
This project was supported by the National Science Foundation (EAGER ECCS-1462397, DMS-1621798, and DMS-1719549).
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Peng, Z., Xu, Y., Yan, M. et al. On the Convergence of Asynchronous Parallel Iteration with Unbounded Delays. J. Oper. Res. Soc. China 7, 5–42 (2019). https://doi.org/10.1007/s40305-017-0183-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40305-017-0183-1