Parallel coordinate descent methods for big data optimization

1.1 Big data optimization

Recently there has been a surge in interest in the design of algorithms suitable for solving convex optimization problems with a huge number of variables [12, 15]. Indeed, the size of problems arising in fields such as machine learning [1], network analysis [29], PDEs [27], truss topology design [16] and compressed sensing [5] usually grows with our capacity to solve them, and is projected to grow dramatically in the next decade. In fact, much of computational science is currently facing the “big data” challenge, and this work is aimed at developing optimization algorithms suitable for the task.

1.2 Coordinate descent methods

Coordinate descent methods (CDM) are one of the most successful classes of algorithms in the big data optimization domain. Broadly speaking, CDMs are based on the strategy of updating a single coordinate (or a single block of coordinates) of the vector of variables at each iteration. This often drastically reduces memory requirements as well as the arithmetic complexity of a single iteration, making the methods easily implementable and scalable. In certain applications, a single iteration can amount to as few as 4 multiplications and additions only [16]! On the other hand, many more iterations are necessary for convergence than it is usual for classical gradient methods. Indeed, the number of iterations a CDM requires to solve a smooth convex optimization problem is \(O(\tfrac{n \tilde{L} R^2}{\epsilon })\), where \(\epsilon \) is the error tolerance, \(n\) is the number variables (or blocks of variables), \(\tilde{L}\) is the average of the Lipschitz constants of the gradient of the objective function associated with the variables (blocks of variables) and \(R\) is the distance from the starting iterate to the set of optimal solutions. On balance, as observed by numerous authors, serial CDMs are much more efficient for big data optimization problems than most other competing approaches, such as gradient methods [10, 16].

1.3 Parallelization

We wish to point out that for truly huge-scale problems it is absolutely necessary to parallelize. This is in line with the rise and ever increasing availability of high performance computing systems built around multi-core processors, GPU-accelerators and computer clusters, the success of which is rooted in massive parallelization. This simple observation, combined with the remarkable scalability of serial CDMs, leads to our belief that the study of parallel coordinate descent methods (PCDMs) is a very timely topic.

1.4 Research idea

The work presented in this paper was motivated by the desire to answer the following question:

Our starting point was the following simple observation. Assume that we wish to minimize a separable function \(F\) of \(n\) variables (i.e., a function that can be written as a sum of \(n\) functions each of which depends on a single variable only). For simplicity, in this thought experiment, assume that there are no constraints. Clearly, the problem of minimizing \(F\) can be trivially decomposed into \(n\) independent univariate problems. Now, if we have \(n\) processors/threads/cores, each assigned with the task of solving one of these problems, the number of parallel iterations should not depend on the dimension of the problem. In other words, we get an \(n\)-times speedup compared to the situation with a single processor only. Any parallel algorithm of this type can be viewed as a parallel coordinate descent method. Hence, PCDM with \(n\) processors should be \(n\)-times faster than a serial one. If \(\tau \) processors are used instead, where \(1\le \tau \le n\), one would expect a \(\tau \)-times speedup.

By extension, one would perhaps expect that optimization problems with objective functions which are “close to being separable” would also be amenable to acceleration by parallelization, where the acceleration factor \(\tau \) would be reduced with the reduction of the “degree of separability”. One of the main messages of this paper is an affirmative answer to this. Moreover, we give explicit and simple formulae for the speedup factors.

As it turns out, and as we discuss later in this section, many real-world big data optimization problems are, quite naturally, “close to being separable”. We believe that this means that PCDMs is a very promising class of algorithms for structured big data optimization problems.

1.5 Minimizing a partially separable composite objective

1.6 Examples of partially separable functions

1.7 Brief literature review

Several papers were written recently studying the iteration complexity of serial CDMs of various flavours and in various settings. We will only provide a brief summary here, for a more detailed account we refer the reader to [15].

Classical CDMs update the coordinates in a cyclic order; the first attempt at analyzing the complexity of such a method is due to [21]. Stochastic/randomized CDMs, that is, methods where the coordinate to be updated is chosen randomly, were first analyzed for quadratic objectives [4, 24], later independently generalized to \(L_1\)-regularized problems [23] and smooth block-structured problems [10], and finally unified and refined in [15, 19]. The problems considered in the above papers are either unconstrained or have (block) separable constraints. Recently, randomized CDMs were developed for problems with linearly coupled constraints [7, 8].

A greedy CDM for \(L_1\)-regularized problems was first analyzed in [16]; more work on this topic include [2, 5]. A CDM with inexact updates was first proposed and analyzed in [26]. Partially separable problems were independently studied in [13], where an asynchronous parallel stochastic gradient algorithm was developed to solve them.

When writing this paper, the authors were aware only of the parallel CDM proposed and analyzed in [1]. Several papers on the topic appeared around the time this paper was finalized or after [6, 14, 22, 22, 28]. Further papers on various aspects of the topic of parallel CDMs, building on the work in this paper, include [3, 17, 18, 25].

1.8 Contents

We start in Sect. 2 by describing the block structure of the problem, establishing notation and detailing assumptions. Subsequently we propose and comment in detail on two parallel coordinate descent methods. In Sect. 3 we summarize the main contributions of this paper. In Sect. 4 we deal with issues related to the selection of the blocks to be updated in each iteration. It will involve the development of some elementary random set theory. Sections 5 and 6 deal with issues related to the computation of the update to the selected blocks and develop a theory of Expected Separable Overapproximation (ESO), which is a novel tool we propose for the analysis of our algorithms. In Sect. 7 we analyze the iteration complexity of our methods and finally, Sect. 8 reports on promising computational results. For instance, we conduct an experiment with a big data (cca 350GB) LASSO problem with a billion variables. We are able to solve the problem using one of our methods on a large memory machine with 24 cores in 2 h, pushing the difference between the objective value at the starting iterate and the optimal point from \(10^{22}\) down to \(10^{-14}\). We also conduct experiments on real data problems coming from machine learning.

2.1 Block structure, notation and assumptions

The block structure² of (1) is given by a decomposition of \(\mathbf {R}^N\) into \(n\) subspaces as follows. Let \(U\in \mathbf {R}^{N\times N}\) be a column permutation³ of the \(N\times N\) identity matrix and further let \(U= [U_1,U_2,\ldots ,U_n]\) be a decomposition of \(U\) into \(n\) submatrices, with \(U_i\) being of size \(N\times N_i\), where \(\sum _i N_i = N\).

In view of the above proposition, from now on we write \(x^{(i)}\mathop {=}\limits ^{\text {def}}U_i^T x \in \mathbf {R}^{N_i}\), and refer to \(x^{(i)}\) as the \(i\) th block of \(x\). The definition of partial separability in the introduction is with respect to these blocks. For simplicity, we will sometimes write \(x = (x^{(1)},\ldots ,x^{(n)})\).

2.2 Algorithms

In this paper we develop and study two generic parallel coordinate descent methods. The main method is PCDM1; PCDM2 is its “regularized” version which explicitly enforces monotonicity. As we will see, both of these methods come in many variations, depending on how Step  3 is performed.

Open image in new window

Open image in new window

Let us comment on the individual steps of the two methods.

Step 3. At the beginning of iteration \(k\) we pick a random set (\(S_k\)) of blocks to be updated (in parallel) during that iteration. The set \(S_k\) is a realization of a random set-valued mapping \(\hat{S}\) with values in \(2^{[n]}\) or, more precisely, it the sets \(S_k\) are iid random sets with the distribution of \(\hat{S}\). For brevity, in this paper we refer to such a mapping by the name sampling. We limit our attention to uniform samplings, i.e., random sets having the following property: \(\mathbf {P}(i \in \hat{S})\) is independent of \(i\). That is, the probability that a block gets selected is the same for all blocks. Although we give an iteration complexity result covering all such samplings (provided that each block has a chance to be updated, i.e., \(\mathbf {P}(i \in \hat{S}) > 0\)), there are interesting subclasses of uniform samplings (such as doubly uniform and nonoverlapping uniform samplings; see Sect. 4) for which we give better results.

Section 6 is devoted to the computation of \(\beta \) and \(w\) for partially separable \(f\) and various special classes of uniform samplings \(\hat{S}\). Typically we will have \(w_i=L_i\), while \(\beta \) will depend on easily computable properties of \(f\) and \(\hat{S}\). For example, if \(\hat{S}\) is chosen as a subset of \({[n]}\) of cardinality \(\tau \), with each subset chosen with the same probability (we say that \(\hat{S}\) is \(\tau \)-nice) then, assuming \(n>1\), we may choose \(w=L\) and \(\beta =1+ \tfrac{(\omega -1)(\tau -1)}{n-1}\), where \(\omega \) is the degree of partial separability of \(f\). More generally, if \(\hat{S}\) is any uniform sampling with the property \(|\hat{S}|=\tau \) with probability 1, then we may choose \(w=L\) and \(\beta =\min \{\omega ,\tau \}\). Note that in both cases \(w=L\) and that the latter \(\beta \) is always larger than (or equal to) the former one. This means, as we will see in Sect. 7, that we can give better complexity results for the former, more specialized, sampling. We analyze several more options for \(\hat{S}\) than the two just described, and compute parameters \(\beta \) and \(w\) that should be used with them (for a summary, see Table 4).

Step 5. The reason why, besides PCDM1, we also consider PCDM2, is the following: in some situations we are not able to analyze the iteration complexity of PCDM1 (non-strongly-convex \(F\) where monotonicity of the method is not guaranteed by other means than by directly enforcing it by inclusion of Step 5). Let us remark that this issue arises for general \(\Omega \) only. It does not exist for \(\Omega =0\), \(\Omega (\cdot ) = \lambda \Vert \cdot \Vert _1\) and for \(\Omega \) encoding simple constraints on individual blocks; in these cases one does not need to consider PCDM2. Even in the case of general \(\Omega \) we sometimes get monotonicity for free, in which case there is no need to enforce it. Let us stress, however, that we do not recommend implementing PCDM2 as this would introduce too much overhead; in our experience PCDM1 works well even in cases when we can only analyze PCDM2.

4.1 Uniform, doubly uniform and nonoverlapping uniform samplings

A sampling is proper if \(p_i>0\) for all blocks \(i\). That is, from the perspective of PCDM, under a proper sampling each block gets updated with a positive probability at each iteration. Clearly, PCDM can not converge for a sampling that is not proper. A sampling \(\hat{S}\) is uniform if all blocks get updated with the same probability, i.e., if \(p_i=p_j\) for all \(i,j\). We show in (33) that, necessarily, \(p_i = \tfrac{\mathbf {E}[|\hat{S}|]}{n}\). Further, we say \(\hat{S}\) is nil if \(\mathbf {P}(\emptyset ) = 1\). Note that a uniform sampling is proper if and only if it is not nil.

The following simple result says that the intersection between the class of DU and NU samplings is very thin. A sampling is called vacuous if \(\mathbf {P}(\emptyset )>0\).

4.2 Technical results

The consequences are summarized in the next theorem and the discussion that follows.

5.1 Deterministic separable overapproximation (DSO) of partially separable functions

The following theorem will be useful in deriving ESO for uniform samplings (Sect. 6.1) and nonoverlapping uniform samplings (Sect. 6.2). It will also be useful in establishing monotonicity of some ESOs (Theorems 10 and 11).

6.1 Uniform samplings

The above lemma will now be used to establish ESO for arbitrary (proper) uniform samplings.

Besides establishing an ESO result, we have just shown that, in the case of \(\tau \)-uniform samplings with a conservative estimate for \(\beta \), PCDM1 is monotonic, i.e., \(F(x_{k+1})\le F(x_k)\). In particular, PCDM1 and PCDM2 coincide. We call the estimate \(\beta = \min \{\omega ,\tau \}\) “conservative” because it can be improved (made smaller) in special cases; e.g., for the \(\tau \)-nice sampling. Indeed, Theorem 12 establishes an ESO for the \(\tau \)-nice sampling with the same \(w\) (\(w=L\)), but with \(\beta = 1 + \tfrac{(\omega -1)(\tau -1)}{n-1}\), which is better (and can be much better than) \(\min \{\omega ,\tau \}\). Other things equal, smaller \(\beta \) directly translates into better complexity. The price for the small \(\beta \) in the case of the \(\tau \)-nice sampling is the loss of monotonicity. This is not a problem for strongly convex objective, but for merely convex objective this is an issue as the analysis techniques we developed are only applicable to the monotonic method PCDM2 (see Theorem 17).

6.2 Nonoverlapping uniform samplings

6.3 Nice samplings

In this section we establish an ESO for nice samplings.

6.4 Doubly uniform samplings

We are now ready, using a bootstrapping argument, to formulate and prove a result covering all doubly uniform samplings.

Note that Theorem 13 reduces to that of Theorem 12 in the special case of a nice sampling, and gives the same result as Theorem 11 in the case of the serial and fully parallel samplings.

7.1 Iteration complexity: convex case

The following lemma will be used to finish off the proof of the complexity result of this section.

This lemma was recently extended in [26] so as to aid the analysis of a serial coordinate descent method with inexact updates, i.e., with \(h(x)\) chosen as an approximate rather than exact minimizer of \(H_{1,L}(x,\cdot )\) (see (17)). While in this paper we deal with exact updates only, the results can be extended to the inexact case.

The speedup of the serial sampling (i.e., of the algorithm based on it) is 1 as we are comparing it to itself. On the other end of the spectrum is the fully parallel sampling with a speedup of \(\tfrac{n}{\omega }\). If the degree of partial separability is small, then this factor will be high — especially so if \(n\) is huge, which is the domain we are interested in. This provides an affirmative answer to the research question stated in italics in the introduction.

Let us now look at the speedup factor in the case of a \(\tau \)-nice sampling. Letting \(r= \tfrac{\omega -1}{\max (1,n-1)} \in [0,1]\) (degree of partial separability normalized), the speedup factor can be written as

$$\begin{aligned} s(r) = \frac{\tau }{1+ r(\tau -1)}. \end{aligned}$$

Note that as long as \(r\le \tfrac{k-1}{\tau -1}\approx \tfrac{k}{\tau }\), the speedup factor will be at least \(\tfrac{\tau }{k}\). Also note that \(\max \{1,\tfrac{\tau }{\omega }\} \le s(r)\le \min \{\tau , \tfrac{n}{\omega }\}\). Finally, if a speedup of at least \(s\) is desired, where \(s\in [0,\tfrac{n}{\omega }]\), one needs to use at least \(\frac{1-r}{1/s-r}\) processors. For illustration, in Fig. 1 we plotted \(s(r)\) for a few values of \(\tau \). Note that for small values of \(\tau \), the speedup is significant and can be as large as the number of processors (in the separable case). We wish to stress that in many applications \(\omega \) will be a constant independent of \(n\), which means that \(r\) will indeed be very small in the huge-scale optimization setting.

Open image in new window

7.2 Iteration complexity: strongly convex case

In this section we assume that \(F\) is strongly convex with respect to the norm \(\Vert \cdot \Vert _w\) and show that \(F(x_k)\) converges to \(F^*\) linearly, with high probability.

8.1 A LASSO problem with 1 billion variables

In this experiment we solve a single randomly generated huge-scale LASSO instance, i.e., (1) with

$$\begin{aligned} f(x)=\tfrac{1}{2}\Vert Ax-b\Vert _2^2, \qquad \Omega (x) = \Vert x\Vert _1, \end{aligned}$$

where \(A=[a_1,\ldots ,a_n]\) has \(2\times 10^9\) rows and \(N=n=10^9\) columns. We generated the problem using a modified primal-dual generator [15] enabling us to choose the optimal solution \(x^*\) (and hence, indirectly, \(F^*\)) and thus to control its cardinality \(\Vert x^*\Vert _0\), as well as the sparsity level of \(A\). In particular, we made the following choices: \(\Vert x^*\Vert _0 = 10^5\), each column of \(A\) has exactly 20 nonzeros and the maximum cardinality of a row of \(A\) is \(\omega = 35\) (the degree of partial separability of \(f\)). The histogram of cardinalities is displayed in Fig. 2.

Open image in new window

We solved the problem using PCDM1 with \(\tau \)-nice sampling \(\hat{S}\), \(\beta = 1+ \tfrac{(\omega -1)(\tau -1)}{n-1}\) and \(w=L=(\Vert a_1\Vert ^2_2,\ldots ,\Vert a_n\Vert _2^2)\), for \(\tau =1,2,4,8,16, 24\), on a single large-memory computer utilizing \(\tau \) of its 24 cores. The problem description took around 350GB of memory space. In fact, in our implementation we departed from the just described setup in two ways. First, we implemented an asynchronous version of the method; i.e., one in which cores do not wait for others to update the current iterate within an iteration before reading \(x_{k+1}\) and proceeding to another update step. Instead, each core reads the current iterate whenever it is ready with the previous update step and applies the new update as soon as it is computed. Second, as mentioned in Sect. 4, the \(\tau \)-independent sampling is for \(\tau \ll n\) a very good approximation of the \(\tau \)-nice sampling. We therefore allowed each processor to pick a block uniformly at random, independently from the other processors.

Choice of the first column of Table 6 In Table 6 we show the development of the gap \(F(x_k)-F^*\) as well as the elapsed time. The choice and meaning of the first column of the table, \(\tfrac{\tau k}{n}\), needs some commentary. Note that exactly \(\tau k\) coordinate updates are performed after \(k\) iterations. Hence, the first column denotes the total number of coordinate updates normalized by the number of coordinates \(n\). As an example, let \(\tau _1=1\) and \(\tau _2=24\). Then if the serial method is run for \(k_1=24\) iterations and the parallel one for \(k_2=1\) iteration, both methods would have updated the same number (\(\tau _1 k_1 = \tau _2 k_2 = 24\)) of coordinates; that is, they would “be” in the same row of Table 6. In summary, each row of the table represents, in the sense described above, the “same amount of work done” for each choice of \(\tau \). We have highlighted in bold elapsed time after 13 and 26 passes over data for \(\tau = 1, 2, 4, 8, 16.\) Note that for any fixed \(\tau \), the elapsed time has approximately doubled, as one would expect. More importantly, note that we can clearly observe close to linear speedup in the number of processors \(\tau \).

8.2 Theory versus reality

In our second experiment we demonstrate numerically that our parallelization speedup estimates are in some sense tight. For this purpose it is not necessary to reach for complicated problems and high dimensions; we hence minimize the function \(\frac{1}{2} \Vert Ax-b\Vert _2^2\) with \(A\in \mathbf {R}^{3000 \times 1000}\). Matrix \(A\) was generated so that its every row contains exactly \(\omega \) non-zero values all of which are equal (recall the construction in point 3 at the end of Sect. 5.1).

Open image in new window

We generated 4 matrices with \(\omega =5, 10, 50\) and \( 100\) and measured the number of iterations needed for PCDM1 used with \(\tau \)-nice sampling to get within \(\epsilon = 10^{-6}\) of the optimal value. The experiment was done for a range of values of \(\tau \) (between 1 core and 1000 cores).

The solid lines in Fig. 4 present the theoretical speedup factor for the \(\tau \)-nice sampling, as presented in Table 5. The markers in each case correspond to empirical speedup factor defined as

$$\begin{aligned} \frac{\# \ \text{ of } \text{ iterations } \text{ till }\ \epsilon \text{-solution } \text{ is } \text{ found } \text{ by } \text{ PCDM1 } \text{ used } \text{ with } \text{ serial } \text{ sampling }}{\# \ \text{ of } \text{ iterations } \text{ till }\ \epsilon \text{-solution } \text{ is } \text{ found } \text{ by } \text{ PCDM1 } \text{ used } \text{ with }\ \tau \text{-nice } \text{ sampling }}. \end{aligned}$$

As can be seen in Fig. 4, the match between theoretical prediction and reality is remarkable! A partial explanation of this phenomenon lies in the fact that we have carefully designed the problem so as to ensure that the degree of partial separability is equal to the Lipschitz constant \(\sigma \) of \(\nabla f\) (i.e., that it is not a gross overestimation of it; see Sect. 5.1). This fact is useful since it is possible to prove complexity results with \(\omega \) replaced by \(\sigma \). However, this answer is far from satisfying, and a deeper understanding of the phenomenon remains an open problem.

Open image in new window

8.3 Training linear SVMs with bad data for PCDM

We consider the rcv1.binary dataset.¹² The training data has \(n = 677,399\) examples, \(d= 47,236\) features, \(49,556,258\) nonzero elements and requires cca 1GB of RAM for storage. Hence, this is a small-scale problem. The degree of partial separability of \(f\) is \(\omega = 291,516\) (i.e., the maximum number of examples sharing a given feature). This is a very large number relative to \(n\), and hence our theory would predict rather bad behavior for PCDM. We use PCDM1 with \(\tau \)-nice sampling ( approximating it by \(\tau \)-independent sampling for added efficiency) with \(\beta \) following Theorem 12: \(\beta =1+ \frac{(\tau -1)(\omega -1)}{n-1}\).

The results of our experiments are summarized in Fig. 5. Each column corresponds to a different level of regularization: \(\lambda \in \{1,10^{-3},10^{-5}\}\). The rows show the (1) duality gap, (2) dual suboptimality, (3) train error and (4) test error; each for 1,4 and 16 processors (\(\tau = 1,4,16\)). Observe that the plots in the first two rows are nearly identical; which means that the method is able to solve the primal problem at about the same speed as it can solve the dual problem.¹³

Observe also that in all cases, duality gap of around \(0.01\) is sufficient for training as training error (classification performance of the SVM on the train data) does not decrease further after this point. Also observe the effect of \(\lambda \) on training accuracy: accuracy increases from about \(92\,\%\) for \(\lambda =1\), through \(95.3\,\%\) for \(\lambda =10^{-3}\) to above \(97.8\,\%\) with \(\lambda =10^{-5}\). In our case, choosing smaller \(\lambda \) does not lead to overfitting; the test error on test dataset (# features =677,399, # examples = 20,242) increases as \(\lambda \) decreases, quickly reaching about \(95\,\%\) (after 2 seconds of training) for \(\lambda =0.001\) and for the smallest \(\lambda \) going beyond \(97\,\%\).

Note that PCDM with \(\tau =16\) is about 2.5\(\times \) faster than PCDM with \(\tau =1\). This is much less than linear speedup, but is fully in line with our theoretical predictions. Indeed, for \(\tau =16\) we get \(\beta = 7.46\). Consulting Table 5, we see that the theory says that with \(\tau =16\) processors we should expect the parallelization speedup to be \(PSF= \tau /\beta = 2.15 \).

Open image in new window

8.4 \(L2\)-regularized logistic regression with good data for PCDM

Figure 6 depicts the evolution of the regularized loss \(F(x_k)\) throughout the run of the 4 versions of PCDM (starting with \(x_0\) for which \(F(x_0) = 13,352,855\)). Each marker corresponds to approximately \(n/3\) coordinate updates (\(n\) coordinate updates will be referred to as an “epoch”). Observe that as more processors are used, it takes less time to achieve any given level of loss; nearly in exact proportion to the increase in the number of processors.

Open image in new window

Let us remark that the training and testing accuracy stopped increasing after having trained the classifier for 1 epoch; they were \(86.07\) and \(88.77\,\%\), respectively. This is in agreement with the common wisdom in machine learning that training beyond a single pass through the data rarely improves testing accuracy (as it may lead to overfitting). This is also the reason behind the success of light-touch methods, such as coordinate descent and stochastic gradient descent, in machine learning applications.