Mathematical Programming

, Volume 144, Issue 1, pp 1–38

Iteration complexity of randomized block-coordinate descent methods for minimizing a composite function

Full Length Paper Series A

DOI: 10.1007/s10107-012-0614-z

Cite this article as:
Richtárik, P. & Takáč, M. Math. Program. (2014) 144: 1. doi:10.1007/s10107-012-0614-z

Abstract

In this paper we develop a randomized block-coordinate descent method for minimizing the sum of a smooth and a simple nonsmooth block-separable convex function and prove that it obtains an \(\varepsilon \)-accurate solution with probability at least \(1-\rho \) in at most \(O((n/\varepsilon ) \log (1/\rho ))\) iterations, where \(n\) is the number of blocks. This extends recent results of Nesterov (SIAM J Optim 22(2): 341–362, 2012), which cover the smooth case, to composite minimization, while at the same time improving the complexity by the factor of 4 and removing \(\varepsilon \) from the logarithmic term. More importantly, in contrast with the aforementioned work in which the author achieves the results by applying the method to a regularized version of the objective function with an unknown scaling factor, we show that this is not necessary, thus achieving first true iteration complexity bounds. For strongly convex functions the method converges linearly. In the smooth case we also allow for arbitrary probability vectors and non-Euclidean norms. Finally, we demonstrate numerically that the algorithm is able to solve huge-scale \(\ell _1\)-regularized least squares problems with a billion variables.

Keywords

Block coordinate descentHuge-scale optimization Composite minimizationIteration complexityConvex optimizationLASSO Sparse regressionGradient descent Coordinate relaxationGauss–Seidel method

Mathematics Subject Classification (2000)

65K0590C0590C0690C25

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2012

Authors and Affiliations

  1. 1.School of MathematicsUniversity of EdinburghEdinburghUK