Templates for convex cone problems with applications to sparse signal recovery

  • Stephen R. Becker
  • Emmanuel J. Candès
  • Michael C. Grant
Full Length Paper

Abstract

This paper develops a general framework for solving a variety of convex cone problems that frequently arise in signal processing, machine learning, statistics, and other fields. The approach works as follows: first, determine a conic formulation of the problem; second, determine its dual; third, apply smoothing; and fourth, solve using an optimal first-order method. A merit of this approach is its flexibility: for example, all compressed sensing problems can be solved via this approach. These include models with objective functionals such as the total-variation norm, ||Wx||1 where W is arbitrary, or a combination thereof. In addition, the paper introduces a number of technical contributions such as a novel continuation scheme and a novel approach for controlling the step size, and applies results showing that the smooth and unsmoothed problems are sometimes formally equivalent. Combined with our framework, these lead to novel, stable and computationally efficient algorithms. For instance, our general implementation is competitive with state-of-the-art methods for solving intensively studied problems such as the LASSO. Further, numerical experiments show that one can solve the Dantzig selector problem, for which no efficient large-scale solvers exist, in a few hundred iterations. Finally, the paper is accompanied with a software release. This software is not a single, monolithic solver; rather, it is a suite of programs and routines designed to serve as building blocks for constructing complete algorithms.

Keywords

Optimal first-order methods Nesterov’s accelerated descent algorithms Proximal algorithms Conic duality Smoothing by conjugation The Dantzig selector The LASSO Nuclear-norm minimization 

Mathematics Subject Classification (2000)

90C05 90C06 90C25 62J077 

References

  1. 1.
    Afonso, M.V., Bioucas-Dias, J.M., Figueiredo, M.A.T.: An augmented Lagrangian approach to the constrained optimization formulation of imaging inverse problems. IEEE Trans. Image Process. 19(11), (2010). doi:10.1109/TIP.2010.2076294
  2. 2.
    Auslender A., Teboulle M.: Interior gradient and proximal methods for convex and conic optimization. SIAM J. Optim. 16(3), 697–725 (2006)MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Beck A., Teboulle M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: Convex Optimization in Signal Processing and Communications. Gradient-Based Algorithms with Applications in Signal Recovery Problems. Cambridge University Press (2010)Google Scholar
  5. 5.
    Becker S., Bobin J., Candès E.J.: NESTA: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4(1), 1–39 (2011)MathSciNetMATHCrossRefGoogle Scholar
  6. 6.
    Becker, S., Candès, E.J., Grant, M.: Templates for first-order conic solvers user guide. Technical report (2010). Preprint. http://tfocs.stanford.edu
  7. 7.
    van den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31(2), 890 (2009). doi:10.1137/080714488 . http://link.aip.org/link/SJOCE3/v31/i2/p890/s1&Agg=doi
  8. 8.
    Bertsekas D.P., Nedić A., Ozdaglar A.E.: Convex Analysis and Optimization. Athena Scientific, Cambridge (2003)MATHGoogle Scholar
  9. 9.
    Boyd D., vandenberghe L.: Convex Optimization. Cambridge University Press, Cambridge (2004)MATHGoogle Scholar
  10. 10.
    Cai J.F., Candès E.J., Shen Z.: A singular value thresholding algorithm for matrix completion. SIAM J. Optim. 20, 1956–1982 (2010)MathSciNetMATHCrossRefGoogle Scholar
  11. 11.
    Candès, E.J., Eldar, Y.C., Needell, D.: Compressed sensing with coherent and redundant dictionaries. Tech. rep. (2010). Preprint available at http://arxiv.org/abs/1005.2613
  12. 12.
    Candès E.J., Guo F.: New multiscale transforms, minimum total-variation synthesis: applications to edge-preserving image reconstruction. Signal Process. 82(11), 1519–1543 (2002)MATHCrossRefGoogle Scholar
  13. 13.
    Candès, E.J., Plan, Y.: Tight oracle bounds for low-rank matrix recovery from a minimal number of random measurements. In: CoRR, abs/1001.0339 (2010)Google Scholar
  14. 14.
    Candès E.J., Recht B.: Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009)MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Candès, E.J., Romberg, J.K.: Practical signal recovery from random projections. In: SPIE Conference on Computational Imaging, pp. 76–86 (2005)Google Scholar
  16. 16.
    Candès, E.J., Romberg, J.K.: 1-magic. Technical report, Caltech (2007). http://www.acm.caltech.edu/l1magic/
  17. 17.
    Candès E.J., Tao T.: The Dantzig selector: statistical estimation when p is much larger than n. Ann. Stat. 35(6), 2313–2351 (2007)MATHCrossRefGoogle Scholar
  18. 18.
    Candès E.J., Tao T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Trans. Inf. Theory 56(5), 2053–2080 (2010)CrossRefGoogle Scholar
  19. 19.
    Candès E.J., Wakin M.B., Boyd S.P.: Enhancing sparsity by reweighted 1 minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Chambolle A., Pock T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2010)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Ciarlet P.G.: Introduction to Numerical Linear Algebra and Optimisation. Cambridge University Press, Cambridge (1989)Google Scholar
  22. 22.
    Combettes P.L., Dũng D., Vũ B.C.: Dualization of signal recovery problems. Set-Valued Var. Anal. 18, 373–404 (2010)MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Combettes P.L., Pesquet J.C.: A Douglas-Rachford splitting approach to nonsmooth convex variational signal recovery. IEEE J. Sel. Topics Signal Process. 1(4), 564–574 (2007)CrossRefGoogle Scholar
  24. 24.
    Combettes P.L., Wajs V.R.: Signal recovery by proximal forward-backward splitting. SIAM Multiscale Model. Simul. 4(4), 1168–1200 (2005)MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    Donoho D.L., Tsaig Y.: Fast solution of 1 minimization problems when the solution may be sparse. IEEE Trans. Inform. Theory 54(11), 4789–4812 (2008)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Efron B., Hastie T., Johnstone I., Tibshirani R.: Least angle regression. Ann. Stat. 32(2), 407–499 (2004)MathSciNetMATHCrossRefGoogle Scholar
  27. 27.
    Elad M., Milanfar P., Rubinstein R.: Analysis versus synthesis in signal priors. Inverse Problems 23, 947–968 (2007)MathSciNetMATHCrossRefGoogle Scholar
  28. 28.
    Figueiredo M.A.T., Nowak R., Wright S.J.: Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007)CrossRefGoogle Scholar
  29. 29.
    Friedlander, M.P., Tseng, P.: Exact regularization of convex programs. SIAM J. Optim. 18(4), 1326–1350 (2007). doi:10.1137/060675320 . http://link.aip.org/link/?SJE/18/1326/1 Google Scholar
  30. 30.
    Friedman J., Hastie T., Tibshirani R.: Regularization paths for generalized linear models via coordinate descent. J. Stat. Softw. 33(1), 1–22 (2010)Google Scholar
  31. 31.
    Fukushima M., Mine H.: A generalized proximal point algorithm for certain non-convex minimization problems. Int. J. Syst. Sci. 12(8), 989–1000 (1981)MathSciNetMATHCrossRefGoogle Scholar
  32. 32.
    Grant, M., Boyd, S.: CVX: Matlab software for disciplined convex programming, version 1.21. http://cvxr.com/cvx (2010)
  33. 33.
    Gross, D.: Recovering low-rank matrices from few coefficients in any basis. In: CoRR, abs/0910.1879 (2009)Google Scholar
  34. 34.
    Gu, M., Lim, L.H., Wu, C.J.: PARNES: a rapidly convergent algorithm for accurate recovery of sparse and approximately sparse signals. Technical report (2009). Preprint http://arxiv.org/abs/0911.0492
  35. 35.
    Güler, O.: New proximal point algorithms for convex minimization. SIAM J. Optim. 2(4), 649–664 (1992). doi:10.1137/0802032 . http://link.aip.org/link/?SJE/2/649/1 Google Scholar
  36. 36.
    Hale E.T., Yin W., Zhang Y.: Fixed-point continuation for 1-minimization: methodology and convergence. SIAM J. Optim. 19(3), 1107–1130 (2008)MathSciNetMATHCrossRefGoogle Scholar
  37. 37.
    Hiriart-Urruty J.B., Lemaréchal C.: Convex Analysis and Minimization Algorithms, vols. I and II. Springer, Berlin (1993)Google Scholar
  38. 38.
    James G., Radchenko P., Lv J.: DASSO: Connections Between the Dantzig Selector and Lasso. J. R. Stat. Soc. B 71, 127–142 (2009)MathSciNetMATHCrossRefGoogle Scholar
  39. 39.
    Koh, K., Kim, S.J., Boyd, S.P.: Solver for l1-regularized least squares problems. Technical report, Stanford University. http://www.stanford.edu/~boyd/l1_ls/ (2007)
  40. 40.
    Lan, G., Lu, Z., Monteiro, R.D.C.: Primal-dual first-order methods with o(1/ε) iteration-complexity for cone programming. Math. Program. (2009). doi:10.1007/s10107-008-0261-6 . http://www.springerlink.com/index/10.1007/s10107-008-0261-6
  41. 41.
    Larsen, R.M.: PROPACK: Software for Large and Sparse SVD Calculations. http://soi.stanford.edu/~rmunk/PROPACK/ (2004)
  42. 42.
    Liu, Y.J., Sun, D., Toh, K.C.: An implementable proximal point algorithmic framework for nuclear norm minimization. Math. Program. (2011). doi:10.1007/s10107-010-0437-8
  43. 43.
    Lorenz, D.: Constructing test instances for basis pursuit denoising. Technical report. arXiv:1103.2897 (2011)Google Scholar
  44. 44.
    Lu, Z.: Primal-dual first-order methods for a class of cone programming. INFORMS J. Comput. Preprint http://www.math.sfu.ca/~zhaosong/ResearchPapers/pdfirst_DS_2ndrev.pdf(2009)
  45. 45.
    Malgouyres F., Zeng T.: A predual proximal point algorithm solving a non negative basis pursuit denoising model. Int. J. Comput. Vis. 83(3), 294–311 (2009)CrossRefGoogle Scholar
  46. 46.
    Mangasarian O.L., Meyer R.R.: Nonlinear perturbation of linear programs. SIAM J. Control Optim. 17, 745–752 (1979)MathSciNetMATHCrossRefGoogle Scholar
  47. 47.
    Moreau J.J.: Proximité et dualité dans un espace hilbertien. Bull. Soc. Math. France 93, 273–299 (1965)MathSciNetMATHGoogle Scholar
  48. 48.
    Mosek ApS: The MOSEK Optimization Tools Version 2.5. http://www.mosek.com(2002)
  49. 49.
    Nemirovski A., Yudin D.: Problem complexity and method efficiency in optimization. Wiley-Interscience Series in Discrete Mathematics. Wiley, New York (1983)Google Scholar
  50. 50.
    Nesterov Y.: A method for unconstrained convex minimization problem with the rate of convergence \({\mathcal{O}(1/k^2)}\) . Doklady AN USSR (translated as Soviet Math. Docl.) 269, 543–547 (1983)MathSciNetGoogle Scholar
  51. 51.
    Nesterov, Y.: On an approach to the construction of optimal methods of minimization of smooth convex functions. Ekonomika i Mateaticheskie Metody 24, 509–517 (1988, in Russian)Google Scholar
  52. 52.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course. Applied Optimization, vol. 87. Kluwer, Boston (2004)Google Scholar
  53. 53.
    Nesterov Y.: Smooth minimization of non-smooth functions. Math. Program. Ser. A 103, 127–152 (2005)MathSciNetMATHCrossRefGoogle Scholar
  54. 54.
    Nesterov, Y.: Gradient methods for minimizing composite objective function. Technical report, CORE 2007/76, Université Catholique de Louvain, Louvain-la-Neuve, Belgium (2007)Google Scholar
  55. 55.
    Osher S., Mao Y., Dong B., Yin W.: Fast linearized Bregman iteration for compressive sensing and sparse denoising. Commun. Math. Sci. 8(1), 93–111 (2010)MathSciNetMATHGoogle Scholar
  56. 56.
    Rockafellar R.T.: Convex Analysis. Princeton University Press, Princeton (1970)MATHGoogle Scholar
  57. 57.
    Rockafellar R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetMATHCrossRefGoogle Scholar
  58. 58.
    Romberg, J.K.: The Dantzig selector and generalized thresholding. In: Proceedings of IEEE Conference on Information Science and System. Princeton, New Jersey (2008)Google Scholar
  59. 59.
    Rudin L.I., Osher S., Fatemi E.: Nonlinear total variation noise removal algorithm. Physica D 60, 259–268 (1992)MATHCrossRefGoogle Scholar
  60. 60.
    Saunders, M., Kim, B.: PDCO: Primal-dual interior method for convex objectives. Technical report, Stanford University. http://www.stanford.edu/group/SOL/software/pdco.html(2002)
  61. 61.
    Starck J.L., Ngyuen M.K., Murtagh F.: Wavelets and curvelets for image deconvolution: a combined approach. Signal Process. 83, 2279–2283 (2003)MATHCrossRefGoogle Scholar
  62. 62.
    Tibshirani R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58(1), 267–288 (1996)MathSciNetMATHGoogle Scholar
  63. 63.
    Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization. (2008). http://www.math.washington.edu/~tseng/papers.html , last accessed Sept 2009
  64. 64.
    Weiss P., Blanc-Féraud L., Aubert G.: Efficient schemes for total variation minimization under constraints in image processing. SIAM J. Sci. Comput. 31, 2047–2080 (2009)MathSciNetMATHCrossRefGoogle Scholar
  65. 65.
    Wen Z., Yin W., Goldfarb D., Zhang Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization and continuation. SIAM J. Sci. Comput. 32(4), 1832–1857 (2010)MathSciNetMATHCrossRefGoogle Scholar
  66. 66.
    Wright, S.J.: Solving 1-regularized regression problems. In: International Conference Combinatorics and Optimization, Waterloo (2007)Google Scholar
  67. 67.
    Wright S.J., Nowak R.D., Figueiredo M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  68. 68.
    Yin, W.: Analysis and generalizations of the linearized Bregman method. SIAM J. Imaging Sci. 3(4), 856–877 (2010). http://dx.doi.org/10.1137/090760350
  69. 69.
    Yin W., Osher S., Goldfarb D., Darbon J.: Bregman iterative algorithms for 1 minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1(1), 143–168 (2008)MathSciNetMATHCrossRefGoogle Scholar

Copyright information

© Springer and Mathematical Optimization Society 2011

Authors and Affiliations

  • Stephen R. Becker
    • 1
  • Emmanuel J. Candès
    • 2
  • Michael C. Grant
    • 1
  1. 1.Applied and Computational MathematicsCalifornia Institute of TechnologyPasadenaUSA
  2. 2.Departments of Mathematics and of StatisticsStanford UniversityStanfordUSA

Personalised recommendations