Skip to main content
Book cover

Splitting Methods in Communication, Imaging, Science, and Engineering pp 301–343Cite as

Operator Splitting Methods in Compressive Sensing and Sparse Approximation

  • 3486 Accesses

Part of the Scientific Computation book series (SCIENTCOMP)

Abstract

Compressive sensing and sparse approximation have many emerging applications, and are a relatively new driving force for the development of splitting methods in optimization. Many sparse coding problems are well described by variational models with ℓ 1-norm penalties and constraints that are designed to promote sparsity. Successful algorithms need to take advantage of the separable structure of potentially complicated objectives by “splitting” them into simpler pieces and iteratively solving a sequence of simpler convex minimization problems. In particular, isolating ℓ 1 terms from the rest of the objective leads to simple soft thresholding updates or ℓ 1 ball projections. A few basic splitting techniques can be used to design a huge variety of relevant algorithms. This chapter will focus on operator splitting strategies that are based on proximal operators, duality, and alternating direction methods. These will be explained in the context of basis pursuit variants and through compressive sensing applications.

Keywords

  • Compressed Sensing
  • Alternating Direction Method Of Multipliers (ADMM)
  • ADMM Iterations
  • Primal-dual Hybrid Gradient (PDHG)
  • Douglas-Rachford Splitting

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-41589-5_9
  • Chapter length: 43 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   169.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-41589-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   219.99
Price excludes VAT (USA)
Hardcover Book
USD   219.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 9.1
Fig. 9.2
Fig. 9.3

Notes

  1. 1.

    It is more correct to say light diverted towards/away from the detector results in a 0∕1 coefficient. In practice, 0∕1 measurements are converted to + 1∕ − 1 measurements by subtracting the average image intensity. Measurement matrices with + 1∕ − 1 coefficients are more well conditioned than their 0∕1 counterparts, and are easier to handle numerically.

References

  1. Arrow, K.J., Hurwicz, L., Uzawa, H.: Studies in Linear and Non-Linear Programming. Stanford University Press (1958)

    Google Scholar 

  2. Bach, F., Jenatton, R., Mairal, J., Obozinski, G.: Convex optimization with sparsity-inducing norms. In: S. Sra, S. Nowozin, S. Wright (eds.) Optimization for Machine Learning, pp. 19–53. The MIT Press, Cambridge, MA (2012)

    Google Scholar 

  3. Bajwa, W.U., Sayeed, A.M., Nowak, R.: A restricted isometry property for structurally-subsampled unitary matrices. In: Communication, Control, and Computing, 2009. Allerton 2009. 47th Annual Allerton Conference on, pp. 1005–1012. IEEE (2009)

    Google Scholar 

  4. Baraniuk, R., Davenport, M., DeVore, R., Wakin, M.: A simple proof of the restricted isometry property for random matrices. Constructive Approximation 28 (3), 253–263 (2008)

    CrossRef  MATH  Google Scholar 

  5. Bauschke, H.H., Combettes, P.L., Luke, D.R.: Finding best approximation pairs relative to two closed convex sets in Hilbert spaces. Journal of Approximation Theory 127 (2), 178–192 (2004)

    CrossRef  MATH  Google Scholar 

  6. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences 2 (1), 183–202 (2009)

    CrossRef  MATH  Google Scholar 

  7. Beck, A., Teboulle, M.: Gradient-based algorithms with applications to signal recovery. In: D. Palomar, Y. Eldar (eds.) Convex Optimization in Signal Processing and Communications, pp. 3–51. Cambridge University Press, Cambridge, UK (2010)

    Google Scholar 

  8. Becker, S., Fadili, J.: A quasi-Newton proximal splitting method. In: Advances in Neural Information Processing Systems (NIPS), pp. 2618–2626 (2012)

    Google Scholar 

  9. Van den Berg, E., Friedlander, M.: Probing the Pareto frontier for basis pursuit solutions. SIAM Journal on Scientific Computing 31 (2), 890–912 (2008)

    CrossRef  MATH  Google Scholar 

  10. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, Cambridge, MA (1982)

    MATH  Google Scholar 

  11. Bertsekas, D.P.: Projected Newton methods for optimization problems with simple constraints. SIAM Journal on Control and Optimization 20 (2), 221–246 (1982)

    CrossRef  MATH  Google Scholar 

  12. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Nashua, NH (1999)

    MATH  Google Scholar 

  13. Bertsekas, D.P.: Extended monotropic programming and duality. Journal of Optimization Theory and Applications 139 (2), 209–225 (2008)

    CrossRef  MATH  Google Scholar 

  14. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods, vol. 23. Prentice Hall, Englewood Cliffs, NJ (1989)

    MATH  Google Scholar 

  15. Bonettini, S., Ruggiero, V.: On the convergence of primal-dual hybrid gradient algorithms for total variation image restoration. Journal of Mathematical Imaging and Vision 44 (3), 236–253 (2012)

    CrossRef  MATH  Google Scholar 

  16. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning 3 (1), 1–122 (2011)

    CrossRef  MATH  Google Scholar 

  17. Bredies, K., Sun, H.: Preconditioned Douglas-Rachford splitting methods for convex-concave saddle-point problems. SIAM Journal on Numerical Analysis 53 (1), 421–444 (2015)

    CrossRef  MATH  Google Scholar 

  18. Bregman, L.M.: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics 7 (3), 200–217 (1967)

    CrossRef  MATH  Google Scholar 

  19. Brucker, P.: An O(n) algorithm for quadratic knapsack problems. Operations Research Letters 3 (3), 163–166 (1984)

    CrossRef  MATH  Google Scholar 

  20. Burger, M., Gilboa, G., Osher, S., Xu, J.: Nonlinear inverse scale space methods. Communications in Mathematical Sciences 4 (1), 179–212 (2006)

    CrossRef  MATH  Google Scholar 

  21. Burger, M., Möller, M., Benning, M., Osher, S.: An adaptive inverse scale space method for compressed sensing. Mathematics of Computation 82 (281), 269–299 (2013)

    CrossRef  MATH  Google Scholar 

  22. Burke, J.V., Qian, M.: A variable metric proximal point algorithm for monotone operators. SIAM Journal on Control and Optimization 37 (2), 353–375 (1999)

    CrossRef  MATH  Google Scholar 

  23. Cai, J.F., Candès, E.J., Shen, Z.: A singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization 20 (4), 1956–1982 (2010)

    CrossRef  MATH  Google Scholar 

  24. Cai, J.F., Osher, S., Shen, Z.: Linearized Bregman iterations for compressed sensing. Mathematics of Computation 78 (267), 1515–1536 (2009)

    CrossRef  MATH  Google Scholar 

  25. Candès, E., Romberg, J.: Sparsity and incoherence in compressive sampling. Inverse Problems 23 (3), 969 (2007)

    CrossRef  MATH  Google Scholar 

  26. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. Information Theory, IEEE Transactions on 52 (2), 489–509 (2006)

    CrossRef  MATH  Google Scholar 

  27. Candès, E.J., Romberg, J.K., Tao, T.: Stable signal recovery from incomplete and inaccurate measurements. Communications on pure and applied mathematics 59 (8), 1207–1223 (2006)

    CrossRef  MATH  Google Scholar 

  28. Candès, E.J., Tao, T.: Decoding by linear programming. Information Theory, IEEE Transactions on 51 (12), 4203–4215 (2005)

    CrossRef  MATH  Google Scholar 

  29. Censor, Y., Zenios, S.A.: Proximal minimization algorithm with D-functions. Journal of Optimization Theory and Applications 73 (3), 451–464 (1992)

    CrossRef  MATH  Google Scholar 

  30. Chambolle, A.: An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision 20 (1–2), 89–97 (2004)

    Google Scholar 

  31. Chambolle, A., De Vore, R.A., Lee, N.Y., Lucier, B.J.: Nonlinear wavelet image processing: variational problems, compression, and noise removal through wavelet shrinkage. Image Processing, IEEE Transactions on 7 (3), 319–335 (1998)

    CrossRef  MATH  Google Scholar 

  32. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision 40 (1), 120–145 (2011)

    CrossRef  MATH  Google Scholar 

  33. Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM Journal on Scientific Computing 20 (6), 1964–1977 (1999)

    CrossRef  MATH  Google Scholar 

  34. Chan, T.F., Mulet, P.: On the convergence of the lagged diffusivity fixed point method in total variation image restoration. SIAM Journal on Numerical Analysis 36 (2), 354–367 (1999)

    CrossRef  MATH  Google Scholar 

  35. Chen, C., He, B., Ye, Y., Yuan, X.: The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Mathematical Programming 155 (1–2), 57–79 (2016)

    CrossRef  MATH  Google Scholar 

  36. Chen, G., Teboulle, M.: Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM Journal on Optimization 3 (3), 538–543 (1993)

    CrossRef  MATH  Google Scholar 

  37. Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Mathematical Programming 64 (1), 81–101 (1994)

    CrossRef  MATH  Google Scholar 

  38. Chen, L., Sun, D., Toh, K.C.: A note on the convergence of ADMM for linearly constrained convex optimization problems. arXiv preprint arXiv:1507.02051 (2015)

    Google Scholar 

  39. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM Review 43 (1), 129–159 (2001)

    CrossRef  MATH  Google Scholar 

  40. Chen, Y., Hager, W., Huang, F., Phan, D., Ye, X., Yin, W.: Fast algorithms for image reconstruction with application to partially parallel MR imaging. SIAM Journal on Imaging Sciences 5 (1), 90–118 (2012)

    CrossRef  MATH  Google Scholar 

  41. Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing. In: Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer (2011)

    Google Scholar 

  42. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward-backward splitting. Multiscale Modeling & Simulation 4 (4), 1168–1200 (2005)

    CrossRef  MATH  Google Scholar 

  43. Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. Journal of Optimization Theory and Applications 158 (2), 460–479 (2013)

    CrossRef  MATH  Google Scholar 

  44. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics 57 (11), 1413–1457 (2004)

    CrossRef  MATH  Google Scholar 

  45. Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. In: R. Glowinski, S. Osher, W. Yin (eds.) Splitting Methods in Communication and Imaging, Science and Engineering, Chapter 4 Springer (2016)

    Google Scholar 

  46. Deng, W., Lai, M.J., Peng, Z., Yin, W.: Parallel multi-block ADMM with o(1/k) convergence. arXiv preprint arXiv:1312.3040 (2013)

    Google Scholar 

  47. Dong, B., Shen, Z., et al.: MRA based wavelet frames and applications. IAS Lecture Notes Series, Summer Program on “The Mathematics of Image Processing”, Park City Mathematics Institute 19 (2010)

    Google Scholar 

  48. Donoho, D.: Compressed sensing. Information Theory, IEEE Transactions on 52 (4), 1289–1306 (2006)

    CrossRef  MATH  Google Scholar 

  49. Donoho, D.: For most large underdetermined systems of equations, the minimal ℓ 1-norm near solution approximates the sparsest near-solution. Communications on Pure and Applied Mathematics 59 (7), 907–934 (2006)

    CrossRef  Google Scholar 

  50. Donoho, D., Huo, X.: Uncertainty principles and ideal atomic decomposition. Information Theory, IEEE Transactions on 47 (7), 2845–2862 (2001)

    CrossRef  MATH  Google Scholar 

  51. Donoho, D.L., Johnstone, J.M.: Ideal spatial adaptation by wavelet shrinkage. Biometrika 81 (3), 425–455 (1994)

    CrossRef  MATH  Google Scholar 

  52. Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Transactions of the American mathematical Society 82 (2), 421–439 (1956)

    CrossRef  MATH  Google Scholar 

  53. Duarte, M., Davenport, M., Takhar, D., Laska, J., Sun, T., Kelly, K., Baraniuk, R.: Single-pixel imaging via compressive sampling: Building simpler, smaller, and less-expensive digital cameras. Signal Processing Magazine, IEEE 25 (2), 83–91 (2008)

    CrossRef  Google Scholar 

  54. Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization. Ph.D. thesis, Massachusetts Institute of Technology (1989)

    Google Scholar 

  55. Eckstein, J.: Nonlinear proximal point algorithms using bregman functions, with applications to convex programming. Mathematics of Operations Research 18 (1), 202–226 (1993)

    CrossRef  MATH  Google Scholar 

  56. Eckstein, J.: Augmented Lagrangian and alternating direction methods for convex optimization: A tutorial and some illustrative computational results. RUTCOR Research Reports 32 (2012)

    Google Scholar 

  57. Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming 55 (1–3), 293–318 (1992)

    CrossRef  MATH  Google Scholar 

  58. Elad, M.: Why simple shrinkage is still relevant for redundant representations? Information Theory, IEEE Transactions on 52 (12), 5559–5569 (2006)

    MATH  Google Scholar 

  59. Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences 3 (4), 1015–1046 (2010)

    CrossRef  MATH  Google Scholar 

  60. Fadili, M., Starck, J.: Sparse representation-based image deconvolution by iterative thresholding. Astronomical Data Analysis (ADA) ’06, Marseille, France (2006)

    Google Scholar 

  61. Figueiredo, M., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. Image Processing, IEEE Transactions on 12 (8), 906–916 (2003)

    CrossRef  MATH  Google Scholar 

  62. Figueiredo, M.A., Bioucas-Dias, J.M., Nowak, R.D.: Majorization–minimization algorithms for wavelet-based image restoration. Image Processing, IEEE Transactions on 16 (12), 2980–2991 (2007)

    CrossRef  Google Scholar 

  63. Fortin, M., Glowinski, R.: Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems. North-Holland, Amsterdam (1983)

    MATH  Google Scholar 

  64. Frick, K., Lorenz, D.A., Resmerita, E.: Morozov’s principle for the augmented Lagrangian method applied to linear inverse problems. Multiscale Modeling & Simulation 9 (4), 1528–1548 (2011)

    CrossRef  MATH  Google Scholar 

  65. Friedlander, M., Tseng, P.: Exact regularization of convex programs. SIAM Journal on Optimization 18 (4), 1326–1350 (2007)

    CrossRef  MATH  Google Scholar 

  66. Fukushima, M.: Application of the alternating direction method of multipliers to separable convex programming problems. Computational Optimization and Applications 1 (1), 93–111 (1992)

    CrossRef  MATH  Google Scholar 

  67. Fukushima, M., Mine, H.: A generalized proximal point algorithm for certain non-convex minimization problems. International Journal of Systems Science 12 (8), 989–1000 (1981)

    CrossRef  MATH  Google Scholar 

  68. Gabay, D.: Applications of the method of multipliers to variational inequalities. In: M. Fortin, R. Glowinski (eds.) Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems. North-Holland, Amsterdam (1983)

    Google Scholar 

  69. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Computers & Mathematics with Applications 2 (1), 17–40 (1976)

    CrossRef  MATH  Google Scholar 

  70. Glowinski, R., Le Tallec, P.: Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics. SIAM, Philadelphia, PA (1989)

    CrossRef  MATH  Google Scholar 

  71. Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. ESAIM: Mathematical Modelling and Numerical Analysis 9 (R2), 41–76 (1975)

    MATH  Google Scholar 

  72. Goldstein, T., Li, M., Yuan, X.: Adaptive primal-dual splitting methods for statistical learning and image processing. In: Advances in Neural Information Processing Systems (NIPS), pp. 2080–2088 (2015)

    Google Scholar 

  73. Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM Journal on Imaging Sciences 7 (3), 1588–1623 (2014)

    CrossRef  MATH  Google Scholar 

  74. Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM Journal on Imaging Sciences 2 (2), 323–343 (2009)

    CrossRef  MATH  Google Scholar 

  75. Goldstein, T., Studer, C., Baraniuk, R.: A field guide to forward-backward splitting with a FASTA implementation. arXiv preprint arXiv:1411.3406 (2014)

    Google Scholar 

  76. Goldstein, T., Xu, L., Kelly, K.F., Baraniuk, R.: The STONE transform: Multi-resolution image enhancement and compressive video. IEEE Transactions on Image Processing 24 (12), 5581–5593 (2015)

    CrossRef  Google Scholar 

  77. Gupta, M., Agrawal, A., Veeraraghavan, A., Narasimhan, S.G.: Flexible voxels for motion-aware videography. In: Proc. of the 11th European Conference on Computer Vision: Part I, ECCV’10, pp. 100–114. Springer-Verlag, Berlin, Heidelberg (2010)

    Google Scholar 

  78. Hale, E.T., Yin, W., Zhang, Y.: Fixed-point continuation for ℓ 1-minimization: Methodology and convergence. SIAM Journal on Optimization 19 (3), 1107–1130 (2008)

    CrossRef  MATH  Google Scholar 

  79. He, B., Peng, Z., Wang, X.: Proximal alternating direction-based contraction methods for separable linearly constrained convex optimization. Frontiers of Mathematics in China 6 (1), 79–114 (2011)

    CrossRef  MATH  Google Scholar 

  80. He, B., Tao, M., Xu, M., Yuan, X.: An alternating direction-based contraction method for linearly constrained separable convex programming problems. Optimization 62 (4), 573–596 (2013)

    CrossRef  MATH  Google Scholar 

  81. He, B., You, Y., Yuan, X.: On the convergence of primal-dual hybrid gradient algorithm. SIAM Journal on Imaging Sciences 7 (4), 2526–2537 (2014)

    CrossRef  MATH  Google Scholar 

  82. He, B., Yuan, X.: Convergence analysis of primal-dual algorithms for a saddle-point problem: from contraction perspective. SIAM Journal on Imaging Sciences 5 (1), 119–149 (2012)

    CrossRef  MATH  Google Scholar 

  83. Hestenes, M.R.: Multiplier and gradient methods. Journal of Optimization Theory and Applications 4 (5), 303–320 (1969)

    CrossRef  MATH  Google Scholar 

  84. Hong, M., Chang, T., Wang, X., Razaviyayn, M., Ma, S., Luo, Z.Q.: A block successive upper bound minimization method of multipliers for linearly constrained convex optimization. arXiv preprint arXiv:1401.7079 (2014)

    Google Scholar 

  85. Hong, M., Luo, Z.Q.: On the linear convergence of the alternating direction method of multipliers. arXiv preprint arXiv:1208.3922 (2012)

    Google Scholar 

  86. Ji, J.X., Son, J.B., Rane, S.D.: PULSAR: A Matlab toolbox for parallel magnetic resonance imaging using array coils and multiple channel receivers. Concepts in Magnetic Resonance Part B: Magnetic Resonance Engineering 31 (1), 24–36 (2007)

    CrossRef  Google Scholar 

  87. Keeling, S.L., Clason, C., Hintermüller, M., Knoll, F., Laurain, A., Von Winckel, G.: An image space approach to cartesian based parallel MR imaging with total variation regularization. Medical Image Analysis 16 (1), 189–200 (2012)

    CrossRef  Google Scholar 

  88. Knoll, F., Clason, C., Bredies, K., Uecker, M., Stollberger, R.: Parallel imaging with nonlinear reconstruction using variational penalties. Magnetic Resonance in Medicine 67 (1), 34–41 (2012)

    CrossRef  Google Scholar 

  89. Kunisch, K., Hintermüller, M.: Total bounded variation regularization as a bilaterally constrained optimization problem. SIAM Journal on Applied Mathematics 64 (4), 1311–1333 (2004)

    CrossRef  MATH  Google Scholar 

  90. Lange, K., Hunter, D., Yang, I.: Optimization transfer using surrogate objective functions. Journal of Computational and Graphical Statistics 9 (1), 1–20 (2000)

    Google Scholar 

  91. Lee, J., Sun, Y., Saunders, M.: Proximal Newton-type methods for convex optimization. In: Advances in Neural Information Processing Systems (NIPS), pp. 836–844 (2012)

    Google Scholar 

  92. Li, G., Pong, T.K.: Global convergence of splitting methods for nonconvex composite optimization. SIAM Journal on Optimization 25 (4), 2434–2460 (2015)

    CrossRef  MATH  Google Scholar 

  93. Lions, P., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM Journal on Numerical Analysis 16 (6), 964–979 (1979)

    CrossRef  MATH  Google Scholar 

  94. Marcia, R.F., Harmany, Z.T., Willett, R.M.: Compressive coded aperture imaging. In: Proc. SPIE, p. 72460 (2009)

    Google Scholar 

  95. Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Revue Française d’Informatique et de Recherche Opérationnelle 4 (3), 154–158 (1970)

    MATH  Google Scholar 

  96. Moeller, M., Zhang, X.: Fast sparse reconstruction: Greedy inverse scale space flows. Mathematics of Computation 85 (297), 179–208 (2016)

    CrossRef  MATH  Google Scholar 

  97. Moreau, J.J.: Proximité et dualité dans un espace Hilbertien. Bulletin de la Société Mathématique de France 93, 273–299 (1965)

    MATH  Google Scholar 

  98. Nesterov, Y.: A method for unconstrained convex minimization problem with the rate of convergence O(1∕k 2). Soviet Mathematics Doklady 269 (3), 543–547 (1983)

    Google Scholar 

  99. Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterative regularization method for total variation-based image restoration. Multiscale Modeling & Simulation 4 (2), 460–489 (2005)

    CrossRef  MATH  Google Scholar 

  100. Osher, S., Ruan, F., Xiong, J., Yao, Y., Yin, W.: Sparse recovery via differential inclusions. Applied and Computational Harmonic Analysis (2016)

    Google Scholar 

  101. Ouyang, H., He, N., Tran, L., Gray, A.: Stochastic alternating direction method of multipliers. In: Proceedings of the 30th International Conference on Machine Learning, vol. 28, pp. 80–88 (2013)

    Google Scholar 

  102. Ouyang, Y., Chen, Y., Lan, G., Eduardo Pasiliao, J.: An accelerated linearized alternating direction method of multipliers. SIAM Journal on Imaging Sciences 8 (1), 644–681 (2015)

    CrossRef  MATH  Google Scholar 

  103. Parikh, N., Boyd, S.P.: Proximal algorithms. Foundations and Trends in Optimization 1 (3), 127–239 (2014)

    CrossRef  Google Scholar 

  104. Park, J.Y., Wakin, M.B.: Multiscale algorithm for reconstructing videos from streaming compressive measurements. Journal of Electronic Imaging 22 (2), 021,001–021,001 (2013)

    Google Scholar 

  105. Passty, G.B.: Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. Journal of Mathematical Analysis and Applications 72 (2), 383–390 (1979)

    CrossRef  MATH  Google Scholar 

  106. Patrinos, P., Stella, L., Bemporad, A.: Douglas-Rachford splitting: Complexity estimates and accelerated variants. In: Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on, pp. 4234–4239 (2014)

    Google Scholar 

  107. Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 1762–1769. IEEE (2011)

    Google Scholar 

  108. Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the Mumford-Shah functional. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1133–1140. IEEE (2009)

    Google Scholar 

  109. Popov, L.: A modification of the Arrow-Hurwicz method for search of saddle points. Mathematical Notes 28 (5), 845–848 (1980)

    CrossRef  MATH  Google Scholar 

  110. Powell, M.: A method for nonlinear constraints in minimization problems. In: R. Fletcher (ed.) Optimization. Academic Press, New York, NY (1969)

    Google Scholar 

  111. Reddy, D., Veeraraghavan, A., Chellappa, R.: P2C2: Programmable pixel compressive camera for high speed imaging. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR ’11, pp. 329–336 (2011)

    Google Scholar 

  112. Rockafellar, R.: Monotropic programming: descent algorithms and duality. Nonlinear Programming 4, 327–366 (1981)

    MATH  Google Scholar 

  113. Rockafellar, R.T.: Convex Analysis. Princeton University Press (1970)

    Google Scholar 

  114. Rockafellar, R.T.: A dual approach to solving nonlinear programming problems by unconstrained optimization. Mathematical Programming 5 (1), 354–373 (1973)

    CrossRef  MATH  Google Scholar 

  115. Rockafellar, R.T.: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Mathematics of Operations Research 1 (2), 97–116 (1976)

    CrossRef  MATH  Google Scholar 

  116. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM Journal on Control and Optimization 14 (5) (1976)

    Google Scholar 

  117. Rockafellar, R.T., Wets, R.J.B.: Variational Analysis. Springer Dordrecht (2009)

    Google Scholar 

  118. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60 (1), 259–268 (1992)

    CrossRef  MATH  Google Scholar 

  119. Sankaranarayanan, A.C., Studer, C., Baraniuk, R.G.: CS-MUVI: Video compressive sensing for spatial-multiplexing cameras. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–10. IEEE (2012)

    Google Scholar 

  120. Scherzer, O., Groetsch, C.: Inverse scale space theory for inverse problems. In: Scale-Space and Morphology in Computer Vision, vol. 2106, pp. 317–325. Springer, Berlin Heidelberg (2001)

    Google Scholar 

  121. Schmidt, M., Kim, D., Sra, S.: Projected Newton-type methods in machine learning. In: S. Sra, S. Nowozin, S. Wright (eds.) Optimization for Machine Learning, pp. 305–330. MIT Press (2011)

    Google Scholar 

  122. Shefi, R., Teboulle, M.: Rate of convergence analysis of decomposition methods based on the proximal method of multipliers for convex minimization. SIAM Journal on Optimization 24 (1), 269–297 (2014)

    CrossRef  MATH  Google Scholar 

  123. Sra, S.: Fast projections onto mixed-norm balls with applications. Data Mining and Knowledge Discovery 25 (2), 358–377 (2012)

    CrossRef  MATH  Google Scholar 

  124. Suzuki, T.: Stochastic dual coordinate ascent with alternating direction method of multipliers. In: Proceedings of the 31st International Conference on Machine Learning (ICML-14), pp. 736–744 (2014)

    Google Scholar 

  125. Tibshirani, R.: Regression shrinkage and selection via the Lasso. Journal of the Royal Statistical Society. Series B (Methodological) 58 (1), 267–288 (1996)

    Google Scholar 

  126. Valkonen, T.: A primal–dual hybrid gradient method for nonlinear operators with applications to MRI. Inverse Problems 30 (5), 055,012 (2014)

    Google Scholar 

  127. Vogel, C., Oman, M.: Iterative methods for total variation denoising. SIAM Journal on Scientific Computing 17 (1), 227–238 (1996)

    CrossRef  MATH  Google Scholar 

  128. Wang, H., Banerjee, A.: Online alternating direction method (longer version). arXiv preprint arXiv:1306.3721 (2013)

    Google Scholar 

  129. Wang, H., Banerjee, A.: Bregman alternating direction method of multipliers. In: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Weinberger. (eds.) Advances in Neural Information Processing Systems 27 (NIPS), pp. 2816–2824 (2014)

    Google Scholar 

  130. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM Journal on Imaging Sciences 1 (3), 248–272 (2008)

    CrossRef  MATH  Google Scholar 

  131. Wright, S.J., Nowak, R.D., Figueiredo, M.A.: Sparse reconstruction by separable approximation. Signal Processing, IEEE Transactions on 57 (7), 2479–2493 (2009)

    CrossRef  Google Scholar 

  132. Yin, W.: Analysis and generalizations of the linearized Bregman method. SIAM Journal on Imaging Sciences 3 (4), 856–877 (2010)

    CrossRef  MATH  Google Scholar 

  133. Yin, W., Osher, S.: Error forgetting of Bregman iteration. Journal of Scientific Computing 54 (2–3), 684–695 (2012)

    MATH  Google Scholar 

  134. Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for ℓ 1-minimization with applications to compressed sensing. SIAM Journal on Imaging Sciences 1 (1), 143–168 (2008)

    CrossRef  MATH  Google Scholar 

  135. Yuan, M., Lin, Y.: Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 68 (1), 49–67 (2006)

    Google Scholar 

  136. Zhang, X., Burger, M., Bresson, X., Osher, S.: Bregmanized nonlocal regularization for deconvolution and sparse reconstruction. SIAM Journal on Imaging Sciences 3 (3), 253–276 (2010)

    CrossRef  MATH  Google Scholar 

  137. Zhang, X., Burger, M., Osher, S.: A unified primal-dual algorithm framework based on Bregman iteration. Journal of Scientific Computing 46 (1), 20–46 (2010)

    CrossRef  MATH  Google Scholar 

  138. Zhang, Y.: Theory of compressive sensing via ℓ 1-minimization: a Non-RIP analysis and extensions. Journal of the Operations Research Society of China 1 (1), 79–105 (2013)

    CrossRef  MATH  Google Scholar 

  139. Zhong, W., Kwok, J.: Fast stochastic alternating direction method of multipliers. In: Proceedings of the 31st International Conference on Machine Learning (ICML), pp. 46–54 (2014)

    Google Scholar 

  140. Zhu, M., Chan, T.: An efficient primal-dual hybrid gradient algorithm for total variation image restoration. CAM report 08-34, UCLA (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Maryland, College Park, MD, USA

    Tom Goldstein

  2. Shanghai Jiao Tong University, Institute of Natural Sciences and School of Mathematical Sciences, Shanghai, China

    Xiaoqun Zhang

Authors
  1. Tom Goldstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoqun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tom Goldstein .

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Houston, Houston, Texas, USA

    Roland Glowinski

  2. Department of Mathematics, UCLA, Los Angeles, California, USA

    Stanley J. Osher

  3. Department of Mathematics, UCLA, Los Angeles, California, USA

    Wotao Yin

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Goldstein, T., Zhang, X. (2016). Operator Splitting Methods in Compressive Sensing and Sparse Approximation. In: Glowinski, R., Osher, S., Yin, W. (eds) Splitting Methods in Communication, Imaging, Science, and Engineering. Scientific Computation. Springer, Cham. https://doi.org/10.1007/978-3-319-41589-5_9

Download citation