Computational Optimization and Applications

, Volume 56, Issue 3, pp 507–530 | Cite as

An efficient augmented Lagrangian method with applications to total variation minimization

  • Chengbo LiEmail author
  • Wotao Yin
  • Hong Jiang
  • Yin Zhang


Based on the classic augmented Lagrangian multiplier method, we propose, analyze and test an algorithm for solving a class of equality-constrained non-smooth optimization problems (chiefly but not necessarily convex programs) with a particular structure. The algorithm effectively combines an alternating direction technique with a nonmonotone line search to minimize the augmented Lagrangian function at each iteration. We establish convergence for this algorithm, and apply it to solving problems in image reconstruction with total variation regularization. We present numerical results showing that the resulting solver, called TVAL3, is competitive with, and often outperforms, other state-of-the-art solvers in the field.


Compressive sensing Non-smooth optimization Augmented Lagrangian method Nonmonotone line search Barzilai-Borwein method Single-pixel camera 



The work of the first author was supported in part by NSF Grant DMS-0811188. The work of the second author was supported in part by NSF grants DMS-07-48839 and ECCS-1028790, as well as ONR Grant N00014-08-1-1101. The work of the fourth author was supported in part by NSF Grant DMS-0811188, ONR Grant N00014-08-1-1101, and NSF Grant DMS-1115950. The first and the fourth authors also appreciate a gift fund from Bell Labs, Alcatel-Lucent to Rice University that partially supported their travels to international conferences. Last but not least, we thank the two anonymous referees for their constructive criticism and their helpful suggestions.


  1. 1.
    Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988) MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Becker, S., Bobin, J., Candès, E.: NESTA: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4, 1–39 (2011) MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Bioucas-Dias, J., Figueiredo, M.: A new TwIST: two-step iterative thresholding algorithm for image restoration. IEEE Trans. Image Process. 16(12), 2992–3004 (2007) MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bioucas-Dias, J., Figueiredo, M.: Two-step algorithms for linear inverse problems with non-quadratic regularization. In: IEEE International Conference on Image Processing—ICIP 2007, San Antonio, TX, USA, September 2007 Google Scholar
  5. 5.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23(11), 1222–1239 (2001) CrossRefGoogle Scholar
  6. 6.
    Candès, E., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006) CrossRefzbMATHGoogle Scholar
  7. 7.
    Candès, E., Tao, T.: Near optimal signal recovery from random projections: universal encoding strategies. IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006) CrossRefGoogle Scholar
  8. 8.
    Chambolle, A.: An algorithm for total variation minimization and applications. J. Math. Imaging Vis. 20, 89–97 (2004) MathSciNetCrossRefGoogle Scholar
  9. 9.
    Chan, T., Wong, C.K.: Total variation blind deconvolution. IEEE Trans. Image Process. 7(3), 370–375 (1998) CrossRefGoogle Scholar
  10. 10.
    Chang, T., He, L., Fang, T.: MR image reconstruction from sparse radial samples using Bregman iteration. In: ISMRM (2006) Google Scholar
  11. 11.
    Donoho, D.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006) MathSciNetCrossRefGoogle Scholar
  12. 12.
    Donoho, D.: Neighborly polytopes and sparse solution of underdetermined linear equations. IEEE Trans. Inf. Theory (2006) Google Scholar
  13. 13.
    Duarte, M.F., Sarvotham, S., Baron, D., Wakin, M.B., Baraniuk, R.G.: Distributed compressed sensing of jointly sparse signals. In: 39th Asilomar Conference on Signals, Systems and Computers, pp. 1537–1541 (2005) Google Scholar
  14. 14.
    Fortin, M., Glowinski, R.: Méthodes de Lagrangien Augmenté. Application à la Résolution Numérique de Problèmes aux Limites. Dunod-Bordas, Paris (1982) (in French) Google Scholar
  15. 15.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximations. Comput. Appl. Math. 2, 17–40 (1976) CrossRefzbMATHGoogle Scholar
  16. 16.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer, Berlin (1984) CrossRefzbMATHGoogle Scholar
  17. 17.
    Glowinski, R., Marrocco, A.: Sur l’approximation par éléments finis d’ordre un et la résolution par pénalisation-dualité d’une classe de problèmes de Dirichlet nonlinéaires. C. R. Math. Acad. Sci. Paris 278A, 1649–1652 (1974) (in French) MathSciNetGoogle Scholar
  18. 18.
    Goldstein, T., Osher, S.: The split Bregman method for L1 regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009) MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Grippo, L., Lampariello, F., Lucidi, S.: A nonmonotone line search technique for Newton’s method. SIAM J. Numer. Anal. 23, 707–716 (1986) MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Hager, W.W., Phan, D.T., Zhang, H.: Gradient-based methods for sparse recovery. SIAM J. Imaging Sci. 4, 146–165 (2011) MathSciNetCrossRefGoogle Scholar
  21. 21.
    Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969) MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Jiang, H., Li, C., Haimi-Cohen, R., Wilford, P., Zhang, Y.: Scalable video coding using compressive sensing. Bell Labs Tech. J. 16, 149–169 (2012) CrossRefGoogle Scholar
  23. 23.
    Laska, J., Kirolos, S., Duarte, M., Ragheb, T., Baraniuk, R., Massoud, Y.: Theory and implementation of an analog-to-information converter using random demodulation. In: Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), New Orleans, Louisiana (2007) Google Scholar
  24. 24.
    Li, C., Jiang, H., Wilford, P., Zhang, Y.: Video coding using compressive sensing for wireless communications. In: IEEE Wireless Communications and Networking Conference (WCNC), pp. 2077–2082 (2011). doi: 10.1109/WCNC.2011.5779474 Google Scholar
  25. 25.
    Li, C., Sun, T., Kelly, K., Zhang, Y.: A compressive sensing and unmixing scheme for hyperspectral data processing. IEEE Trans. Image Process. 21, 1200–1210 (2012) MathSciNetCrossRefGoogle Scholar
  26. 26.
  27. 27.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979) MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Natarajan, B.K.: Sparse approximate solutions to linear systems. SIAM J. Comput. 24, 227–234 (1995) MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Nesterov, Yu.: Smooth minimization of non-smooth functions. Math. Program., Ser. A 103, 127–152 (2005) MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Osher, S., Burger, M., Goldfarb, D., Xu, J., Yin, W.: An iterated regularization method for total variation based image restoration. Multiscale Model. Simul. 4, 460–489 (2005) MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Powell, M.J.D.: A method for nonlinear constraints in minimization problems. In: Fletcher, R. (ed.) Optimization, pp. 283–298. Academic Press, London (1969) Google Scholar
  32. 32.
    Rockafellar, R.T.: The multiplier method of Hestenes and Powell applied to convex programming. J. Optim. Theory Appl. 12(6), 555–562 (1973) MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 259–268 (1992) Google Scholar
  34. 34.
    Sun, T., Woods, G.L., Duarte, M.F., Kelly, K.F., Li, C., Zhang, Y.: OBIC measurements without lasers or raster-scanning based on compressive sensing. In: Proceedings of the 35th International Symposium for Testing and Failure Analysis (2009) Google Scholar
  35. 35.
    Takhar, D., Laska, J.N., Wakin, M.B., Duarte, M.F., Baron, D., Sarvotham, S., Kelly, K.F., Baraniuk, R.G.: A new compressive imaging camera architecture using optical-domain compression. In: Computational Imaging IV, vol. 6065, pp. 43–52 (2006) CrossRefGoogle Scholar
  36. 36.
    Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1(4), 248–272 (2008) MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Yang, J., Yin, W., Zhang, Y.: A fast alternating direction method for TVL1-L2 signal reconstruction from partial Fourier data. Technical Report, TR08-27, CAAM, Rice University (2008) Google Scholar
  38. 38.
    Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for 1-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1, 143–168 (2008) MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Yin, W., Morgan, S., Yang, J., Zhang, Y.: Practical compressive sensing with Toeplitz and circulant matrices. In: Proceedings of Visual Communications and Image Processing (VCIP) (2010) Google Scholar
  40. 40.
    Zhang, H., Hager, W.W.: A nonmonotone line search technique and its application to unconstrained optimization. SIAM J. Optim. 14, 1043–1056 (2004) MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Xiao, Y., Song, H.: An inexact alternating directions algorithm for constrained total variation regularized compressive sensing problems. J. Math. Imaging Vis. 44, 114–127 (2012) MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Department of Computational and Applied MathematicsRice UniversityHoustonUSA
  2. 2.Bell LaboratoriesAlcatel-LucentMurray HillUSA

Personalised recommendations