A Preconditioned Conjugate Gradient Method with Active Set Strategy for \(\ell _1\)-Regularized Least Squares

  • Wan-You Cheng
  • Dong-Hui Li


In the paper, we consider the \(\ell _1\)-regularized least square problem which has been intensively involved in the fields of signal processing, compressive sensing, linear inverse problems and statistical inference. The considered problem has been proved recently to be equivalent to a nonnegatively constrained quadratic programming (QP). In this paper, we use a recently developed active conjugate gradient method to solve the resulting QP problem. To improve the algorithm’s performance, we design a subspace exact steplength as well as a precondition technique. The performance comparisons illustrate that the proposed algorithm is competitive and even performs little better than several state-of-the-art algorithms.


Compressed sensing \(\ell _1\)-Regularized optimization Conjugate gradient method Precondition 

Mathematics Subject Classification

90C06 90C25 65Y20 94A08 


  1. 1.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Daubechies, I., Defrise, M., Mol, C.D.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57, 1413–1457 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Figueiredo, M.A.T., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process. 12, 906–916 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Bioucas-Dias, J.M., Figueiredo, M.A.T.: A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Elad, M., Matalon, B., Zibulevsky, M.: Subspace optimization methods for linear least squares with non-quadratic regularization. Appl. Comput. Harmon. Anal. 23, 346–367 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation for \(\ell _1\) minimization: methodology and convergence. SIAM J. Optim. 19, 1107–1130 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Hager, W.W., Phan, D.T., Zhang, H.: Gradient-based methods for sparse recovery. SIAM J. Imaging Sci. 4, 146–165 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Wen, Z.W., Yin, W.T., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization, and continuation. SIAM J. Sci. Comput. 32, 1832–1857 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Wen, Z.W., Yin, W.T., Zhang, H., Goldfarb, D.: On the convergence of an active-set method for \(\ell _1\) minimization. Optim. Methods Softw. 27, 1127–1146 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Wright, J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Afonso, M., Bioucas-Dias, J., Figueiredo, M.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear program. SIAM J. Optim. 23, 2183–2207 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Tseng, P., Yun, S.W.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117, 387–423 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Yin, W.T., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for \(\ell _1\)-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1, 143–168 (2011)CrossRefzbMATHGoogle Scholar
  16. 16.
    Nesterov, Y.: Gradient methods for minimizing composite objective function (2007) [Online].
  17. 17.
    Aybat, S., Iyengar, G.: A first-order smoothed penalty method for compressed sensing. SIAM J. Optim. 21, 287–313 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Cands, E., Romberg, J.: \(\ell _1\)-magic: A collection of MATLAB routines for solving the convex optimization programs central to compressive sampling 2006 [Online].
  19. 19.
    Saunders, M.: PDCO: Primal-Dual Interior Method for Convex Objectives (2002) [Online].
  20. 20.
    Kim, S.J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: An interior-point method for large-scale \(\ell _1\)-regularized least squares. IEEE J. Sel. Top. Signal Process. 1, 606–617 (2007)CrossRefGoogle Scholar
  21. 21.
    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1, 586–597 (2007)CrossRefGoogle Scholar
  22. 22.
    Baraniuk, R.: Compressive sensing. IEEE Signal Process. Mag. 24, 118–121 (2007)CrossRefGoogle Scholar
  23. 23.
    Beck, A., Eldar, Y.C.: Sparsity constrained nonlinear optimization: optimality conditions and algorithms. SIAM J. Optim 23, 1480–1509 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Cheng, W.Y., Dai, Y.H.: Gradient-based method with active set strategy for \(\ell _1\) optimization (2018).
  25. 25.
    Landi, G.: A modified Newton projection method for \(\ell _1\)-regularized least squares image deblurring. J. Math. Imaging Vis. 5, 195–208 (2015)CrossRefzbMATHGoogle Scholar
  26. 26.
    Cheng, W.Y., Liu, Q.F., Li, D.H.: An accurate actives set conjugate gradient method with project search for bound constrained optimization. Optim. Lett. 8, 763–776 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Operations Research Society of China, Periodicals Agency of Shanghai University, Science Press, and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of ComputerDongguan University of TechnologyDongguanChina
  2. 2.School of Mathematical SciencesSouth China Normal UniversityGuangzhouChina

Personalised recommendations