A Preconditioned Conjugate Gradient Method with Active Set Strategy for \(\ell _1\)-Regularized Least Squares



In the paper, we consider the \(\ell _1\)-regularized least square problem which has been intensively involved in the fields of signal processing, compressive sensing, linear inverse problems and statistical inference. The considered problem has been proved recently to be equivalent to a nonnegatively constrained quadratic programming (QP). In this paper, we use a recently developed active conjugate gradient method to solve the resulting QP problem. To improve the algorithm’s performance, we design a subspace exact steplength as well as a precondition technique. The performance comparisons illustrate that the proposed algorithm is competitive and even performs little better than several state-of-the-art algorithms.


Compressed sensing \(\ell _1\)-Regularized optimization Conjugate gradient method Precondition 

Mathematics Subject Classification

90C06 90C25 65Y20 94A08 


  1. 1.
    Bruckstein, A.M., Donoho, D.L., Elad, M.: From sparse solutions of systems of equations to sparse modeling of signals and images. SIAM Rev. 51, 34–81 (2009)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Daubechies, I., Defrise, M., Mol, C.D.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57, 1413–1457 (2004)MathSciNetCrossRefMATHGoogle Scholar
  3. 3.
    Figueiredo, M.A.T., Nowak, R.D.: An EM algorithm for wavelet-based image restoration. IEEE Trans. Image Process. 12, 906–916 (2003)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183–202 (2009)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Bioucas-Dias, J.M., Figueiredo, M.A.T.: A new twist: two-step iterative shrinkage/thresholding algorithms for image restoration. IEEE Trans. Image Process. 16, 2992–3004 (2007)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Elad, M., Matalon, B., Zibulevsky, M.: Subspace optimization methods for linear least squares with non-quadratic regularization. Appl. Comput. Harmon. Anal. 23, 346–367 (2007)MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Hale, E.T., Yin, W.T., Zhang, Y.: Fixed-point continuation for \(\ell _1\) minimization: methodology and convergence. SIAM J. Optim. 19, 1107–1130 (2008)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Hager, W.W., Phan, D.T., Zhang, H.: Gradient-based methods for sparse recovery. SIAM J. Imaging Sci. 4, 146–165 (2011)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Wen, Z.W., Yin, W.T., Goldfarb, D., Zhang, Y.: A fast algorithm for sparse reconstruction based on shrinkage, subspace optimization, and continuation. SIAM J. Sci. Comput. 32, 1832–1857 (2010)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Wen, Z.W., Yin, W.T., Zhang, H., Goldfarb, D.: On the convergence of an active-set method for \(\ell _1\) minimization. Optim. Methods Softw. 27, 1127–1146 (2012)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Wright, J., Nowak, R.D., Figueiredo, M.A.T.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57, 2479–2493 (2009)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Afonso, M., Bioucas-Dias, J., Figueiredo, M.: Fast image recovery using variable splitting and constrained optimization. IEEE Trans. Image Process. 19, 2345–2356 (2010)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Boley, D.: Local linear convergence of the alternating direction method of multipliers on quadratic or linear program. SIAM J. Optim. 23, 2183–2207 (2013)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Tseng, P., Yun, S.W.: A coordinate gradient descent method for nonsmooth separable minimization. Math. Program. 117, 387–423 (2009)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Yin, W.T., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for \(\ell _1\)-minimization with applications to compressed sensing. SIAM J. Imaging Sci. 1, 143–168 (2011)CrossRefMATHGoogle Scholar
  16. 16.
    Nesterov, Y.: Gradient methods for minimizing composite objective function (2007) [Online]. http://www.optimization-online.org/DB_HTML/2007/09/1784.html
  17. 17.
    Aybat, S., Iyengar, G.: A first-order smoothed penalty method for compressed sensing. SIAM J. Optim. 21, 287–313 (2011)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Cands, E., Romberg, J.: \(\ell _1\)-magic: A collection of MATLAB routines for solving the convex optimization programs central to compressive sampling 2006 [Online]. www.acm.caltech.edu/l1magic/
  19. 19.
    Saunders, M.: PDCO: Primal-Dual Interior Method for Convex Objectives (2002) [Online]. http://www.stanford.edu/group/SOL/software/pdco.html
  20. 20.
    Kim, S.J., Koh, K., Lustig, M., Boyd, S., Gorinevsky, D.: An interior-point method for large-scale \(\ell _1\)-regularized least squares. IEEE J. Sel. Top. Signal Process. 1, 606–617 (2007)CrossRefGoogle Scholar
  21. 21.
    Figueiredo, M.A.T., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1, 586–597 (2007)CrossRefGoogle Scholar
  22. 22.
    Baraniuk, R.: Compressive sensing. IEEE Signal Process. Mag. 24, 118–121 (2007)CrossRefGoogle Scholar
  23. 23.
    Beck, A., Eldar, Y.C.: Sparsity constrained nonlinear optimization: optimality conditions and algorithms. SIAM J. Optim 23, 1480–1509 (2013)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Cheng, W.Y., Dai, Y.H.: Gradient-based method with active set strategy for \(\ell _1\) optimization (2018).  https://doi.org/10.1090/mcom/3238
  25. 25.
    Landi, G.: A modified Newton projection method for \(\ell _1\)-regularized least squares image deblurring. J. Math. Imaging Vis. 5, 195–208 (2015)CrossRefMATHGoogle Scholar
  26. 26.
    Cheng, W.Y., Liu, Q.F., Li, D.H.: An accurate actives set conjugate gradient method with project search for bound constrained optimization. Optim. Lett. 8, 763–776 (2014)MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91, 201–213 (2002)MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Operations Research Society of China, Periodicals Agency of Shanghai University, Science Press, and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of ComputerDongguan University of TechnologyDongguanChina
  2. 2.School of Mathematical SciencesSouth China Normal UniversityGuangzhouChina

Personalised recommendations