Skip to main content
SpringerLink
Log in

A Note on the Complexity of Proximal Iterative Hard Thresholding Algorithm

  • Published:
Journal of the Operations Research Society of China Aims and scope Submit manuscript

Abstract

The iterative hard thresholding (IHT) algorithm is a powerful and efficient algorithm for solving \(\ell _0\)-regularized problems and inspired many applications in sparse-approximation and image-processing fields. Recently, some convergence results are established for the proximal scheme of IHT, namely proximal iterative hard thresholding (PIHT) algorithm (Blumensath and Davies, in J Fourier Anal Appl 14:629–654, 2008; Hu et al., Methods 67:294–303, 2015; Lu, Math Program 147:125–154, 2014; Trzasko et al., IEEE/SP 14th Workshop on Statistical Signal Processing, 2007) on solving the related \(\ell _0\)-optimization problems. However, the complexity analysis for the PIHT algorithm is not well explored. In this paper, we aim to provide some complexity estimations for the PIHT sequences. In particular, we show that the complexity of the sequential iterate error is at o(1 / k). Under the assumption that the objective function is composed of a quadratic convex function and \(\ell _0\) regularization, we show that the PIHT algorithm has \({\mathbb {R}}\)-linear convergence rate. Finally, we illustrate some applications of this algorithm for compressive sensing reconstruction and sparse learning and validate the estimated error bounds.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Blumensath, T., Davies, M.E.: Iterative thresholding for sparse approximations. J. Fourier Anal. Appl. 14(5–6), 629–654 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  2. Hu, Y.H., Li, C., Yang, X.Q.: Proximal gradient algorithm for group sparse optimization. Methods 67(3), 294–303 (2015)

    Google Scholar 

  3. Lu, Z.: Iterative hard thresholding methods for \(\ell _0\) regularized convex cone programming. Math. Program. 147(1–2), 125–154 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  4. Trzasko, J., Manduca, A., Borisch, E.: Sparse mri reconstruction via multiscale l0-continuation. In: IEEE/SP 14th Workshop on Statistical Signal Processing, pp. 176–180 (2007)

  5. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  6. Chan, R., Chan, T., Shen, L., Shen, Z.: Wavelet algorithms for high-resolution image reconstruction. SIAM J. Sci. Comput. 24, 1408–1432 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  7. Chan, R.H., Riemenschneider, S.D., Shen, L., Shen, Z.: Tight frame: an efficient way for high-resolution image reconstruction. Appl. Comput. Harmon. Anal. 17(1), 91–115 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  8. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)

    Article  MATH  Google Scholar 

  9. Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  10. Shen, Z., Toh, K.-C., Yun, S.: An accelerated proximal gradient algorithm for frame-based image restoration via the balanced approach. SIAM J. Imaging Sci. 4(2), 573–596 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  11. Tseng, P.: On accelerated proximal gradient methods for convex–concave optimization. SIAM J. Optim. 12, 724–739 (2008)

    Google Scholar 

  12. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MATH  Google Scholar 

  13. Zhang, X., Lu, Y., Chan, T.: A novel sparsity reconstruction method from poisson data for 3D bioluminescence tomography. J. Sci. Comput. 1, 17–32 (2012)

    Google Scholar 

  14. Dong, B., Zhang, Y.: An efficient algorithm for \(l_0\) minimization in wavelet frame based image restoration. J. Sci. Comput. 54(2–3), 350–368 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  15. Zhang, Y., Dong, B., Zhaosong, L.: \(\ell _0\) minimization for wavelet frame based image restoration. Math. Comput. 82(282), 995–1015 (2013)

    Article  MATH  Google Scholar 

  16. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  17. Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  18. Lions, P.-L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16(6), 964–979 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  19. Blumensath, T., Davies, M.E.: Iterative hard thresholding for compressed sensing. Appl. Comput. Harmon. Anal. 27(3), 265–274 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  20. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  21. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-seidel methods. Math. Program. 137(1–2), 91–129 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  22. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. Math. Program. 146, 1–36 (2013)

    MathSciNet  Google Scholar 

  23. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the kurdyka-łojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  24. Bao, C., Dong, B., Hou, L., Shen, Z., Zhang, X., Zhang, X.: Proximal iterative hard thresholding methods for wavelet frame based image restoration. CAM report 14-85, UCLA (2014)

  25. Dong, Y.: The proximal point algorithm revisited. J. Optim. Theory Appl. 161(2), 478–489 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  26. Hastie, T., Tibshirani, R., Friedman, J., Franklin, J.: The elements of statistical learning: data mining, inference and prediction. Math. Intel. 27(2), 83–85 (2005)

    Google Scholar 

  27. Lichman, M.: UCI machine learning repository (2013)

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their helpful comments in improving the presentation of this paper.

Author information

Authors and Affiliations

  1. Department of Mathematics, Shanghai Jiao Tong University, Shanghai, 200240, China

    Xue Zhang & Xiao-Qun Zhang

  2. Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China

    Xiao-Qun Zhang

Authors
  1. Xue Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiao-Qun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiao-Qun Zhang.

Additional information

The authors were partially supported by the National Natural Science Foundation of China (No. 91330102) and 973 program (No. 2015CB856000).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Zhang, XQ. A Note on the Complexity of Proximal Iterative Hard Thresholding Algorithm. J. Oper. Res. Soc. China 3, 459–473 (2015). https://doi.org/10.1007/s40305-015-0105-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40305-015-0105-z

Keywords

Mathematics Subject Classification

Search

Navigation