Skip to main content
SpringerLink
Log in

Regularization properties of Krylov iterative solvers CGME and LSMR for linear discrete ill-posed problems with an application to truncated randomized SVDs

  • Original Paper
  • Published:
Numerical Algorithms Aims and scope Submit manuscript

Abstract

For the large-scale linear discrete ill-posed problem \(\min \limits \|Ax-b\|\) or Ax = b with b contaminated by Gaussian white noise, the following Krylov solvers are commonly used: LSQR, and its mathematically equivalent CGLS (i.e., the Conjugate Gradient (CG) method applied to ATAx = ATb), CGME (i.e., the CG method applied to \(\min \limits \|AA^{T}y-b\|\) or AATy = b with x = ATy), and LSMR (i.e., the minimal residual (MINRES) method applied to ATAx = ATb). These methods have intrinsic regularizing effects, where the number k of iterations plays the role of the regularization parameter. In this paper, we analyze the regularizing effects of CGME and LSMR and establish a number of results including the filtered SVD expansion of CGME iterates, which prove that the 2-norm filtering best possible regularized solutions by CGME and LSMR are less accurate than and at least as accurate as those by LSQR, respectively. We also prove that the semi-convergence of CGME and LSMR always occurs no later and sooner than that of LSQR, respectively. As a byproduct, using the analysis approach for CGME, we improve a fundamental result on the accuracy of the truncated rank k approximate SVD of A generated by randomized algorithms, and reveal how the truncation step damages the accuracy. Numerical experiments justify our results on CGME and LSMR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Aster, R.C., Borchers, B., Thurber, C.H.: Parameter Estimation and Inverse Problems, 2nd edn. Elsevier, New York (2013)

    MATH  Google Scholar 

  2. Berisha, S., Nagy, J.G.: Restore tools: Iterative methods for image restoration. Available from http://www.mathcs.emory.edu/~nagy/RestoreTools (2012)

  3. Björck, Å: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)

    Book  Google Scholar 

  4. Björck, Å: Numerical Methods in Matrix Computations, Texts in Applied Mathematics, vol. 59. Springer (2015)

  5. Chung, J., Palmer, K.: A hybrid LSMR algorithm for large-scale Tikhonov regularization. SIAM J. Sci. Comput. 37, S562–S580 (2015)

    Article  MathSciNet  Google Scholar 

  6. Craig, E.J.: The N-step iteration procedures. J. Math. Phys. 34, 64–73 (1955)

    Article  MathSciNet  Google Scholar 

  7. Demmel, J.: Applied Numerical Linear Algebra. SIAM, hiladelphia (1997)

    Book  Google Scholar 

  8. Eicke, B, Lious, A.K., Plato, R.: The instability of some gradient methods for ill-posed problems. Numer. Math. 58, 129–134 (1990)

    Article  MathSciNet  Google Scholar 

  9. Engl, H.W.: Regularization methods for the stable solution of inverse problems. Surveys Math. Indust. 3, 71–143 (1993)

    MathSciNet  MATH  Google Scholar 

  10. Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer Academic Publishers (2000)

  11. Fong, D.C.L., Saunders, M.: LSMR: An iterative algorithm for sparse least-squares problems. SIAM J. Sci. Comput. 33, 2950–2971 (2011)

    Article  MathSciNet  Google Scholar 

  12. Gazzola, S., Hansen, P.C., Nagy, J.G.: IR tools: A MATLAB package of iterative regularization methods and large-scale test problems. Numer. Algor. 81, 773–811 (2019)

    Article  MathSciNet  Google Scholar 

  13. Gazzola, S., Novati, P.: Inheritance of the discrete Picard condition in Krylov subspace methods. BIT Numer. Math. 56, 893–918 (2016)

    Article  MathSciNet  Google Scholar 

  14. Gilyazov, S.F., Gol’dman, N.L.: Regularization of Ill-Posed Problems by Iteration Methods. Kluwer Academic Publishers, Dordrecht (2000)

    Book  Google Scholar 

  15. Golub, G.H., O’Leary, D.P.: Some history of the conjugate gradient and Lanczos algorithms: 1948–1976. SIAM Rev 31, 50–102 (1989)

    Article  MathSciNet  Google Scholar 

  16. Halko, N., Martinsson, P.G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53, 217–288 (2011)

    Article  MathSciNet  Google Scholar 

  17. Hanke, M.: Conjugate Gradient Type Methods for Ill-Posed Problems. Pitman Research Notes in Mathematics Series, vol. 327. Longman, Essex (1995)

    Google Scholar 

  18. Hanke, M.: On Lanczos based methods for the regularization of discrete ill-posed problems. BIT Numer. Math. 41, 1008–1018 (2001). Suppl.

    Article  MathSciNet  Google Scholar 

  19. Hanke, M., Hansen, P.C.: Regularization methods for large-scale problems. Surveys Math. Indust. 3, 253–315 (1993)

    MathSciNet  MATH  Google Scholar 

  20. Hansen, P.C.: The discrete Picard condition for discrete ill-posed problems. BIT 30, 658–672 (1990)

    Article  MathSciNet  Google Scholar 

  21. Hansen, P.C.: Truncated singular value decomposition solutions to discrete ill-posed problems with ill-determined numerical rank. SIAM J. Sci. Statist. Comput. 11, 502–518 (1990)

    Article  MathSciNet  Google Scholar 

  22. Hansen, P.C.: Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. SIAM, Philadelphia (1998)

    Book  Google Scholar 

  23. Hansen, P.C.: Regularization Tools version 4.0 for Matlab 7.3. Numer. Algor. 46, 189–194 (2007)

    Article  MathSciNet  Google Scholar 

  24. Hansen, P.C.: Discrete Inverse Problems: Insight and Algorithms. SIAM, Philadelphia (2010)

    Book  Google Scholar 

  25. Hansen, P.C., Saxild-Hansen, M.: AIR tools–a MATLAB package of algebraic iterative reconstruction methods. J. Comput. Appl. Math. 236, 2167–2178 (2012)

    Article  MathSciNet  Google Scholar 

  26. Hansen, P.C., Jorgensen, J.S.: AIR Tools II: Algebraic iterative reconstruction methods, improved implementation. Numer. Algor. 79, 107–137 (2018)

    Article  MathSciNet  Google Scholar 

  27. Hestenes, M.R., Stiefel, E.: Methods of conjugate gradients for solving linear systems. J. Res. Nat. Bur. Stand. 49, 409–436 (1952)

    Article  MathSciNet  Google Scholar 

  28. Hnětynková, M.R., Kubínová, M., Plešinger, M.: Noise representation in residuals of LSQR, LSMR, and Craig regularization. Linear Algebra Appl. 533, 357–379 (2017)

    Article  MathSciNet  Google Scholar 

  29. Hnětynková, M.R., Plešinger, P., Strakoš, Z.: The regularizing effect of the Golub-Kahan iterative bidiagonalization and revealing the noise level in the data. BIT Numer. Math. 49, 669–696 (2009)

    Article  MathSciNet  Google Scholar 

  30. Hofmann, B.: Regularization for Applied Inverse and Ill-Posed Problems. Teubner, Stuttgart (1986)

    Book  Google Scholar 

  31. Huang, Y., Jia, Z.: Some results on the regularization of LSQR for large-scale ill-posed problems. Sci. China Math. 60, 701–718 (2017)

    Article  MathSciNet  Google Scholar 

  32. Jia, Z.: Approximation accuracy of the Krylov subspaces for linear discrete ill-posed problems. arXiv:math.NA/1805.10132v2

  33. Jia, Z.: The low rank approximations and Ritz values in LSQR for linear discrete ill-posed problems. arXiv:math.NA/1811.03454v1

  34. Jia, Z.: The Krylov subspaces, low rank approximations and Ritz values in LSQR for linear discrete ill-posed problems: the multiple singular value case. In preparation (2019)

  35. Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Springer, New York (2005)

    MATH  Google Scholar 

  36. Kern, M.: Numerical Methods for Inverse Problems. Wiley (2016)

  37. Kirsch, A.: An Introduction to the Mathematical Theory of Inverse Problems, 2nd edn. Springer, New York (2011)

    Book  Google Scholar 

  38. Meurant, G.: The Lanczos and Conjugate Gradient Algorithms: From Theory to Finite Precision Computations. SIAM, Philadelphia (2006)

    Book  Google Scholar 

  39. Natterer, F.: The Mathematics of Computerized Tomography. SIAM, Philadelphia (2001)

    Book  Google Scholar 

  40. Paige, C.C., Saunders, M.A.: Solutions of sparse indefinite systems of linear equations. SIAM J. Numer. Anal. 12, 617–629 (1975)

    Article  MathSciNet  Google Scholar 

  41. Paige, C.C., Saunders, M.A.: LSQR: An algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8, 43–71 (1982)

    Article  MathSciNet  Google Scholar 

  42. Parlett, B.N.: The Symmetric Eigenvalue Problem. SIAM, Philadelphia (1998)

    Book  Google Scholar 

  43. Stewart, G.W., Sun, J.-G.: Matrix Perturbation Theory. Academic Press, Inc., Boston (1990)

  44. Varah, J.M.: A practical examination of some numerical methods for linear discrete ill-posed problems. SIAM Rev. 21, 100–111 (1979)

    Article  MathSciNet  Google Scholar 

  45. Vogel, C.R.: Computational Methods for Inverse Problems. SIAM, Philadelphia (2002)

    Book  Google Scholar 

  46. Wilkinson, J.H.: The Algebraic Eigenvalue Problem. Clarendon Press, Oxford (1965)

    MATH  Google Scholar 

Download references

Acknowledgements

I thank two referees very much for their careful reading of the paper and for their valuable suggestions and comments, which helped me to improve the presentation.

Funding

This work was financially supported in part by the National Natural Science Foundation of China (No. 11771249).

Author information

Authors and Affiliations

  1. Department of Mathematical Sciences, Tsinghua University, Beijing, 100084, China

    Zhongxiao Jia

Authors
  1. Zhongxiao Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhongxiao Jia.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, Z. Regularization properties of Krylov iterative solvers CGME and LSMR for linear discrete ill-posed problems with an application to truncated randomized SVDs. Numer Algor 85, 1281–1310 (2020). https://doi.org/10.1007/s11075-019-00865-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11075-019-00865-w

Keywords

Mathematics subject classification (2010)

Search

Navigation