Skip to main content
SpringerLink
Log in

Unbiased predictive risk estimation of the Tikhonov regularization parameter: convergence with increasing rank approximations of the singular value decomposition

  • Published:
BIT Numerical Mathematics Aims and scope Submit manuscript

Abstract

The truncated singular value decomposition may be used to find the solution of linear discrete ill-posed problems in conjunction with Tikhonov regularization and requires the estimation of a regularization parameter that balances between the sizes of the fit to data function and the regularization term. The unbiased predictive risk estimator is one suggested method for finding the regularization parameter when the noise in the measurements is normally distributed with known variance. In this paper we provide an algorithm using the unbiased predictive risk estimator that automatically finds both the regularization parameter and the number of terms to use from the singular value decomposition. Underlying the algorithm is a new result that proves that the regularization parameter converges with the number of terms from the singular value decomposition. For the analysis it is sufficient to assume that the discrete Picard condition is satisfied for exact data and that noise completely contaminates the measured data coefficients for a sufficiently large number of terms, dependent on both the noise level and the degree of ill-posedness of the system. A lower bound for the regularization parameter is provided leading to a computationally efficient algorithm. Supporting results are compared with those obtained using the method of generalized cross validation. Simulations for two-dimensional examples verify the theoretical analysis and the effectiveness of the algorithm for increasing noise levels, and demonstrate that the relative reconstruction errors obtained using the truncated singular value decomposition are less than those obtained using the singular value decomposition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. A box plot is a visual representation of summary statistics for a given sample. Horizontal lines of each plotted box represent the \(75\%\), \(50\%\) (median), and \(25\%\) quantiles, with outliers plotted as individual crosses or points.

References

  1. Abascal, J.-F.P.J., Arridge, S.R., Bayford, R.H., Holder, D.S.: Comparison of methods for optimal choice of the regularization parameter for linear electrical impedance tomography of brain function. Physiol. Meas. 29, 1319–1334 (2008)

    Article  Google Scholar 

  2. Aster, R.C., Borchers, B., Thurber, C.H.: Parameter Estimation and Inverse Problems, 2nd edn. Elsevier, Amsterdam (2013)

    MATH  Google Scholar 

  3. Bakushinskii, A.B.: Remarks on choosing a regularization parameter using the quasi-optimality and ratio criterion. USSR Comput. Math. Math. Phys. 24(4), 181182 (1984)

    Article  MathSciNet  Google Scholar 

  4. Bauer, F., Lukas, M.A.: Comparing parameter choice methods for regularization of ill-posed problems. Math. Comput. Simul. 81, 1795–1841 (2011)

    Article  MathSciNet  Google Scholar 

  5. Björck, A.: Numerical Methods for Least Squares Problems. Society for Industrial and Applied Mathematics, Philadelphia (1996)

    Book  Google Scholar 

  6. Chung, J., Nagy, J.G., O’Leary, D.P.: A weighted GCV method for Lanczos hybrid regularization. Electron. Trans. Numer. Anal. 28, 149–167 (2008)

    MathSciNet  MATH  Google Scholar 

  7. Drineas, P., Mahoney, M.W.: RandNLA: randomized numerical linear algebra. Commun. ACM 59, 80–90 (2016)

    Article  Google Scholar 

  8. Drineas, P., Mahoney, M.W., Muthukrishnan, S., Sarlós, T.: Faster least squares approximation. Numer. Math. 117, 219–249 (2011)

    Article  MathSciNet  Google Scholar 

  9. Fenu, C., Reichel, L., Rodrigues, G., Sadok, H.: GCV for Tikhonov regularization by partial SVD. BIT Numer. Math. 57, 1019–1039 (2017)

    Article  MathSciNet  Google Scholar 

  10. Gazzola, S., Hansen, P.C., Nagy, J.G.: IR tools: a MATLAB package of iterative regularization methods and large-scale test problems. Numer. Algorithms. https://doi.org/10.1007/s11075-018-0570-7 (2018)

    Article  MathSciNet  Google Scholar 

  11. Golub, G.H., Heath, M., Wahba, G.: Generalized cross-validation as a method for choosing a good ridge parameter. Technometrics 21, 215–223 (1979)

    Article  MathSciNet  Google Scholar 

  12. Golub, G.H., van Loan, C.F.: Matrix Computations, 3rd edn. Johns Hopkins Press, Baltimore (1996)

    MATH  Google Scholar 

  13. Gower, R.M., Richtárik, P.: Randomized iterative methods for linear systems. SIAM J. Matrix Anal. Appl. 36, 1660–1690 (2015)

    Article  MathSciNet  Google Scholar 

  14. Hämäläinen, K., Harhanen, L., Kallonen, A., Kujanpää, A., Niemi, E., Siltanen, S.: Tomographic X-ray data of a walnut. arXiv:1502.04064 (2015)

  15. Hämarik, U., Palm, R., Raus, T.: On minimizationstrategies for choice of the regularization parameter in ill-posed problems. Numer. Funct. Anal. Optim. 30, 924–950 (2009)

    Article  MathSciNet  Google Scholar 

  16. Hämarik, U., Palm, R., Raus, T.: A family of rules for parameter choice in Tikhonov regularization of ill-posed problems with inexact noise level. J. Comput. Appl. Math. 236, 2146–2157 (2012)

    Article  MathSciNet  Google Scholar 

  17. Hanke, M., Hansen, P.C.: Regularization methods for large-scale problems. Surv. Math. Ind. 3, 253–315 (1993)

    MathSciNet  MATH  Google Scholar 

  18. Hansen, J.K., Hogue, J.D., Sander, G.K., Renaut, R.A., Popat, S.C.: Non-negatively constrained least squares and parameter choice by the residual periodogram for the inversion of electrochemical impedance spectroscopy data. J. Comput. Appl. Math. 278, 52–74 (2015)

    Article  MathSciNet  Google Scholar 

  19. Hansen, P.C.: The discrete Picard condition for discrete ill-posed problems. BIT Numer. Math. 30, 658–672 (1990)

    Article  MathSciNet  Google Scholar 

  20. Hansen, P.C.: Regularization tools—a Matlab package for analysis and solution of discrete ill-posed problems. Numer. Algorithms 46, 189–194 (1994)

    Article  MathSciNet  Google Scholar 

  21. Hansen, P.C.: Rank-Deficient and Discrete Ill-Posed Problems. Society for Industrial and Applied Mathematics, Philadelphia (1998)

    Book  Google Scholar 

  22. Hansen, P.C.: Discrete Inverse Problems. Society for Industrial and Applied Mathematics, Philadelphia (2010)

    Book  Google Scholar 

  23. Hofmann, B.: Regularization for Applied Inverse and Ill-posed Problems: A Numerical Approach, Teubner-Texte zur Mathematik. Teubner, B.G., Berlin (1986)

    Book  Google Scholar 

  24. Levin, E., Meltzer, A.Y.: Estimation of the regularization parameter in linear discrete ill-posed problems using the Picard parameter. SIAM J. Sci. Comput. 39, A2741–A2762 (2017)

    Article  MathSciNet  Google Scholar 

  25. Lin, Y., Wohlberg, B., Guo, H.: UPRE method for total variation parameter selection. Signal Process. 90, 2546–2551 (2010)

    Article  Google Scholar 

  26. Mahoney, M.W.: Randomized algorithms for matrices and data. Found. Trends® Mach. Learn. 3, 123–224 (2011)

    MATH  Google Scholar 

  27. Mead, J.L., Renaut, R.A.: A Newton root-finding algorithm for estimating the regularization parameter for solving ill-conditioned least squares problems. Inverse Probl. 25, 025002 (2009)

    Article  MathSciNet  Google Scholar 

  28. Meng, X., Saunders, M.A., Mahoney, M.W.: LSRN: a parallel iterative solver for strongly over- or underdetermined systems. SIAM J. Sci. Comput. 36, C95–C118 (2014)

    Article  MathSciNet  Google Scholar 

  29. Morozov, V.A.: On the solution of functional equations by the method of regularization. Sov. Math. Dokl. 7, 414–417 (1966)

    MathSciNet  MATH  Google Scholar 

  30. Renaut, R.A., Horst, M., Wang, Y., Cochran, D., Hansen, J.: Efficient estimation of regularization parameters via downsampling and the singular value expansion. BIT Numer. Math. 57, 499–529 (2017)

    Article  Google Scholar 

  31. Renaut, R.A., Vatankhah, S., Ardestani, V.E.: Hybrid and iteratively reweighted regularization by unbiased predictive risk and weighted GCV for projected systems. SIAM J. Sci. Comput. 39, B221–B243 (2017)

    Article  MathSciNet  Google Scholar 

  32. Rokhlin, V., Tygert, M.: A fast randomized algorithm for overdetermined linear least-squares regression. Proc. Natl. Acad. Sci. 105, 13212–13217 (2008)

    Article  MathSciNet  Google Scholar 

  33. Stein, C.M.: Estimation of the mean of a multivariate normal distribution. Ann. Stat. 9, 1135–1151 (1981)

    Article  MathSciNet  Google Scholar 

  34. Taroudaki, V., O’Leary, D.P.: Near-optimal spectral filtering and error estimation for solving ill-posed problems. SIAM J. Sci. Comput. 37, A2947–A2968 (2015)

    Article  MathSciNet  Google Scholar 

  35. Toma, A., Sixou, B., Peyrin, F.: Iterative choice of the optimal regularization parameter in tv image restoration. Inverse Probl. Imaging 9, 1171 (2015)

    Article  MathSciNet  Google Scholar 

  36. Vatankhah, S., Ardestani, V.E., Renaut, R.A.: Automatic estimation of the regularization parameter in 2D focusing gravity inversion: application of the method to the Safo manganese mine in the northwest of Iran. J. Geophys. Eng. 11, 045001 (2014)

    Article  Google Scholar 

  37. Vatankhah, S., Ardestani, V.E., Renaut, R.A.: Application of the \(\chi ^2\) principle and unbiased predictive risk estimator for determining the regularization parameter in 3-D focusing gravity inversion. Geophys. J. Int. 200, 265–277 (2015)

    Article  Google Scholar 

  38. Vatankhah, S., Renaut, R.A., Ardestani, V.E.: 3-D projected \(\ell _1\) inversion of gravity data using truncated unbiased predictive risk estimator for regularization parameter estimation. Geophys. J. Int. 210, 1872–1887 (2017)

    Article  Google Scholar 

  39. Vatankhah, S., Renaut, R.A., Ardestani, V.E.: A fast algorithm for regularized focused 3-D inversion of gravity data using the randomized SVD. Geophysics (2018)

  40. Vatankhah, S., Renaut, R.A., Ardestani, V.E.: Total variation regularization of the 3-D gravity inverse problem using a randomized generalized singular value decomposition. Geophys. J. Int. 213, 695–705 (2018)

    Article  Google Scholar 

  41. Vogel, C.: Computational Methods for Inverse Problems. Society for Industrial and Applied Mathematics, Philadelphia (2002)

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ, 85287-1804, USA

    Rosemary A. Renaut & Anthony W. Helmstetter

  2. Institute of Geophysics, University of Tehran, Tehran, Iran

    Saeed Vatankhah

Authors
  1. Rosemary A. Renaut
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anthony W. Helmstetter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saeed Vatankhah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rosemary A. Renaut.

Additional information

Communicated by Lothar Reichel.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rosemary Renaut acknowledges the support of NSF Grant DMS 1418377: “Novel Regularization for Joint Inversion of Nonlinear Problems”.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Renaut, R.A., Helmstetter, A.W. & Vatankhah, S. Unbiased predictive risk estimation of the Tikhonov regularization parameter: convergence with increasing rank approximations of the singular value decomposition. Bit Numer Math 59, 1031–1061 (2019). https://doi.org/10.1007/s10543-019-00762-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10543-019-00762-7

Keywords

Mathematics Subject Classification

Search

Navigation