Vietnam Journal of Mathematics

, Volume 44, Issue 1, pp 49–69 | Cite as

Algorithms that Satisfy a Stopping Criterion, Probably

  • Uri AscherEmail author
  • Farbod Roosta-Khorasani


Iterative numerical algorithms are typically equipped with a stopping criterion, where the iteration process is terminated when some error or misfit measure is deemed to be below a given tolerance. This is a useful setting for comparing algorithm performance, among other purposes. However, in practical applications, a precise value for such a tolerance is rarely known; rather, only some possibly vague idea of the desired quality of the numerical approximation is at hand. In this review paper, we first discuss four case studies from different areas of numerical computation, where uncertainty in the error tolerance value and in the stopping criterion is revealed in different ways. This leads us to think of approaches to relax the notion of exactly satisfying a tolerance value. We then concentrate on a probabilistic relaxation of the given tolerance in the context of our fourth case study which allows, for instance, derivation of proven bounds on the sample size of certain Monte Carlo methods. We describe an algorithm that becomes more efficient in a controlled way as the uncertainty in the tolerance increases and demonstrate this in the context of some particular applications of inverse problems.


Error tolerance Mathematical software Iterative method Inverse problem Monte Carlo method Trace estimation Large scale simulation DC resistivity 

Mathematics Subject Classification (2010)

65C20 65C05 65M32 65L05 65F10 



The authors thank Eldad Haber and Arieh Iserles for several fruitful discussions and the anonymous referees for their comments.


  1. 1.
    Achlioptas, D.: Database-friendly random projections. In: Proceedings of the 20th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp 274–281 (2001)Google Scholar
  2. 2.
    Akaike, H.: On a successive transformation of probability distribution and its application to the analysis of the optimum gradient method. Ann. Inst. Stat. Math. Tokyo 11, 1–16 (1959)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Arridge, S.R.: Optical tomography in medical imaging. Inverse Probl 15, R41 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Ascher, U., Greif, C.: First Course in Numerical Methods. Computational Science and Engineering. SIAM, Philadelphia (2011)CrossRefzbMATHGoogle Scholar
  5. 5.
    Ascher, U., Mattheij, R., Russell, R.: Numerical Solution of Boundary Value Problems for Ordinary Differential Equations. Classics in Applied Mathematics. SIAM, Philadelphia (1995)CrossRefzbMATHGoogle Scholar
  6. 6.
    Ascher, U., Petzold, L.: Computer Methods for Ordinary Differential and Differential-Algebraic Equations. SIAM (1998)Google Scholar
  7. 7.
    Ascher, U., Reich, S.: The midpoint scheme and variants for hamiltonian systems: advantages and pitfalls. SIAM J. Sci. Comput. 21, 1045–1065 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Avron, H., Toledo, S.: Randomized algorithms for estimating the trace of an implicit symmetric positive semi-definite matrix. JACM 58, 8 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Barzilai, J., Borwein, J.: Two point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    van den Berg, E., Friedlander, M.P.: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput. 31, 890–912 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Boas, D.A., Brooks, D.H., Miller, E.L., DiMarzio, C.A., Kilmer, M., Gaudette, R.J., Zhang, Q.: Imaging the body with diffuse optical tomography. IEEE Signal Process. Mag. 18, 57–75 (2001)CrossRefGoogle Scholar
  12. 12.
    Borcea, L., Berryman, J.G., Papanicolaou, G.C.: High-contrast impedance tomography. Inverse Probl. 12, 835–858 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Cao, Y., Petzold, L.: A posteriori error estimation and global error control for ordinary differential equations by the adjoint method. SIAM J. Sci. Comput. 26, 359–374 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Cheney, M., Isaacson, D., Newell, J.C.: Electrical impedance tomography. SIAM Rev 41, 85–101 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Dahlquist, G., Björck, A.: Numerical Methods. Prentice-Hall, NJ (1974)zbMATHGoogle Scholar
  16. 16.
    Dai, Y.-H., Fletcher, R.: Projected Barzilai–Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100, 21–47 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    van den Doel, K., Ascher, U.: The chaotic nature of faster gradient descent methods. J. Sci. Comput. 51, 560–581 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    van den Doel, K., Ascher, U., Haber, E.: The lost honour of 2-based regularization. In: Cullen, M. et al. (eds.) Large Scale Inverse Problems: Computational Method and Applications in the Earth Sciences, pp 181–203. De Gruyter, Berlin (2013)Google Scholar
  19. 19.
    Dorn, O., Miller, E.L., Rappaport, C.M.: A shape reconstruction method for electromagnetic tomography using adjoint fields and level sets. Inverse Probl. 16, 1119–1156 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Engl, H.W., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer Academic Publishers, Dordrecht (1996)CrossRefzbMATHGoogle Scholar
  21. 21.
    Fichtner, A.: Full Seismic Waveform Modeling and Inversion. Springer, Berlin–Heidelberg (2011)CrossRefGoogle Scholar
  22. 22.
    Friedlander, A., Martínez, J.M., Molina, B., Raydan, M.: Gradient method with retard and generalizations. SIAM J. Numer. Anal. 36, 275–289 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Gao, H., Osher, S., Zhao, H.: Quantitative photoacoustic tomography. In: Ammari, H (ed.) Mathematical Modeling in Biomedical Imaging II, pp 131–158. Springer, Berlin–Heidelberg (2012)Google Scholar
  24. 24.
    Greenbaum, A.: Iterative Methods for Solving Linear Systems. SIAM, Philadelphia (1997)CrossRefzbMATHGoogle Scholar
  25. 25.
    Haber, E., Ascher, U., Oldenburg, D.: Inversion of 3D electromagnetic data in frequency and time domain using an inexact all-at-once approach. Geophysics 69, 1216–1228 (2004)CrossRefGoogle Scholar
  26. 26.
    Haber, E., Chung, M., Herrmann, F.: An effective method for parameter estimation with PDE constraints with multiple right-hand sides. SIAM J. Optim. 22, 739–757 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Haber, E., Heldmann, S., Ascher, U.: Adaptive finite volume method for distributed non-smooth parameter identification. Inverse Probl. 23, 1659–1676 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Hairer, E., Lubich, C., Wanner, G.: Geometric Numerical Integration. Springer, Berlin–Heidelberg (2002)CrossRefzbMATHGoogle Scholar
  29. 29.
    Hairer, E., Norsett, S., Wanner, G.: Solving Ordinary Differential Equations I. Springer, Berlin–Heidelberg (1993)zbMATHGoogle Scholar
  30. 30.
    Hairer, E., Wanner, G.: Solving Ordinary Differential Equations II. Springer, Berlin–Heidelberg (1996)CrossRefzbMATHGoogle Scholar
  31. 31.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, New York (2001)CrossRefzbMATHGoogle Scholar
  32. 32.
    Heath, M.: Scientific Computing: An Introductory Survey, 2nd edn. McGraw-Hill, New York (2002)zbMATHGoogle Scholar
  33. 33.
    Herrmann, F.J., Erlangga, Y.A., Lin, T.T.: Compressive simultaneous full-waveform simulation. Geophysics 74, A35–A40 (2009)CrossRefGoogle Scholar
  34. 34.
    Higham, D.J.: Global error versus tolerance for explicit Runge–Kutta methods. IMA. J. Numer. Anal 11, 457–480 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  35. 35.
    Holodnak, J.T., Ipsen, I.C.F.: Randomized approximation of the Gram matrix: exact computation and probabilistic bounds. SIAM J. Matrix Anal. Appl. 36, 110–137 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Hutchinson, M.F.: A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines. Commun. Stat. Simul. Comput. 19, 433–450 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Ipsen, I.C.F., Wentworth, T.: The effect of coherence on sampling from matrices with orthonormal columns, and preconditioned least squares problems. SIAM J. Matrix Anal. Appl. 35, 1490–1520 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Kaipo, J., Somersalo, E.: Statistical and Computational Inverse Problems. Springer, New York (2005)Google Scholar
  39. 39.
    Leimkuhler, B., Reich, S.: Simulating Hamiltonian Dynamics. Cambridge University Press, Cambridge (2004)zbMATHGoogle Scholar
  40. 40.
    Morozov, V.A.: Methods for Solving Incorrectly Posed Problems. Springer, New York (1984)CrossRefGoogle Scholar
  41. 41.
    Neelamani, R., Krohn, C.E., Krebs, J.R., Rohmberg, J., Deffenbaugh, M., Anderson, J.: Efficient seismic forward modeling and acquisition using simultaneous random sources and sparsity. Geophysics 75, WB15–WB27 (2010)CrossRefGoogle Scholar
  42. 42.
    Newman, G.A., Alumbaugh, D.L.: Frequency-domain modelling of airborne electromagnetic responses using staggered finite differences. Geophys. Prospect. 43, 1021–1042 (1995)CrossRefGoogle Scholar
  43. 43.
    Oldenburg, D.W., Haber, E., Shekhtman, R.: Three dimensional inversion of multi-source time domain electromagnetic data. Geophysics 78, E47–E57 (2013)CrossRefGoogle Scholar
  44. 44.
    Pidlisecky, A., Haber, E., Knight, R.: RESINVM3D: A 3D resistivity inversion package. Geophysics 72, H1—H10 (2007)CrossRefGoogle Scholar
  45. 45.
    Raydan, M.R.: Convergence properties of the Barzilai and Borwein gradient method. PhD thesis, Rice University. Houston, Texas (1991)Google Scholar
  46. 46.
    Roosta-Khorasani, F., Ascher, U.: Improved bounds on sample size for implicit matrix trace estimators. Found. Comput. Math. (2014). doi: 10.1007/s10208-014-9220-1
  47. 47.
    Roosta-Khorasani, F., Székely, G.J., Ascher, U.: Assessing stochastic algorithms for large scale nonlinear least squares problems using extremal probabilities of linear combinations of gamma random variables. SIAM/ASA J. Uncertain. Quantif. 3, 61–90 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Roosta-Khorasani, F., van den Doel, K., Ascher, U.: Stochastic algorithms for inverse problems involving PDEs and many measurements. SIAM J. Sci. Comput. 36, S3–S22 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Shapiro, A., Dentcheva, D., Ruszczynski, D.: Lectures on Stochastic Programming: Modeling and Theory. SIAM, Piladelphia (2009)CrossRefzbMATHGoogle Scholar
  50. 50.
    Smith, N.C., Vozoff, K.: Two dimensional DC resistivity inversion for dipole dipole data. IEEE Trans. Geosci. Remote Sens. GE-22, 21–28 (1984)Google Scholar
  51. 51.
    Trottenberg, U., Oosterlee, C.W., Schüller, A.: Multigrid. Academic Press (2001)Google Scholar
  52. 52.
    Vogel, C.: Computational Methods for Inverse Problem. SIAM, Philadelphia (2002)CrossRefzbMATHGoogle Scholar
  53. 53.
    Young, J., Ridzal, D.: An application of random projection to parameter estimation in partial differential equations. SIAM J. Sci. Comput. 34, A2344–A2365 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  54. 54.
    Yuan, Z., Jiang, H.: Quantitative photoacoustic tomography: recovery of optical absorption coefficient maps of heterogeneous media. Appl. Phys. Lett. 88, 231101 (2006)CrossRefGoogle Scholar

Copyright information

© Vietnam Academy of Science and Technology (VAST) and Springer Science+Business Media Singapore 2015

Authors and Affiliations

  1. 1.Computer Science Institute (ICSI) and Department of StatisticsUniversity of CaliforniaBerkeleyUSA

Personalised recommendations