Handbook of Uncertainty Quantification pp 1-118 | Cite as

# The Bayesian Approach to Inverse Problems

## Abstract

These lecture notes highlight the mathematical and computational structure relating to the formulation of, and development of algorithms for, the Bayesian approach to inverse problems in differential equations. This approach is fundamental in the quantification of uncertainty within applications involving the blending of mathematical models with data. The finite-dimensional situation is described first, along with some motivational examples. Then the development of probability measures on separable Banach space is undertaken, using a random series over an infinite set of functions to construct draws; these probability measures are used as priors in the Bayesian approach to inverse problems. Regularity of draws from the priors is studied in the natural Sobolev or Besov spaces implied by the choice of functions in the random series construction, and the Kolmogorov continuity theorem is used to extend regularity considerations to the space of Hölder continuous functions. Bayes’ theorem is derived in this prior setting, and here interpreted as finding conditions under which the posterior is absolutely continuous with respect to the prior, and determining a formula for the Radon-Nikodym derivative in terms of the likelihood of the data. Having established the form of the posterior, we then describe various properties common to it in the infinite-dimensional setting. These properties include well-posedness, approximation theory, and the existence of maximum a posteriori estimators. We then describe measure-preserving dynamics, again on the infinite-dimensional space, including Markov chain Monte Carlo and sequential Monte Carlo methods, and measure-preserving reversible stochastic differential equations. By formulating the theory and algorithms on the underlying infinite-dimensional space, we obtain a framework suitable for rigorous analysis of the accuracy of reconstructions, of computational complexity, as well as naturally constructing algorithms which perform well under mesh refinement, since they are inherently well defined in infinite dimensions.

## Keywords

Inverse problems Bayesian inversion Tikhonov regularization and MAP estimators Markov chain Monte Carlo Sequential Monte Carlo Langevin stochastic partial differential equations## Notes

### Acknowledgements

The authors are indebted to Martin Hairer for help in the development of these notes, and in particular for considerable help in structuring the Appendix, for the proof of Theorem 28 (which is a slight generalization to Hilbert scales of Theorem 6.16 in [40]) and for the proof of Corollary 5 (which is a generalization of Corollary 3.22 in [40] to the non-Gaussian setting and to Hölder, rather than Lipschitz, functions {*ψ*_{ k }}). They are also grateful to Joris Bierkens, Patrick Conrad, Matthew Dunlop, Shiwei Lan, Yulong Lu, Daniel Sanz-Alonso, Claudia Schillings and Aretha Teckentrup for careful proof-reading of the notes and related comments. AMS is grateful for various hosts who gave him the opportunity to teach this material in short course form at TIFR-Bangalore (Amit Apte), Göttingen (Axel Munk), PKU-Beijing (Teijun Li), ETH-Zurich (Christoph Schwab) and Cambridge CCA (Arieh Iserles), a process which led to refinements of the material; the authors are also grateful to the students on those courses, who provided useful feedback. The authors would also like to thank Sergios Agapiou and Yuan-Xiang Zhang for help in the preparation of these lecture notes, including type-setting, proof-reading, providing the proof of Lemma 3 and delivering problems classes related to the short courses. AMS is also pleased to acknowledge the financial support of EPSRC, ERC and ONR over the last decade, while the research that underpins this work has been developed.

## References

- 1.Adler, R.: The Geometry of Random Fields. SIAM, Philadelphia (1981)MATHGoogle Scholar
- 2.Adams, R.A., Fournier, J.J.: Sobolev Spaces. Pure and Applied Mathematics. Elsevier, Oxford (2003)MATHGoogle Scholar
- 3.Agapiou, S., Larsson, S., Stuart, A.M.: Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems. Stoch. Process. Appl.
**123**, 3828–3860 (2013)MathSciNetCrossRefMATHGoogle Scholar - 4.Agapiou, S., Stuart, A.M., Zhang, Y.X.: Bayesian posterior consistency for linear severely ill-posed inverse problems. J. Inverse Ill-posed Probl.
**22**(3), 297–321 (2014)MathSciNetCrossRefMATHGoogle Scholar - 5.Alexanderian, A., Petra, N., Stadler, G., Ghattas, O.: A fast and scalable method for A-optimal design of experiments for infinite-dimensional Bayesian nonlinear inverse problems. SIAM J. Sci. Comput.
**38**(1), A243–A272 (2016)MathSciNetCrossRefMATHGoogle Scholar - 6.Alexanderian, A., Gloor, P., Ghattas, O.: On Bayesian A- and D-optimal experimental designs in infinite dimensions. http://arxiv.org/abs/1408.6323 (2016)
- 7.Babuska, I., Tempone, R., Zouraris, G.: Galerkin finite element approximations of stochastic elliptic partial differential equations. SIAM J. Numer. Anal.
**42**, 800–825 (2004)MathSciNetCrossRefMATHGoogle Scholar - 8.Banks, H.T., Kunisch, H.: Estimation Techniques for Distributed Parameter Systems. Birkhauser, Boston (1989)CrossRefMATHGoogle Scholar
- 9.Beskos, A., Pinski, F.J., Sanz-Serna, J.-M., Stuart, A.M.: Hybrid Monte-Carlo on Hilbert spaces. Stoch. Process. Appl.
**121**, 2201–2230 (2011)MathSciNetCrossRefMATHGoogle Scholar - 10.Beskos, A., Jasra, A., Muzaffer, E.A., Stuart, A.M.: Sequential Monte Carlo methods for Bayesian elliptic inverse problems. Stat. Comput. (2015)MATHGoogle Scholar
- 11.Bernardo, J., Smith, A.: Bayesian Theory. Wiley, Chichester (1994)CrossRefMATHGoogle Scholar
- 12.Billingsley, P.: Convergence of Probability Measures. Wiley, New York (1968)MATHGoogle Scholar
- 13.Bochner, S.: Integration von Funktionen, deren Werte die Elemente eines Vektorraumes sind. Fund. Math.
**20**, 262–276 (1933)MATHGoogle Scholar - 14.Bogachev, V.I.: Gaussian Measures. Mathematical Surveys and Monographs, vol. 62. American Mathematical Society, Providence (1998)Google Scholar
- 15.Bogachev, V.I.: Measure Theory, vol. I, II. Springer, Berlin (2007)CrossRefMATHGoogle Scholar
- 16.Bui-Thanh, T., Ghattas, O., Martin, J., Stadler, G.: A computational framework for infinite-dimensional Bayesian inverse problems. Part I: the linearized case, with application to global seismic inversion. SIAM J. Sci. Comput.
**35**(6), A2494–A2523 (2013)Google Scholar - 17.Cotter, S., Dashti, M., Robinson, J., Stuart, A.: Bayesian inverse problems for functions and applications to fluid mechanics. Inverse Probl.
**25**. doi:10.1088/0266–5611/25/11/115008 (2009)Google Scholar - 18.Cohen, A., DeVore, R., Schwab, C.: Convergence rates of best
*n*-term Galerkin approximations for a class of elliptic sPDEs. Found. Comput. Math.**10**, 615–646 (2010)MathSciNetCrossRefMATHGoogle Scholar - 19.Cohen, A., DeVore, R., Schwab, Ch.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs. Anal. Appl.
**9**(1), 11–47 (2011)MathSciNetCrossRefMATHGoogle Scholar - 20.Cotter, S., Dashti, M., Stuart, A.: Approximation of Bayesian inverse problems. SIAM J. Numer. Anal.
**48**, 322–345 (2010)MathSciNetCrossRefMATHGoogle Scholar - 21.Cotter, S., Roberts, G., Stuart, A., White, D.: MCMC methods for functions: modifying old algorithms to make them faster. Stat. Sci.
**28**, 424–446 (2013)MathSciNetCrossRefMATHGoogle Scholar - 22.Csiszar, I., Körner, J.: Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press, Cambridge (2011)CrossRefMATHGoogle Scholar
- 23.Dacorogna, B.: Introduction to the Calculus of Variations. Translated from the 1992 French original, 2nd edn. Imperial College Press, London (2009)Google Scholar
- 24.Dashti, M., Harris, S., Stuart, A.: Besov priors for Bayesian inverse problems. Inverse Probl. Imaging
**6**, 183–200 (2012)MathSciNetCrossRefMATHGoogle Scholar - 25.Dashti, M., Stuart, A.: Uncertainty quantification and weak approximation of an elliptic inverse problem. SIAM J. Numer. Anal.
**49**, 2524–2542 (2011)MathSciNetCrossRefMATHGoogle Scholar - 26.Del Moral, P.: Feynman-Kac Formulae. Springer, New York (2004)CrossRefMATHGoogle Scholar
- 27.Da Prato, G.: An introduction to infinite-dimensional analysis. Universitext. Springer, Berlin (2006). Revised and extended from the 2001 original by Da PratoGoogle Scholar
- 28.DaPrato, G., Zabczyk, J.: Stochastic Equations in Infinite Dimensions. Encyclopedia of Mathematics and its Applications, vol. 44. Cambridge University Press, Cambridge (1992)Google Scholar
- 29.DaPrato, G., Zabczyk, J.: Ergodicity for Infinite Dimensional Systems. Cambridge University Press, Cambridge (1996)CrossRefGoogle Scholar
- 30.Dashti, M., Law, K.J.H., Stuart, A.M., Voss, J.: MAP estimators and their consistency in Bayesian nonparametric inverse problems. Inverse Probl.
**29**, 095017 (2013)MathSciNetCrossRefMATHGoogle Scholar - 31.Daubechies, I.: Ten lectures on wavelets. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 61. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (1992)Google Scholar
- 32.Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht/Boston (1996)CrossRefMATHGoogle Scholar
- 33.Evans, L.: Partial Differential Equations. AMS, Providence (1998)MATHGoogle Scholar
- 34.Feldman, J.: Equivalence and perpendicularity of Gaussian processes. Pac. J. Math.
**8**, 699–708 (1958)MathSciNetCrossRefMATHGoogle Scholar - 35.Fernique, X.: Intégrabilité des vecteurs Gaussiens. C. R. Acad. Sci. Paris Sér. A-B
**270**, A1698–A1699 (1970)MathSciNetMATHGoogle Scholar - 36.Franklin, J.: Well-posed stochastic extensions of ill-posed linear problems. J. Math. Anal. Appl.
**31**, 682–716 (1970)MathSciNetCrossRefMATHGoogle Scholar - 37.Gardiner, C.W.: Handbook of stochastic methods. Springer, Berlin, 2nd edn. (1985). For Physics, Chemistry and the Natural SciencesGoogle Scholar
- 38.Gibbs, A., Su, F.: On choosing and bounding probability metrics. Int. Stat. Rev.
**70**, 419–435 (2002)CrossRefMATHGoogle Scholar - 39.Graham, I.G., Kuo, F.Y., Nicholls, J.A., Scheichl, R., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo Finite Element methods for Elliptic PDEs with Log-normal Random Coefficients, Seminar for Applied Mathematics, ETH, SAM Report 2013–14 (2013)Google Scholar
- 40.Hairer, M.: Introduction to Stochastic PDEs. Lecture Notes http://arxiv.org/abs/0907.4178 (2009)
- 41.Hairer, M., Stuart, A.M., Voss, J.: Analysis of SPDEs arising in path sampling, part II: the nonlinear case. Ann. Appl. Probab.
**17**, 1657–1706 (2007)MathSciNetCrossRefMATHGoogle Scholar - 42.Hairer, M., Stuart, A., Voss, J.: Sampling conditioned hypoelliptic diffusions. Ann. Appl. Probab.
**21**(2), 669–698 (2011)MathSciNetCrossRefMATHGoogle Scholar - 43.Hairer, M., Stuart, A., Voss, J., Wiberg, P.: Analysis of SPDEs arising in path sampling. Part I: the Gaussian case. Comm. Math. Sci.
**3**, 587–603 (2005)MathSciNetMATHGoogle Scholar - 44.Hairer, M., Stuart, A., Vollmer, S.: Spectral gaps for a Metropolis-Hastings algorithm in infinite dimensions. Ann. Appl. Probab.
**24**(6), 2455–2490 (2014)MathSciNetCrossRefMATHGoogle Scholar - 45.Hájek, Y.: On a property of normal distribution of any stochastic process. Czechoslov. Math. J.
**8**(83), 610–618 (1958)MathSciNetMATHGoogle Scholar - 46.Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika
**57**(1), 97–109 (1970)MathSciNetCrossRefMATHGoogle Scholar - 47.Helin, T., Burger, M.: Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems. Inverse Probl.
**31**(8), 085009 (2015)MathSciNetCrossRefMATHGoogle Scholar - 48.Hildebrandt, T.H.: Integration in abstract spaces. Bull. Am. Math. Soc.
**59**, 111–139 (1953)MathSciNetCrossRefMATHGoogle Scholar - 49.Kahane, J.-P.: Some Random Series of Functions. Cambridge Studies in Advanced Mathematics, vol. 5. Cambridge University Press, Cambridge (1985)Google Scholar
- 50.Kantas, N., Beskos, A., Jasra, A.: Sequential Monte Carlo methods for high-dimensional inverse problems: a case study for the Navier-Stokes equations. arXiv preprint, arXiv:1307.6127Google Scholar
- 51.Kirsch, A.: An Introduction to the Mathematical Theory of Inverse Problems. Springer, New York (1996)CrossRefMATHGoogle Scholar
- 52.Kühn, T., Liese, F.: A short proof of the Hájek-Feldman theorem. Teor. Verojatnost. i Primenen.
**23**(2), 448–450 (1978)MathSciNetMATHGoogle Scholar - 53.Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Applied Mathematical Sciences, vol. 160. Springer, New York (2005)Google Scholar
- 54.Kallenberg, O.: Foundations of Modern Probability, 2nd edn. Probability and its Applications. Springer, New York, (2002)CrossRefMATHGoogle Scholar
- 55.Knapik, B., van Der Vaart, A., van Zanten, J.: Bayesian inverse problems with Gaussian priors. Ann. Stat.
**39**(5), 2626–2657 (2011)MathSciNetCrossRefMATHGoogle Scholar - 56.Knapik, B., van der Vaart, A., van Zanten, J.H.: Bayesian recovery of the initial condition for the heat equation Commun. Stat. Theory Methods
**42**, 1294–1313 (2013)CrossRefMATHGoogle Scholar - 57.Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo methods for very high dimensional integration: the standard (weighted Hilbert space) setting and beyond. ANZIAM J.
**53**, 1–37 (2011)MathSciNetCrossRefMATHGoogle Scholar - 58.Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo Finite Element Methods for a Class of Elliptic Partial Differential Equations with Random Coefficients. SIAM J. Numer. Anal.
**50**(6), 3351–3374 (2012)MathSciNetCrossRefMATHGoogle Scholar - 59.Kuo, F.Y., Sloan, I.H.: Lifting the curse of dimensionality. Not. AMS
**52**(11), 1320–1328 (2005)MathSciNetMATHGoogle Scholar - 60.Kuo, F.Y., Sloan, I.H., Wasilikowski, G.W., Waterhouse, B.J.: Randomly shifted lattice rules with the optimal rate of convergence for unbounded integrands. J. Complex.
**26**, 135–160 (2010)MathSciNetCrossRefMATHGoogle Scholar - 61.Lasanen, S.: Discretizations of generalized random variables with applications to inverse problems. Ann. Acad. Sci. Fenn. Math. Dissertation, University of Oulu, 130 (2002)Google Scholar
- 62.Lasanen, S.: Measurements and infinite-dimensional statistical inverse theory. PAMM
**7**, 1080101–1080102 (2007)CrossRefGoogle Scholar - 63.Lasanen, S.: Non-Gaussian statistical inverse problems part II: posterior convergence for approximated unknowns. Inverse Probl. Imaging
**6**(2), 267 (2012)MathSciNetCrossRefMATHGoogle Scholar - 64.Lasanen, S.: Non-Gaussian statistical inverse problems. Part I: posterior distributions. Inverse Probl. Imaging
**6**(2), 215–266 (2012)MathSciNetMATHGoogle Scholar - 65.Lasanen, S.: Non-Gaussian statistical inverse problems. Part II: posterior distributions. Inverse Probl. Imaging
**6**(2), 267–287 (2012)MathSciNetMATHGoogle Scholar - 66.Ledoux, M.: Isoperimetry and Gaussian analysis. In: Lectures on Probability Theory and Statistics (Saint-Flour, 1994). Lecture Notes in Mathematics, vol. 1648, pp. 165–294. Springer, Berlin (1996)Google Scholar
- 67.Lifshits, M.: Gaussian Random Functions. Mathematics and its Applications, vol. 322. Kluwer, Dordrecht (1995)Google Scholar
- 68.Lehtinen, M.S., Päivärinta, L., Somersalo, E.: Linear inverse problems for generalised random variables. Inverse Probl.
**5**(4), 599–612 (1989). http://stacks.iop.org/0266-5611/5/599 MathSciNetCrossRefMATHGoogle Scholar - 69.Lassas, M., Saksman, E., Siltanen, S.: Discretization-invariant Bayesian inversion and Besov space priors. Inverse Probl. Imaging
**3**, 87–122 (2009)MathSciNetCrossRefMATHGoogle Scholar - 70.Lunardi, A.: Analytic Semigroups and Optimal Regularity in Parabolic Problems. Progress in Nonlinear Differential Equations and their Applications, vol. 16. Birkhäuser Verlag, Basel (1995)Google Scholar
- 71.Mandelbaum, A.: Linear estimators and measurable linear transformations on a Hilbert space. Z. Wahrsch. Verw. Gebiete
**65**(3), 385–397 (1984). http://dx.doi.org/10.1007/BF00533743 MathSciNetCrossRefMATHGoogle Scholar - 72.Mattingly, J., Pillai, N., Stuart, A.: Diffusion limits of the random walk Metropolis algorithm in high dimensions. Ann. Appl. Probl.
**22**, 881–930 (2012)MathSciNetCrossRefMATHGoogle Scholar - 73.Metropolis, N., Rosenbluth, R., Teller, M., Teller, E.: Equations of state calculations by fast computing machines. J. Chem. Phys.
**21**, 1087–1092 (1953)CrossRefGoogle Scholar - 74.Meyer, Y.: Wavelets and operators. Translated from the 1990 French original by D.H. Salinger. Cambridge Studies in Advanced Mathematics, vol. 37. Cambridge University Press, Cambridge (1992)Google Scholar
- 75.Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability. Communications and Control Engineering Series. Springer, London (1993)CrossRefMATHGoogle Scholar
- 76.Neal, R.: Regression and classification using Gaussian process priors. http://www.cs.toronto.edu/~radford/valencia.abstract.html (1998)
- 77.Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1994)MATHGoogle Scholar
- 78.Norris, J.: Markov Chains. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (1998)MATHGoogle Scholar
- 79.Oksendal, B.: Stochastic Differential Equations. An Introduction with Applications. Universitext, 6th edn. Springer, Berlin (2003)Google Scholar
- 80.Pazy, A.: Semigroups of Linear Operators and Applications to Partial Differential Equations. Springer, New York (1983)CrossRefMATHGoogle Scholar
- 81.Petra, N., Martin, J., Stadler, G., Ghattas, O.: A computational framework for infinite-dimensional Bayesian inverse problems. Part II: stochastic Newton MCMC with application to ice sheet flow inverse problems. SIAM J. Sci. Comput.
**36**(4), A1525–A1555 (2014)MathSciNetMATHGoogle Scholar - 82.Pillai, N.S., Stuart, A.M., Thiery, A.H.: Noisy gradient flow from a random walk in Hilbert space. Stoch. PDEs: Anal. Comput.
**2**, 196–232 (2014)MathSciNetCrossRefMATHGoogle Scholar - 83.Pinski, F., Stuart, A.: Transition paths in molecules at finite temperature. J. Chem. Phys.
**132**, 184104 (2010)CrossRefGoogle Scholar - 84.Rebeschini, P., van Handel, R.: Can local particle filters beat the curse of dimensionality? Ann. Appl. Probab.
**25**(5), 2809–2866 (2015)MathSciNetCrossRefMATHGoogle Scholar - 85.Revuz, D., Yor, M.: Continuous Martingales and Brownian Motion. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 293, 2nd edn. Springer, Berlin (1994)Google Scholar
- 86.Richter, G.: An inverse problem for the steady state diffusion equation. SIAM J. Appl. Math.
**41**(2), 210–221 (1981)MathSciNetCrossRefMATHGoogle Scholar - 87.Robinson, J.C.: Infinite-Dimensional Dynamical Systems. Cambridge Texts in Applied Mathematics. Cambridge University Press, Cambridge (2001)CrossRefMATHGoogle Scholar
- 88.Rudin, W.: Real and Complex Analysis, 3rd edn. McGraw-Hill, New York (1987)MATHGoogle Scholar
- 89.Schillings, C., Schwab, C.: Sparse, adaptive Smolyak quadratures for Bayesian inverse problems. Inverse Probl.
**29**, 065011 (2013)MathSciNetCrossRefMATHGoogle Scholar - 90.Schwab, C., Stuart, A.: Sparse deterministic approximation of Bayesian inverse problems. Inverse Probl.
**28**, 045003 (2012)MathSciNetCrossRefMATHGoogle Scholar - 91.Strauss, W.A.: Partial differential equations. An introduction, 2nd edn. Wiley, Chichester (2008)Google Scholar
- 92.Stuart, A.M.: Inverse problems: a Bayesian perspective. Acta Numer.
**19**, 451–559 (2010)MathSciNetCrossRefMATHGoogle Scholar - 93.Stuart, A.M.: Uncertainty quantification in Bayesian inversion. ICM2014. Invited Lecture (2014)Google Scholar
- 94.Tierney, L.: A note on Metropolis-Hastings kernels for general state spaces. Ann. Appl. Probab.
**8**(1), 1–9 (1998)MathSciNetCrossRefMATHGoogle Scholar - 95.Triebel, H.: Theory of function spaces. Mathematik und ihre Anwendungen in Physik und Technik [Mathematics and its Applications in Physics and Technology], vol. 38. Akademische Verlagsgesellschaft Geest & Portig K.-G., Leipzig (1983)Google Scholar
- 96.Triebel, H.: Theory of Function Spaces. II. Monographs in Mathematics, vol. 84. Birkhäuser Verlag, Basel (1992)Google Scholar
- 97.Triebel, H.: Theory of Function Spaces. III. Monographs in Mathematics, vol. 100. Birkhäuser Verlag, Basel (2006)Google Scholar
- 98.Villani, C.: Topics in Optimal Transportation. Graduate Studies in Mathematics, vol. 58. American Mathematical Society, Providence (2003)Google Scholar
- 99.Vollmer, S.: Posterior consistency for Bayesian inverse problems through stability and regression results. Inverse Probl.
**29**, 125011 (2013)MathSciNetCrossRefGoogle Scholar - 100.Yosida, K.: Functional Analysis. Classics in Mathematics. Springer, Berlin (1995). Reprint of the sixth (1980) edition.Google Scholar
- 101.Yin, G., Zhang, Q.: Continuous-Time Markov Chains and Applications. Applications of Mathematics (New York), vol. 37. Springer, New York (1998)Google Scholar