Handbook of Uncertainty Quantification pp 1-118 | Cite as

# The Bayesian Approach to Inverse Problems

## Abstract

These lecture notes highlight the mathematical and computational structure relating to the formulation of, and development of algorithms for, the Bayesian approach to inverse problems in differential equations. This approach is fundamental in the quantification of uncertainty within applications involving the blending of mathematical models with data. The finite-dimensional situation is described first, along with some motivational examples. Then the development of probability measures on separable Banach space is undertaken, using a random series over an infinite set of functions to construct draws; these probability measures are used as priors in the Bayesian approach to inverse problems. Regularity of draws from the priors is studied in the natural Sobolev or Besov spaces implied by the choice of functions in the random series construction, and the Kolmogorov continuity theorem is used to extend regularity considerations to the space of Hölder continuous functions. Bayes’ theorem is derived in this prior setting, and here interpreted as finding conditions under which the posterior is absolutely continuous with respect to the prior, and determining a formula for the Radon-Nikodym derivative in terms of the likelihood of the data. Having established the form of the posterior, we then describe various properties common to it in the infinite-dimensional setting. These properties include well-posedness, approximation theory, and the existence of maximum a posteriori estimators. We then describe measure-preserving dynamics, again on the infinite-dimensional space, including Markov chain Monte Carlo and sequential Monte Carlo methods, and measure-preserving reversible stochastic differential equations. By formulating the theory and algorithms on the underlying infinite-dimensional space, we obtain a framework suitable for rigorous analysis of the accuracy of reconstructions, of computational complexity, as well as naturally constructing algorithms which perform well under mesh refinement, since they are inherently well defined in infinite dimensions.

## Keywords

Inverse problems Bayesian inversion Tikhonov regularization and MAP estimators Markov chain Monte Carlo Sequential Monte Carlo Langevin stochastic partial differential equations## Notes

### Acknowledgements

The authors are indebted to Martin Hairer for help in the development of these notes, and in particular for considerable help in structuring the Appendix, for the proof of Theorem 28 (which is a slight generalization to Hilbert scales of Theorem 6.16 in [40]) and for the proof of Corollary 5 (which is a generalization of Corollary 3.22 in [40] to the non-Gaussian setting and to Hölder, rather than Lipschitz, functions {*ψ*_{ k }}). They are also grateful to Joris Bierkens, Patrick Conrad, Matthew Dunlop, Shiwei Lan, Yulong Lu, Daniel Sanz-Alonso, Claudia Schillings and Aretha Teckentrup for careful proof-reading of the notes and related comments. AMS is grateful for various hosts who gave him the opportunity to teach this material in short course form at TIFR-Bangalore (Amit Apte), Göttingen (Axel Munk), PKU-Beijing (Teijun Li), ETH-Zurich (Christoph Schwab) and Cambridge CCA (Arieh Iserles), a process which led to refinements of the material; the authors are also grateful to the students on those courses, who provided useful feedback. The authors would also like to thank Sergios Agapiou and Yuan-Xiang Zhang for help in the preparation of these lecture notes, including type-setting, proof-reading, providing the proof of Lemma 3 and delivering problems classes related to the short courses. AMS is also pleased to acknowledge the financial support of EPSRC, ERC and ONR over the last decade, while the research that underpins this work has been developed.

## References

- 1.Adler, R.: The Geometry of Random Fields. SIAM, Philadelphia (1981)zbMATHGoogle Scholar
- 2.Adams, R.A., Fournier, J.J.: Sobolev Spaces. Pure and Applied Mathematics. Elsevier, Oxford (2003)zbMATHGoogle Scholar
- 3.Agapiou, S., Larsson, S., Stuart, A.M.: Posterior contraction rates for the Bayesian approach to linear ill-posed inverse problems. Stoch. Process. Appl.
**123**, 3828–3860 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 4.Agapiou, S., Stuart, A.M., Zhang, Y.X.: Bayesian posterior consistency for linear severely ill-posed inverse problems. J. Inverse Ill-posed Probl.
**22**(3), 297–321 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 5.Alexanderian, A., Petra, N., Stadler, G., Ghattas, O.: A fast and scalable method for A-optimal design of experiments for infinite-dimensional Bayesian nonlinear inverse problems. SIAM J. Sci. Comput.
**38**(1), A243–A272 (2016)MathSciNetCrossRefzbMATHGoogle Scholar - 6.Alexanderian, A., Gloor, P., Ghattas, O.: On Bayesian A- and D-optimal experimental designs in infinite dimensions. http://arxiv.org/abs/1408.6323 (2016)
- 7.Babuska, I., Tempone, R., Zouraris, G.: Galerkin finite element approximations of stochastic elliptic partial differential equations. SIAM J. Numer. Anal.
**42**, 800–825 (2004)MathSciNetCrossRefzbMATHGoogle Scholar - 8.Banks, H.T., Kunisch, H.: Estimation Techniques for Distributed Parameter Systems. Birkhauser, Boston (1989)CrossRefzbMATHGoogle Scholar
- 9.Beskos, A., Pinski, F.J., Sanz-Serna, J.-M., Stuart, A.M.: Hybrid Monte-Carlo on Hilbert spaces. Stoch. Process. Appl.
**121**, 2201–2230 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 10.Beskos, A., Jasra, A., Muzaffer, E.A., Stuart, A.M.: Sequential Monte Carlo methods for Bayesian elliptic inverse problems. Stat. Comput. (2015)zbMATHGoogle Scholar
- 11.Bernardo, J., Smith, A.: Bayesian Theory. Wiley, Chichester (1994)CrossRefzbMATHGoogle Scholar
- 12.Billingsley, P.: Convergence of Probability Measures. Wiley, New York (1968)zbMATHGoogle Scholar
- 13.Bochner, S.: Integration von Funktionen, deren Werte die Elemente eines Vektorraumes sind. Fund. Math.
**20**, 262–276 (1933)zbMATHGoogle Scholar - 14.Bogachev, V.I.: Gaussian Measures. Mathematical Surveys and Monographs, vol. 62. American Mathematical Society, Providence (1998)Google Scholar
- 15.Bogachev, V.I.: Measure Theory, vol. I, II. Springer, Berlin (2007)CrossRefzbMATHGoogle Scholar
- 16.Bui-Thanh, T., Ghattas, O., Martin, J., Stadler, G.: A computational framework for infinite-dimensional Bayesian inverse problems. Part I: the linearized case, with application to global seismic inversion. SIAM J. Sci. Comput.
**35**(6), A2494–A2523 (2013)Google Scholar - 17.Cotter, S., Dashti, M., Robinson, J., Stuart, A.: Bayesian inverse problems for functions and applications to fluid mechanics. Inverse Probl.
**25**. doi:10.1088/0266–5611/25/11/115008 (2009)Google Scholar - 18.Cohen, A., DeVore, R., Schwab, C.: Convergence rates of best
*n*-term Galerkin approximations for a class of elliptic sPDEs. Found. Comput. Math.**10**, 615–646 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 19.Cohen, A., DeVore, R., Schwab, Ch.: Analytic regularity and polynomial approximation of parametric and stochastic elliptic PDEs. Anal. Appl.
**9**(1), 11–47 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 20.Cotter, S., Dashti, M., Stuart, A.: Approximation of Bayesian inverse problems. SIAM J. Numer. Anal.
**48**, 322–345 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 21.Cotter, S., Roberts, G., Stuart, A., White, D.: MCMC methods for functions: modifying old algorithms to make them faster. Stat. Sci.
**28**, 424–446 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 22.Csiszar, I., Körner, J.: Information Theory: Coding Theorems for Discrete Memoryless Systems. Cambridge University Press, Cambridge (2011)CrossRefzbMATHGoogle Scholar
- 23.Dacorogna, B.: Introduction to the Calculus of Variations. Translated from the 1992 French original, 2nd edn. Imperial College Press, London (2009)Google Scholar
- 24.Dashti, M., Harris, S., Stuart, A.: Besov priors for Bayesian inverse problems. Inverse Probl. Imaging
**6**, 183–200 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 25.Dashti, M., Stuart, A.: Uncertainty quantification and weak approximation of an elliptic inverse problem. SIAM J. Numer. Anal.
**49**, 2524–2542 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 26.Del Moral, P.: Feynman-Kac Formulae. Springer, New York (2004)CrossRefzbMATHGoogle Scholar
- 27.Da Prato, G.: An introduction to infinite-dimensional analysis. Universitext. Springer, Berlin (2006). Revised and extended from the 2001 original by Da PratoGoogle Scholar
- 28.DaPrato, G., Zabczyk, J.: Stochastic Equations in Infinite Dimensions. Encyclopedia of Mathematics and its Applications, vol. 44. Cambridge University Press, Cambridge (1992)Google Scholar
- 29.DaPrato, G., Zabczyk, J.: Ergodicity for Infinite Dimensional Systems. Cambridge University Press, Cambridge (1996)CrossRefGoogle Scholar
- 30.Dashti, M., Law, K.J.H., Stuart, A.M., Voss, J.: MAP estimators and their consistency in Bayesian nonparametric inverse problems. Inverse Probl.
**29**, 095017 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 31.Daubechies, I.: Ten lectures on wavelets. CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 61. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (1992)Google Scholar
- 32.Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht/Boston (1996)CrossRefzbMATHGoogle Scholar
- 33.Evans, L.: Partial Differential Equations. AMS, Providence (1998)zbMATHGoogle Scholar
- 34.Feldman, J.: Equivalence and perpendicularity of Gaussian processes. Pac. J. Math.
**8**, 699–708 (1958)MathSciNetCrossRefzbMATHGoogle Scholar - 35.Fernique, X.: Intégrabilité des vecteurs Gaussiens. C. R. Acad. Sci. Paris Sér. A-B
**270**, A1698–A1699 (1970)MathSciNetzbMATHGoogle Scholar - 36.Franklin, J.: Well-posed stochastic extensions of ill-posed linear problems. J. Math. Anal. Appl.
**31**, 682–716 (1970)MathSciNetCrossRefzbMATHGoogle Scholar - 37.Gardiner, C.W.: Handbook of stochastic methods. Springer, Berlin, 2nd edn. (1985). For Physics, Chemistry and the Natural SciencesGoogle Scholar
- 38.Gibbs, A., Su, F.: On choosing and bounding probability metrics. Int. Stat. Rev.
**70**, 419–435 (2002)CrossRefzbMATHGoogle Scholar - 39.Graham, I.G., Kuo, F.Y., Nicholls, J.A., Scheichl, R., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo Finite Element methods for Elliptic PDEs with Log-normal Random Coefficients, Seminar for Applied Mathematics, ETH, SAM Report 2013–14 (2013)Google Scholar
- 40.Hairer, M.: Introduction to Stochastic PDEs. Lecture Notes http://arxiv.org/abs/0907.4178 (2009)
- 41.Hairer, M., Stuart, A.M., Voss, J.: Analysis of SPDEs arising in path sampling, part II: the nonlinear case. Ann. Appl. Probab.
**17**, 1657–1706 (2007)MathSciNetCrossRefzbMATHGoogle Scholar - 42.Hairer, M., Stuart, A., Voss, J.: Sampling conditioned hypoelliptic diffusions. Ann. Appl. Probab.
**21**(2), 669–698 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 43.Hairer, M., Stuart, A., Voss, J., Wiberg, P.: Analysis of SPDEs arising in path sampling. Part I: the Gaussian case. Comm. Math. Sci.
**3**, 587–603 (2005)MathSciNetzbMATHGoogle Scholar - 44.Hairer, M., Stuart, A., Vollmer, S.: Spectral gaps for a Metropolis-Hastings algorithm in infinite dimensions. Ann. Appl. Probab.
**24**(6), 2455–2490 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 45.Hájek, Y.: On a property of normal distribution of any stochastic process. Czechoslov. Math. J.
**8**(83), 610–618 (1958)MathSciNetzbMATHGoogle Scholar - 46.Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika
**57**(1), 97–109 (1970)MathSciNetCrossRefzbMATHGoogle Scholar - 47.Helin, T., Burger, M.: Maximum a posteriori probability estimates in infinite-dimensional Bayesian inverse problems. Inverse Probl.
**31**(8), 085009 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 48.Hildebrandt, T.H.: Integration in abstract spaces. Bull. Am. Math. Soc.
**59**, 111–139 (1953)MathSciNetCrossRefzbMATHGoogle Scholar - 49.Kahane, J.-P.: Some Random Series of Functions. Cambridge Studies in Advanced Mathematics, vol. 5. Cambridge University Press, Cambridge (1985)Google Scholar
- 50.Kantas, N., Beskos, A., Jasra, A.: Sequential Monte Carlo methods for high-dimensional inverse problems: a case study for the Navier-Stokes equations. arXiv preprint, arXiv:1307.6127Google Scholar
- 51.Kirsch, A.: An Introduction to the Mathematical Theory of Inverse Problems. Springer, New York (1996)CrossRefzbMATHGoogle Scholar
- 52.Kühn, T., Liese, F.: A short proof of the Hájek-Feldman theorem. Teor. Verojatnost. i Primenen.
**23**(2), 448–450 (1978)MathSciNetzbMATHGoogle Scholar - 53.Kaipio, J., Somersalo, E.: Statistical and Computational Inverse Problems. Applied Mathematical Sciences, vol. 160. Springer, New York (2005)Google Scholar
- 54.Kallenberg, O.: Foundations of Modern Probability, 2nd edn. Probability and its Applications. Springer, New York, (2002)CrossRefzbMATHGoogle Scholar
- 55.Knapik, B., van Der Vaart, A., van Zanten, J.: Bayesian inverse problems with Gaussian priors. Ann. Stat.
**39**(5), 2626–2657 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 56.Knapik, B., van der Vaart, A., van Zanten, J.H.: Bayesian recovery of the initial condition for the heat equation Commun. Stat. Theory Methods
**42**, 1294–1313 (2013)CrossRefzbMATHGoogle Scholar - 57.Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo methods for very high dimensional integration: the standard (weighted Hilbert space) setting and beyond. ANZIAM J.
**53**, 1–37 (2011)MathSciNetCrossRefzbMATHGoogle Scholar - 58.Kuo, F.Y., Schwab, C., Sloan, I.H.: Quasi-Monte Carlo Finite Element Methods for a Class of Elliptic Partial Differential Equations with Random Coefficients. SIAM J. Numer. Anal.
**50**(6), 3351–3374 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 59.Kuo, F.Y., Sloan, I.H.: Lifting the curse of dimensionality. Not. AMS
**52**(11), 1320–1328 (2005)MathSciNetzbMATHGoogle Scholar - 60.Kuo, F.Y., Sloan, I.H., Wasilikowski, G.W., Waterhouse, B.J.: Randomly shifted lattice rules with the optimal rate of convergence for unbounded integrands. J. Complex.
**26**, 135–160 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 61.Lasanen, S.: Discretizations of generalized random variables with applications to inverse problems. Ann. Acad. Sci. Fenn. Math. Dissertation, University of Oulu, 130 (2002)Google Scholar
- 62.Lasanen, S.: Measurements and infinite-dimensional statistical inverse theory. PAMM
**7**, 1080101–1080102 (2007)CrossRefGoogle Scholar - 63.Lasanen, S.: Non-Gaussian statistical inverse problems part II: posterior convergence for approximated unknowns. Inverse Probl. Imaging
**6**(2), 267 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 64.Lasanen, S.: Non-Gaussian statistical inverse problems. Part I: posterior distributions. Inverse Probl. Imaging
**6**(2), 215–266 (2012)MathSciNetzbMATHGoogle Scholar - 65.Lasanen, S.: Non-Gaussian statistical inverse problems. Part II: posterior distributions. Inverse Probl. Imaging
**6**(2), 267–287 (2012)MathSciNetzbMATHGoogle Scholar - 66.Ledoux, M.: Isoperimetry and Gaussian analysis. In: Lectures on Probability Theory and Statistics (Saint-Flour, 1994). Lecture Notes in Mathematics, vol. 1648, pp. 165–294. Springer, Berlin (1996)Google Scholar
- 67.Lifshits, M.: Gaussian Random Functions. Mathematics and its Applications, vol. 322. Kluwer, Dordrecht (1995)Google Scholar
- 68.Lehtinen, M.S., Päivärinta, L., Somersalo, E.: Linear inverse problems for generalised random variables. Inverse Probl.
**5**(4), 599–612 (1989). http://stacks.iop.org/0266-5611/5/599 MathSciNetCrossRefzbMATHGoogle Scholar - 69.Lassas, M., Saksman, E., Siltanen, S.: Discretization-invariant Bayesian inversion and Besov space priors. Inverse Probl. Imaging
**3**, 87–122 (2009)MathSciNetCrossRefzbMATHGoogle Scholar - 70.Lunardi, A.: Analytic Semigroups and Optimal Regularity in Parabolic Problems. Progress in Nonlinear Differential Equations and their Applications, vol. 16. Birkhäuser Verlag, Basel (1995)Google Scholar
- 71.Mandelbaum, A.: Linear estimators and measurable linear transformations on a Hilbert space. Z. Wahrsch. Verw. Gebiete
**65**(3), 385–397 (1984). http://dx.doi.org/10.1007/BF00533743 MathSciNetCrossRefzbMATHGoogle Scholar - 72.Mattingly, J., Pillai, N., Stuart, A.: Diffusion limits of the random walk Metropolis algorithm in high dimensions. Ann. Appl. Probl.
**22**, 881–930 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 73.Metropolis, N., Rosenbluth, R., Teller, M., Teller, E.: Equations of state calculations by fast computing machines. J. Chem. Phys.
**21**, 1087–1092 (1953)CrossRefGoogle Scholar - 74.Meyer, Y.: Wavelets and operators. Translated from the 1990 French original by D.H. Salinger. Cambridge Studies in Advanced Mathematics, vol. 37. Cambridge University Press, Cambridge (1992)Google Scholar
- 75.Meyn, S.P., Tweedie, R.L.: Markov Chains and Stochastic Stability. Communications and Control Engineering Series. Springer, London (1993)CrossRefzbMATHGoogle Scholar
- 76.Neal, R.: Regression and classification using Gaussian process priors. http://www.cs.toronto.edu/~radford/valencia.abstract.html (1998)
- 77.Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. SIAM, Philadelphia (1994)zbMATHGoogle Scholar
- 78.Norris, J.: Markov Chains. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge (1998)zbMATHGoogle Scholar
- 79.Oksendal, B.: Stochastic Differential Equations. An Introduction with Applications. Universitext, 6th edn. Springer, Berlin (2003)Google Scholar
- 80.Pazy, A.: Semigroups of Linear Operators and Applications to Partial Differential Equations. Springer, New York (1983)CrossRefzbMATHGoogle Scholar
- 81.Petra, N., Martin, J., Stadler, G., Ghattas, O.: A computational framework for infinite-dimensional Bayesian inverse problems. Part II: stochastic Newton MCMC with application to ice sheet flow inverse problems. SIAM J. Sci. Comput.
**36**(4), A1525–A1555 (2014)MathSciNetzbMATHGoogle Scholar - 82.Pillai, N.S., Stuart, A.M., Thiery, A.H.: Noisy gradient flow from a random walk in Hilbert space. Stoch. PDEs: Anal. Comput.
**2**, 196–232 (2014)MathSciNetCrossRefzbMATHGoogle Scholar - 83.Pinski, F., Stuart, A.: Transition paths in molecules at finite temperature. J. Chem. Phys.
**132**, 184104 (2010)CrossRefGoogle Scholar - 84.Rebeschini, P., van Handel, R.: Can local particle filters beat the curse of dimensionality? Ann. Appl. Probab.
**25**(5), 2809–2866 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 85.Revuz, D., Yor, M.: Continuous Martingales and Brownian Motion. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences], vol. 293, 2nd edn. Springer, Berlin (1994)Google Scholar
- 86.Richter, G.: An inverse problem for the steady state diffusion equation. SIAM J. Appl. Math.
**41**(2), 210–221 (1981)MathSciNetCrossRefzbMATHGoogle Scholar - 87.Robinson, J.C.: Infinite-Dimensional Dynamical Systems. Cambridge Texts in Applied Mathematics. Cambridge University Press, Cambridge (2001)CrossRefzbMATHGoogle Scholar
- 88.Rudin, W.: Real and Complex Analysis, 3rd edn. McGraw-Hill, New York (1987)zbMATHGoogle Scholar
- 89.Schillings, C., Schwab, C.: Sparse, adaptive Smolyak quadratures for Bayesian inverse problems. Inverse Probl.
**29**, 065011 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 90.Schwab, C., Stuart, A.: Sparse deterministic approximation of Bayesian inverse problems. Inverse Probl.
**28**, 045003 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 91.Strauss, W.A.: Partial differential equations. An introduction, 2nd edn. Wiley, Chichester (2008)Google Scholar
- 92.Stuart, A.M.: Inverse problems: a Bayesian perspective. Acta Numer.
**19**, 451–559 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 93.Stuart, A.M.: Uncertainty quantification in Bayesian inversion. ICM2014. Invited Lecture (2014)Google Scholar
- 94.Tierney, L.: A note on Metropolis-Hastings kernels for general state spaces. Ann. Appl. Probab.
**8**(1), 1–9 (1998)MathSciNetCrossRefzbMATHGoogle Scholar - 95.Triebel, H.: Theory of function spaces. Mathematik und ihre Anwendungen in Physik und Technik [Mathematics and its Applications in Physics and Technology], vol. 38. Akademische Verlagsgesellschaft Geest & Portig K.-G., Leipzig (1983)Google Scholar
- 96.Triebel, H.: Theory of Function Spaces. II. Monographs in Mathematics, vol. 84. Birkhäuser Verlag, Basel (1992)Google Scholar
- 97.Triebel, H.: Theory of Function Spaces. III. Monographs in Mathematics, vol. 100. Birkhäuser Verlag, Basel (2006)Google Scholar
- 98.Villani, C.: Topics in Optimal Transportation. Graduate Studies in Mathematics, vol. 58. American Mathematical Society, Providence (2003)Google Scholar
- 99.Vollmer, S.: Posterior consistency for Bayesian inverse problems through stability and regression results. Inverse Probl.
**29**, 125011 (2013)MathSciNetCrossRefGoogle Scholar - 100.Yosida, K.: Functional Analysis. Classics in Mathematics. Springer, Berlin (1995). Reprint of the sixth (1980) edition.Google Scholar
- 101.Yin, G., Zhang, Q.: Continuous-Time Markov Chains and Applications. Applications of Mathematics (New York), vol. 37. Springer, New York (1998)Google Scholar