Skip to main content
SpringerLink
Log in

Bernstein–von Mises Theorem and Misspecified Models: A Review

  • Conference paper
  • First Online:
Foundations of Modern Statistics (FMS 2019)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 425))

Included in the following conference series:

  • 418 Accesses

Abstract

This is a review of asymptotic and non-asymptotic behaviour of Bayesian methods under model specification. In particular we focus on consistency, i.e. convergence of the posterior distribution to the point mass at the best parametric approximation to the true model, and conditions for it to be locally Gaussian around this point. For well specified regular models, variance of the Gaussian approximation coincides with the Fisher information, making Bayesian inference asymptotically efficient. In this review, we discuss how this is affected by model misspecification. In particular, we highlight contribution of Volodia Spokoiny to this area. We also discuss approaches to adjust Bayesian inference to make it asymptotically efficient under model misspecification.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Baraud, Y., Birgé, L.: Robust Bayes-like estimation: Rho-Bayes estimation. Ann. Stat. 48(6), 3699–3720, 12 2020

    Google Scholar 

  2. Besag, J.: On the statistical analysis of dirty pictures (with discussion). J. Roy. Statist. Soc. B 48, 259–302 (1986)

    MathSciNet  MATH  Google Scholar 

  3. Bhattacharya, A., Pati, D., Yang, Y.: Bayesian fractional posteriors. Ann. Statist. 47(1), 39–66, 02 2019

    Google Scholar 

  4. Bhattacharya, I., Ghosal, S.: Bayesian inference on multivariate medians and quantiles. Statistica Sinica (2019)

    Google Scholar 

  5. Bhattacharya, I., Martin, R.: Gibbs posterior inference on multivariate quantiles. J. Stat. Plann. Infer. 218, 106–121 (2022)

    Google Scholar 

  6. Bissiri, P.G., Holmes, C.C., Walker, S.G.: A general framework for updating belief distributions. J. R. Statist. Soc.: Ser. B (Statistical Methodology) (2016)

    Google Scholar 

  7. Bochkina, N.A., Green, P.J.: The Bernstein–von Mises theorem and nonregular models. Ann. Statist. 42(5), 1850–1878, 10 2014

    Google Scholar 

  8. Breiman, L.: Bagging predictors. Mach. Learn. 24, 123–140 (1996)

    Article  MATH  Google Scholar 

  9. Chernozhukov, V., Hong, H.: Likelihood estimation and inference in a class of nonregular econometric models. Econometrica 72, 1445–1480 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chib, S., Shin, M., Simoni, A.: Bayesian estimation and comparison of moment condition models. J. Am. Statist. Assoc. 113(524), 1656–1668 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  11. Dalalyan, A., Tsybakov, A.B.: Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Mach. Learn. 72, 39–61 (2008)

    Article  MATH  Google Scholar 

  12. de Heide, R., Kirichenko, A., Mehta, N., Grünwald, P.: Safe-Bayesian generalized linear regression. In: Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 2623 –2633. PMLR (2020)

    Google Scholar 

  13. Diong, M.L., Chaumette, E., Vincent, F.: On the efficiency of maximum-likelihood estimators of misspecified models. In: 25th European Signal Processing Conference (EUSIPCO) (2017)

    Google Scholar 

  14. Fong, E., Holmes, C., Walker, S.G.: Martingale posterior distributions. J. R. Statist. Soc.: Ser. B (Statistical Methodology) (2023).

    Google Scholar 

  15. Grünwald, P.: The safe Bayesian. In: International Conference on Algorithmic Learning Theory, pp. 169–183. Springer (2012)

    Google Scholar 

  16. Grünwald, P., van Ommen, T.: Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Anal. 12(4), 1069–1103, 12 2017

    Google Scholar 

  17. Grünwald, P.D., Mehta, N.A.: Fast rates for general unbounded loss functions: from ERM to generalized Bayes. J. Mach. Learn. Res. 21(56), 1–80 (2020)

    MathSciNet  MATH  Google Scholar 

  18. Holmes, C.C., Walker, S.G.: Assigning a value to a power likelihood in a general Bayesian model. Biometrika 104(2), 497–503 (2017)

    MathSciNet  MATH  Google Scholar 

  19. Huggins, J., Miller, J.: Reproducible model selection using bagged posteriors. Bayesian. Anal. 18(1), 79–104 (2023)

    Google Scholar 

  20. Ibragimov, I., Hasminskij, R.: Statistical Estimation: Asymptotic Theory. Springer (1981)

    Google Scholar 

  21. Kleijn, B.J.K., van der Vaart, A.W.: Misspecification in infinite-dimensional Bayesian statistics. Ann. Statist. 34(2), 837–877, 04 2006

    Google Scholar 

  22. Kleijn, B.J.K., van der Vaart, A.W.: The Bernstein–von-Mises theorem under misspecification. Electron. J. Statist. 6, 354–381 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  23. Knoblauch, J., Jewson, J., Damoulas, T.: Generalized variational inference: Three arguments for deriving new posteriors (2021). arXiv:1904.02063

  24. Lindsay, B.: Composite likelihood methods. Contemp. Math. 80, 221–239 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  25. Lyddon, S.P., Holmes, C.C., Walker, S.G.: General Bayesian updating and the loss-likelihood bootstrap. Biometrika 106, 465–478 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  26. Miller, J.W., Dunson, D.B.: Robust Bayesian inference via coarsening. J. Am. Statist. Assoc. 114(527), 1113–1125 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  27. Müller, U.K.: Risk of Bayesian inference in misspecified models, and the sandwich covariance matrix. Econometrica 81(5), 1805–1849 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  28. Newton, M.A., Polson, N.G., Xu, J.: Weighted Bayesian bootstrap for scalable posterior distributions. Canadian J. Statist. (2021)

    Google Scholar 

  29. Panov, M., Spokoiny, V.: Finite sample Bernstein–von Mises theorem for semiparametric problems. Bayesian Anal. 10(3), 665–710, 09 2015

    Google Scholar 

  30. Pauli, F., Racugno, W., Ventura, L.: Bayesian composite marginal likelihoods. Statistica Sinica 21, 149–164 (2012)

    MathSciNet  MATH  Google Scholar 

  31. Ribatet, M., Cooley, D., Davison, A.C.: Bayesian inference from composite likelihoods, with an application to spatial extremes. Statistica Sinica 22, 813–845 (2012)

    MathSciNet  MATH  Google Scholar 

  32. Rubin, D.B.: The Bayesian bootstrap. Ann. Statist. 9, 130–134 (1981)

    Article  MathSciNet  Google Scholar 

  33. Schennach, S.M.: Bayesian exponentially tilted empirical likelihood. Biometrika 92(1), 31–46 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  34. Spokoiny, V.: Parametric estimation. Finite sample theory. Ann. Statist. 40(6), 2877–2909, 12 2012

    Google Scholar 

  35. Spokoiny, V.: Bayesian inference for nonlinear inverse problems (2020). arXiv:1912.12694

  36. Spokoiny, V., Panov, M.: Accuracy of Gaussian approximation in nonparametric Bernstein–von Mises (2020). arXiv:1910.06028

  37. Stoehr, J., Friel, N.: Calibration of conditional composite likelihood for Bayesian inference on Gibbs random fields. In: Lebanon, G., Vishwanathan, S.V.N. (eds.), Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pp. 921–929. PMLR (2015)

    Google Scholar 

  38. Syring, N., Martin, R.: Gibbs posterior concentration rates under sub-exponential type losses. Bernoulli 29(2), 1080–1108 (2023)

    Google Scholar 

  39. Van der Vaart, A.W.: Asymptotic Statistics, vol. 3. Cambridge University Press (2000)

    Google Scholar 

  40. Varin, C., Reid, N., Firth, D.: An overview of composite likelihood methods. Statistica Sinica 21, 5–42 (2011)

    MathSciNet  MATH  Google Scholar 

  41. Waddell, P.J., Kishino, H., Ota, R.: Very fast algorithms for evaluating the stability of ml and Bayesian phylogenetic trees from sequence data. In Genome Inf. 13, 82–92 (2002)

    Google Scholar 

  42. Wang, Y., Blei, D.M.: Variational Bayes under model misspecification. In: In Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  43. Wang, Y., Kucukelbir, A., Blei, D.M.: Robust probabilistic modeling with Bayesian data reweighting. In: Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 3646–3655. PMLR (2017)

    Google Scholar 

  44. White, H.: Maximum likelihood estimation of misspecified models. Econometrica 50, 1–25 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  45. Wu, P.-S., Martin, R.: A comparison of learning rate selection methods in generalized Bayesian inference (2020). arxiv:2012.11349

  46. Zhang, Y., Nalisnick, E.: On the inconsistency of Bayesian inference for misspecified neural networks. In: Third Symposium on Advances in Approximate Bayesian Inference (2021)

    Google Scholar 

Download references

Acknowledgements

This review was in part motivated by the discussion of the author with Peter Grünwald, Pierre Jacob and Jeffrey Miller during a Research in Groups meeting sponsored by the International Centre for Mathematical Sciences in Edinburgh, UK.

Author information

Authors and Affiliations

  1. University of Edinburgh and Maxwell Institute, Edinburgh, UK

    Natalia Bochkina

Authors
  1. Natalia Bochkina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Natalia Bochkina .

Editor information

Editors and Affiliations

  1. Faculty of Mathematics, University of Duisburg-Essen, Essen, Germany

    Denis Belomestny

  2. Institut Polytechnique de Paris, CREST, ENSAE, Palaiseau, France

    Cristina Butucea

  3. Institute for Applied Mathematics, Heidelberg University, Heidelberg, Baden-Württemberg, Germany

    Enno Mammen

  4. CMAP, Ecole Polytechnique, Palaiseau, France

    Eric Moulines

  5. Institute of Mathematics, Humboldt-Universität zu Berlin, Berlin, Germany

    Markus Reiß

  6. Faculty of Computer Science, HSE University and Moscow State University, Moscow, Russia

    Vladimir V. Ulyanov

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bochkina, N. (2023). Bernstein–von Mises Theorem and Misspecified Models: A Review. In: Belomestny, D., Butucea, C., Mammen, E., Moulines, E., Reiß, M., Ulyanov, V.V. (eds) Foundations of Modern Statistics. FMS 2019. Springer Proceedings in Mathematics & Statistics, vol 425. Springer, Cham. https://doi.org/10.1007/978-3-031-30114-8_10

Download citation

Publish with us

Policies and ethics

Search

Navigation