Skip to main content

Model-Based Machine Learning and Approximate Inference

  • Chapter
  • First Online:
Variational Methods for Machine Learning with Applications to Deep Networks

Abstract

In this chapter, we introduce the building blocks of Model-Based Machine Learning (MBML). We explain what it is and discuss its main enabling techniques: Bayesian inference, graphical models, and, more recently, probabilistic programming. Frequently, models are complex enough so that exact inference is not possible, and one must resort to approximate methods. We broach deterministic distributional approximation methods for approximate inference, focusing on Variational Bayes, Assumed Density Filtering, and Expectation Propagation, going through derivations, advantages, issues, and modern extensions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Berger T (1975) Rate distortion theory and data compression. Springer Vienna, pp 1–39. https://doi.org/10.1007/978-3-7091-2928-9_1

  2. Bingham E, Chen JP, Jankowiak M, Obermeyer F, Pradhan N, Karaletsos T, Singh R, Szerlip P, Horsfall P, Goodman ND (2019) Pyro: deep universal probabilistic programming. J Mach Learn Res 20(28):1–6

    MATH  Google Scholar 

  3. Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin

    MATH  Google Scholar 

  4. Bishop CM (2013) Model-based machine learning. Philos Trans R Soc A: Math Phys Eng Sci 371(1984):1–17

    Article  MathSciNet  MATH  Google Scholar 

  5. Blei DM, Kucukelbir A, McAuliffe JD (2017) Variational inference: a review for statisticians. J Am Stat Assoc 112(518):859–877

    Article  MathSciNet  Google Scholar 

  6. Carpenter B, Gelman A, Hoffman M, Lee D, Goodrich B, Betancourt M, Brubaker M, Guo J, Li P, Riddell A (2017) Stan: A probabilistic programming language. J Stat Softw 76(1):1–32. Articles

    Google Scholar 

  7. Ghahramani Z (2015) Probabilistic machine learning and artificial intelligence. Nature 521(7553):452–459

    Article  Google Scholar 

  8. Goodman ND, Stuhlmüller A (2014) (electronic). The Design and Implementation of Probabilistic Programming Languages. Retrieved 2021-4-5 from http://dippl.org

  9. Hernandez-Lobato J, Li Y, Rowland M, Bui T, Hernandez-Lobato D, Turner R (2016) Black-box alpha divergence minimization. In: Proceedings of the international conference on machine learning, New York, vol 48, pp 1511–1520

    Google Scholar 

  10. Hoffman M, Blei D (2015) Stochastic structured variational inference. In: International conference on artificial intelligence and statistics, San Diego, vol 38, pp 361–369

    Google Scholar 

  11. Hoffman MD, Blei DM, Wang C, Paisley J (2013) Stochastic variational inference. J Mach Learn Res 14:1303–1347

    MathSciNet  MATH  Google Scholar 

  12. Koller D, Friedman N, Bach F (2009) Probabilistic graphical models: principles and techniques. MIT Press, Cambridge

    Google Scholar 

  13. Kucukelbir A, Blei D, Gelman A, Ranganath R, Tran D (2017) Automatic differentiation variational inference. J Mach Learn Res 18:1–45

    MathSciNet  MATH  Google Scholar 

  14. Lloyd S (1982) Least squares quantization in PCM. IEEE Trans Inf Theor 28(2):129–137

    Article  MathSciNet  MATH  Google Scholar 

  15. Louizos C, Welling M (2016) Structured and efficient variational deep learning with matrix Gaussian posteriors. In: Proceedings of the international conference on machine learning, New York, vol 48, pp 1708–1716

    Google Scholar 

  16. MacKay DJ (2003) Information theory, inference and learning algorithms. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  17. Minka TP (2001) Expectation propagation for approximate Bayesian inference. In: Conference in uncertainty in artificial intelligence, San Francisco, pp 362–369

    Google Scholar 

  18. Minka T (2004) Power EP. Tech. rep., Microsoft Research

    Google Scholar 

  19. Minka T (2005) Divergence measures and message passing. Tech. rep., Microsoft Research

    Google Scholar 

  20. Minka T (2007) The EP energy function and minimization schemes. Tech. rep., Microsoft Research

    Google Scholar 

  21. Minka T, Winn J, Guiver J, Zaykov Y, Fabian D, Bronskill J (2018) Inf. NET 0.3. Microsoft Research Cambridge

    Google Scholar 

  22. Mohamed S (2018) Planting the seeds of probabilistic thinking: foundations, tricks and algorithms. Tutorial presentation

    Google Scholar 

  23. Murphy KP (2012) Machine learning: a probabilistic perspective. MIT Press, Cambridge

    MATH  Google Scholar 

  24. Price R (1958) A useful theorem for nonlinear devices having Gaussian inputs. Trans Inf Theor 4(2):69–72

    Article  MathSciNet  MATH  Google Scholar 

  25. Ranganath R, Gerrish S, Blei D (2014) Black box variational inference. In: Proceedings of the international conference on artificial intelligence and statistics, Reykjavik, pp 814–822

    Google Scholar 

  26. Rezende D, Mohamed S (2015) Variational inference with normalizing flows. In: Proceedings of the international conference on machine learning, Lille, vol 37, pp 1530–1538

    Google Scholar 

  27. Robbins H, Monro S (1951) A stochastic approximation method. Ann Math Stat 22(3):400–407

    Article  MathSciNet  MATH  Google Scholar 

  28. Salvatier J, Wiecki TV, Fonnesbeck C (2016) Probabilistic programming in Python using PyMC3. PeerJ Comput Sci 2:e55

    Article  Google Scholar 

  29. Sun S, Chen C, Carin L (2017) Learning structured weight uncertainty in Bayesian neural networks. In: International conference on artificial intelligence and statistics, Fort Lauderdale, vol 54, pp 1283–1292

    Google Scholar 

  30. Tran D, Kucukelbir A, Dieng AB, Rudolph M, Liang D, Blei DM (2016) Edward: a library for probabilistic modeling, inference, and criticism. arXiv e-prints 1610.09787

    Google Scholar 

  31. Tukey JW (1962) The future of data analysis. Ann Math Stat 33(1):1–67

    Article  MathSciNet  MATH  Google Scholar 

  32. van de Meent JW, Paige B, Yang H, Wood F (2018) An introduction to probabilistic programming. arXiv e-prints 1809.10756

    Google Scholar 

  33. Vidakovic B (2011) Bayesian inference using Gibbs sampling—BUGS project. Springer, New York, pp 733–745

    Google Scholar 

  34. Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8(3):229–256

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Pinheiro Cinelli, L., Araújo Marins, M., Barros da Silva, E.A., Lima Netto, S. (2021). Model-Based Machine Learning and Approximate Inference. In: Variational Methods for Machine Learning with Applications to Deep Networks. Springer, Cham. https://doi.org/10.1007/978-3-030-70679-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-70679-1_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-70678-4

  • Online ISBN: 978-3-030-70679-1

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics