Introduction to Bayesian Methods and Decision Theory

  • Simon P. Wilson
  • Rozenn Dahyot
  • Pádraig Cunningham
Part of the Cognitive Technologies book series (COGTECH)


Bayesian methods are a class of statistical methods that have some appealing properties for solving problems in machine learning, particularly when the process being modelled has uncertain or random aspects. In this chapter we look at the mathematical and philosophical basis for Bayesian methods and how they relate to machine learning problems in multimedia. We also discuss the notion of decision theory, for making decisions under uncertainty, that is closely related to Bayesian methods. The numerical methods needed to implement Bayesian solutions are also discussed. Two specific applications of the Bayesian approach that are often used in machine learning – naïve Bayes and Bayesian networks – are then described in more detail.


Posterior Distribution Markov Chain Monte Carlo Bayesian Method Decision Theory Subjective Probability 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    J. O. Berger. Statistical decision theory and Bayesian analysis. Springer-Verlag, New York, second edition, 1993.Google Scholar
  2. 2.
    J. M. Bernardo and A. F. M. Smith. Bayesian theory. Wiley, Chichester, 1994.zbMATHGoogle Scholar
  3. 3.
    R. T. Cox. Probability, frequency and reasonable expectation. Am. J. Phys., 14:1–13, 1946.zbMATHCrossRefGoogle Scholar
  4. 4.
    B. de Finetti. Theory of probability, volume 1. Wiley, New York, 1974.Google Scholar
  5. 5.
    D. Gamerman. Markov chain Monte Carlo: stochastic simulations for Bayesian inference. Chapman and Hall, New York, 1997.Google Scholar
  6. 6.
    P. H. Garthwaite, J. B. Kadane, and A. O’Hagan. Statistical methods for eliciting probability distributions. J. Am. Stat. Assoc., 100:680–701, 2005.zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian data analysis. Chapman and Hall, London, second edition, 2003.Google Scholar
  8. 8.
    W. R. Gilks, N. G. Best, and K. K. C. Tan. Adaptive rejection metropolis sampling within Gibb’s sampling. Appl. Stat., 44:455–472, 1995.zbMATHCrossRefGoogle Scholar
  9. 9.
    W. R. Gilks, G. O. Roberts, and E. I. George. Adaptive rejection sampling. Statistician, 43:179–189, 1994.CrossRefGoogle Scholar
  10. 10.
    G. Grimmett and D. Stirzaker. Probability and Random Processes. Oxford University Press, Oxford, third edition, 2001.Google Scholar
  11. 11.
    R. Kohavi, B. Becker, and D. Sommerfield. Improving simple bayes. In Proceedings of the European Conference on Machine Learning (ECML-87), pages 78–97, 1997.Google Scholar
  12. 12.
    P. M. Lee. Bayesian statistics: an introduction. Hodder Arnold H&S, London, third edition, 2004.Google Scholar
  13. 13.
    D. D. Lewis and M. Ringuette. A comparison of two learning algorithms for text categorization. In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pp. 81–93, Las Vegas, US, 1994.Google Scholar
  14. 14.
    D. V. Lindley. Making decisions. Wiley, London, second edition, 1982.Google Scholar
  15. 15.
    K. L. Mengersen, C. P. Robert, and C. Guihenneuc-Jouyaux. Mcmc convergence diagnostics: a “reviewww”. In J. Berger, J. Bernardo, A. P. Dawid, and A.F.M. Smith, editors, Bayesian Statistics 6, pp. 415–440. Oxford Science Publications, 1999.Google Scholar
  16. 16.
    T. Niblett. Constructing decision trees in noisy domains. In 2nd European Working Session on Learning, pp. 67–78, Bled, Yugoslavia, 1987.Google Scholar
  17. 17.
    S. M. Ross. Simulation. Academic Press, San Diego, third edition, 2001.Google Scholar
  18. 18.
    S. M. Ross. Introduction to probability models. Academic Press, San Diego, eighth edition, 2003.zbMATHGoogle Scholar
  19. 19.
    M. A. Tanner. Tools for statistical inference: methods for the exploration of posterior distributions and likelihood functions. Springer-Verlag, New York, third edition, 1996.zbMATHGoogle Scholar
  20. 20.
    V. uSmìdl and A. Quinn. The variational Bayes method in signal processing. Springer, New York, 2005.Google Scholar
  21. 21.
    G. Winkler. Image analysis, random fields and dynamic Monte Carlo methods. Springer-Verlag, Berlin, second edition, 2006.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Simon P. Wilson
    • 1
  • Rozenn Dahyot
    • 1
  • Pádraig Cunningham
    • 2
  1. 1.Trinity College DublinIreland
  2. 2.University College DublinIreland

Personalised recommendations