Abstract
The statistical evidence (or marginal likelihood) is a key quantity in Bayesian statistics, allowing one to assess the probability of the data given the model under investigation. This paper focuses on refining the power posterior approach to improve estimation of the evidence. The power posterior method involves transitioning from the prior to the posterior by powering the likelihood by an inverse temperature. In common with other tempering algorithms, the power posterior involves some degree of tuning. The main contributions of this article are twofold—we present a result from the numerical analysis literature which can reduce the bias in the estimate of the evidence by addressing the error arising from numerically integrating across the inverse temperatures. We also tackle the selection of the inverse temperature ladder, applying this approach additionally to the Stepping Stone sampler estimation of evidence. A key practical point is that both of these innovations incur virtually no extra cost.
Similar content being viewed by others
References
Atkinson, K., Han, W.: Elementary Numerical Analysis, 3rd edn. Wiley, New York (2004)
Behrens, G., Friel, N., Hurn, M.: Tuning tempered transitions. Stat. Comput. 22(1), 65–78 (2012)
Calderhead, B., Girolami, M.: Estimating Bayes factors via thermodynamic integration and population MCMC. Comput. Stat. Data Anal. 53(12), 4028–4045 (2009)
Chib, S.: Marginal likelihood from the Gibbs output. J. Am. Stat. Assoc. 90(432), 1313–1321 (1995)
Friel, N., Pettitt, A.N.: Marginal likelihood estimation via power posteriors. J. R. Stat. Soc. B 70(3), 589–607 (2008)
Friel, N., Wyse, J.: Estimating the evidence—a review. Stat. Neerl. 66(3), 288–308 (2012)
Green, P.J.: Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82(4), 711–732 (1995)
Lartillot, N., Philippe, H.: Computing Bayes factors using thermodynamic integration. Syst. Biol. 55(2), 195–207 (2006)
Lefebvre, G., Steele, R.J., Vandal, A.C.: A path sampling identity for computing the Kullback-Leibler and J-divergences. Comput. Stat. Data Anal. 54(7), 1719–1731 (2010)
Meng, X.L., Wong, W.H.: Simulating ratios of normalizing constants via a simple identity: a theoretical exploration. Stat. Sin. 6(4), 831–860 (1996)
Neal, R.M.: Annealed importance sampling. Stat. Comput. 11(2), 125–139 (2001)
Richardson, S., Green, P.J.: On Bayesian analysis of mixtures with an unknown number of components (with discussion). J. R. Stat. Soc. B 59(4), 731–792 (1997)
Skilling, J.: Nested sampling for general Bayesian computation. Bayesian Anal. 1(4), 833–860 (2006)
Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., Johannes, R.S.: Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In: Proceedings of the Annual Symposium on Computer Application in Medical Care, p. 261. American Medical Informatics Association, Indianapolis (1988)
Tierney, L., Kadane, J.B.: Accurate approximations for posterior moments and marginal densities. J. Am. Stat. Assoc. 81(393), 82–86 (1986)
Williams, E.: Regression Analysis. Wiley, Chichester (1959)
Xie, W., Lewis, P.O., Fan, Y., Kuo, L., Chen, M.H.: Improving marginal likelihood estimation for Bayesian phylogenetic model selection. Syst. Biol. 60(2), 150–160 (2011)
Acknowledgements
Nial Friel’s research was supported by a Science Foundation Ireland Research Frontiers Program grant, 09/RFP/MTH2199. Jason Wyse’s research was supported through the STATICA project, a Principal Investigator program of Science Foundation Ireland, 08/IN.1/I1879. We are grateful to two anonymous reviewers whose comments on an earlier version have much improved this work.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Friel, N., Hurn, M. & Wyse, J. Improving power posterior estimation of statistical evidence. Stat Comput 24, 709–723 (2014). https://doi.org/10.1007/s11222-013-9397-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11222-013-9397-1