Statistics and Computing

, Volume 25, Issue 4, pp 739–753 | Cite as

Bayesian inference via projections

  • Ricardo SilvaEmail author
  • Alfredo Kalaitzis


Bayesian inference often poses difficult computational problems. Even when off-the-shelf Markov chain Monte Carlo (MCMC) methods are available to the problem at hand, mixing issues might compromise the quality of the results. We introduce a framework for situations where the model space can be naturally divided into two components: (i) a baseline black-box probability distribution for the observed variables and (ii) constraints enforced on functionals of this probability distribution. Inference is performed by sampling from the posterior implied by the first component, and finding projections on the space defined by the second component. We discuss the implications of this separation in terms of priors, model selection, and MCMC mixing in latent variable models. Case studies include probabilistic principal component analysis, models of marginal independence, and a interpretable class of structured ordinal probit models.


MCMC Optimization Latent variable models Structured covariance matrices 



We thank Irini Moustaki for the green consumer data. This work was partially funded by a EPSRC Grant EP/J013293/1.


  1. Andrieu, C., Roberts, G.: The pseudo-marginal approach for efficient Monte Carlo computations. Ann. Stat. 37, 697–725 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  2. Barnard, J., McCulloch, R., Meng, X.: Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Stat. Sin. 10, 1281–1311 (2000)zbMATHMathSciNetGoogle Scholar
  3. Bartholomew, D., Steele, F., Moustaki, I., Galbraith, J.: Analysis of Multivariate Social Science Data, 2nd edn. Chapman & Hall, London (2008)zbMATHGoogle Scholar
  4. Beaumont, M., Zhang, W., Balding, D.: Approximate Bayesian computation in population genetics. Genetics 162, 2025–2035 (2002)Google Scholar
  5. Bickel, P., Levina, E.: Covariance regularization by thresholding. Ann. Stat. 36, 2577–2604 (2008)zbMATHMathSciNetCrossRefGoogle Scholar
  6. Bissiri, P., Holmes, C., Walker, S.: A general framework for updating belief distributions. arXiv:1306.6430 (2013)
  7. Candès, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 11 (2011)MathSciNetCrossRefGoogle Scholar
  8. Care Quality Commission and Aston University: Aston Business School, National Health Service National Staff Survey, 2009 [computer file]. Colchester, Essex: UK Data Archive [distributor], October 2010. Available at, SN: 6570 (2010)
  9. Drovandi, C.C., Pettitt, A.N., Lee, A.: Bayesian indirect inference using a parametric auxiliary model. Stat. Sci. (in press)Google Scholar
  10. Drton, M., Richardson, T.: A new algorithm for maximum likelihood estimation in Gaussian models for marginal independence. In: Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc., (2003)Google Scholar
  11. Gallant, A.R., McCulloch, R.E.: On the determination of general scientific models with application to asset pricing. J. Am. Stat. Assoc. 104(485), 117–131 (2009)MathSciNetCrossRefGoogle Scholar
  12. Gelman, A., Meng, X., Stern, H.: Posterior predictive assessment of model fitness via realized discrepancies. Stat. Sin. 6, 733–807 (1996)zbMATHMathSciNetGoogle Scholar
  13. Gribonval, R., Machart, P.: Reconciling “priors” & “priors” without prejudice? Adv. Neural Inf. Process. Syst. 26, 2193–2201 (2013)Google Scholar
  14. Grzebyk, M., Wild, P., Chouaniere, D.: On identification of multi-factor models with correlated residuals. Biometrika 91, 141–151 (2004)zbMATHMathSciNetCrossRefGoogle Scholar
  15. Jerrum, M., Sinclair, A.: The Markov chain Monte Carlo method: an approach to approximate counting and integration. In: Hochbaum, D.S. (ed.) Approximation Algorithms for NP-hard Problems, pp. 482–520. PWS Publishing Company, Pacific Grove (1996)Google Scholar
  16. Neal, R.: Probabilistic inference using Markov chain monte carlo methods. Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto (1993)Google Scholar
  17. Palla, K., Knowles, D.A., Ghahramani, Z.: A nonparametric variable clustering model. Adv. Neural Inf. Process. Syst. 25, 2987–2995 (2012)Google Scholar
  18. Reeves, R., Pettitt, A.: A theoretical framework for approximate bayesian computation. In: 20th International Workshop on Statistical Modelling, pp. 393–396 (2005)Google Scholar
  19. Richardson, T., Spirtes, P.: Ancestral graph Markov models. Ann. Stat. 30, 962–1030 (2002)zbMATHMathSciNetCrossRefGoogle Scholar
  20. Silva, R.: A MCMC approach for learning the structure of Gaussian acyclic directed mixed graphs. In: Giudici, P., Ingrassia, S., Vichi, M. (eds.) Stat. Models Data Anal., pp. 343–352. Springer, New York (2013)CrossRefGoogle Scholar
  21. Silva, R., Ghahramani, Z.: The hidden life of latent variables: Bayesian learning with mixed graph models. J. Mach. Learn. Res. 10, 1187–1238 (2009)zbMATHMathSciNetGoogle Scholar
  22. Tipping, M., Bishop, C.: Probabilistic principal component analysis. J. R. Stat. Soc. 61(3), 611–622 (1999)zbMATHMathSciNetCrossRefGoogle Scholar
  23. Wang, H.: Scaling it up: Stochastic search structure learning in graphical models. Bayesian Anal. 10, 351–377 (2015)CrossRefGoogle Scholar
  24. Wright, J., Ganesh, A., Rao, S., Peng, Y., Ma, Y.: Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. Adv. Neural Inf. Process. Syst. 22, 2080–2088 (2009)Google Scholar
  25. Yin, G.: Bayesian generalized method of moments. Bayesian Anal. 4, 191–207 (2009)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Statistical Science and CSMLUniversity College LondonLondonUK

Personalised recommendations