# Bayesian inference via projections

- 566 Downloads
- 1 Citations

## Abstract

Bayesian inference often poses difficult computational problems. Even when off-the-shelf Markov chain Monte Carlo (MCMC) methods are available to the problem at hand, mixing issues might compromise the quality of the results. We introduce a framework for situations where the model space can be naturally divided into two components: (i) a baseline black-box probability distribution for the observed variables and (ii) constraints enforced on functionals of this probability distribution. Inference is performed by sampling from the posterior implied by the first component, and finding projections on the space defined by the second component. We discuss the implications of this separation in terms of priors, model selection, and MCMC mixing in latent variable models. Case studies include probabilistic principal component analysis, models of marginal independence, and a interpretable class of structured ordinal probit models.

## Keywords

MCMC Optimization Latent variable models Structured covariance matrices## Notes

### Acknowledgments

We thank Irini Moustaki for the green consumer data. This work was partially funded by a EPSRC Grant EP/J013293/1.

## References

- Andrieu, C., Roberts, G.: The pseudo-marginal approach for efficient Monte Carlo computations. Ann. Stat.
**37**, 697–725 (2009)zbMATHMathSciNetCrossRefGoogle Scholar - Barnard, J., McCulloch, R., Meng, X.: Modeling covariance matrices in terms of standard deviations and correlations, with application to shrinkage. Stat. Sin.
**10**, 1281–1311 (2000)zbMATHMathSciNetGoogle Scholar - Bartholomew, D., Steele, F., Moustaki, I., Galbraith, J.: Analysis of Multivariate Social Science Data, 2nd edn. Chapman & Hall, London (2008)zbMATHGoogle Scholar
- Beaumont, M., Zhang, W., Balding, D.: Approximate Bayesian computation in population genetics. Genetics
**162**, 2025–2035 (2002)Google Scholar - Bickel, P., Levina, E.: Covariance regularization by thresholding. Ann. Stat.
**36**, 2577–2604 (2008)zbMATHMathSciNetCrossRefGoogle Scholar - Bissiri, P., Holmes, C., Walker, S.: A general framework for updating belief distributions. arXiv:1306.6430 (2013)
- Candès, E., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM
**58**(3), 11 (2011)MathSciNetCrossRefGoogle Scholar - Care Quality Commission and Aston University: Aston Business School, National Health Service National Staff Survey, 2009 [computer file]. Colchester, Essex: UK Data Archive [distributor], October 2010. Available at https://www.esds.ac.uk, SN: 6570 (2010)
- Drovandi, C.C., Pettitt, A.N., Lee, A.: Bayesian indirect inference using a parametric auxiliary model. Stat. Sci. (in press)Google Scholar
- Drton, M., Richardson, T.: A new algorithm for maximum likelihood estimation in Gaussian models for marginal independence. In: Proceedings of the 19th Conference on Uncertainty in Artificial Intelligence, Morgan Kaufmann Publishers Inc., (2003)Google Scholar
- Gallant, A.R., McCulloch, R.E.: On the determination of general scientific models with application to asset pricing. J. Am. Stat. Assoc.
**104**(485), 117–131 (2009)MathSciNetCrossRefGoogle Scholar - Gelman, A., Meng, X., Stern, H.: Posterior predictive assessment of model fitness via realized discrepancies. Stat. Sin.
**6**, 733–807 (1996)zbMATHMathSciNetGoogle Scholar - Gribonval, R., Machart, P.: Reconciling “priors” & “priors” without prejudice? Adv. Neural Inf. Process. Syst.
**26**, 2193–2201 (2013)Google Scholar - Grzebyk, M., Wild, P., Chouaniere, D.: On identification of multi-factor models with correlated residuals. Biometrika
**91**, 141–151 (2004)zbMATHMathSciNetCrossRefGoogle Scholar - Jerrum, M., Sinclair, A.: The Markov chain Monte Carlo method: an approach to approximate counting and integration. In: Hochbaum, D.S. (ed.) Approximation Algorithms for NP-hard Problems, pp. 482–520. PWS Publishing Company, Pacific Grove (1996)Google Scholar
- Neal, R.: Probabilistic inference using Markov chain monte carlo methods. Technical Report CRG-TR-93-1, Department of Computer Science, University of Toronto (1993)Google Scholar
- Palla, K., Knowles, D.A., Ghahramani, Z.: A nonparametric variable clustering model. Adv. Neural Inf. Process. Syst.
**25**, 2987–2995 (2012)Google Scholar - Reeves, R., Pettitt, A.: A theoretical framework for approximate bayesian computation. In: 20th International Workshop on Statistical Modelling, pp. 393–396 (2005)Google Scholar
- Richardson, T., Spirtes, P.: Ancestral graph Markov models. Ann. Stat.
**30**, 962–1030 (2002)zbMATHMathSciNetCrossRefGoogle Scholar - Silva, R.: A MCMC approach for learning the structure of Gaussian acyclic directed mixed graphs. In: Giudici, P., Ingrassia, S., Vichi, M. (eds.) Stat. Models Data Anal., pp. 343–352. Springer, New York (2013)CrossRefGoogle Scholar
- Silva, R., Ghahramani, Z.: The hidden life of latent variables: Bayesian learning with mixed graph models. J. Mach. Learn. Res.
**10**, 1187–1238 (2009)zbMATHMathSciNetGoogle Scholar - Tipping, M., Bishop, C.: Probabilistic principal component analysis. J. R. Stat. Soc.
**61**(3), 611–622 (1999)zbMATHMathSciNetCrossRefGoogle Scholar - Wang, H.: Scaling it up: Stochastic search structure learning in graphical models. Bayesian Anal.
**10**, 351–377 (2015)CrossRefGoogle Scholar - Wright, J., Ganesh, A., Rao, S., Peng, Y., Ma, Y.: Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. Adv. Neural Inf. Process. Syst.
**22**, 2080–2088 (2009)Google Scholar - Yin, G.: Bayesian generalized method of moments. Bayesian Anal.
**4**, 191–207 (2009)MathSciNetCrossRefGoogle Scholar