Abstract
We investigate theoretically some properties of variational Bayes approximations based on estimating the mixing coefficients of known densities. We show that, with probability 1 as the sample size n grows large, the iterative algorithm for the variational Bayes approximation converges locally to the maximum likelihood estimator at the rate of O(1/n). Moreover, the variational posterior distribution for the parameters is shown to be asymptotically normal with the same mean but a different covariance matrix compared with those for the maximum likelihood estimator. Furthermore we prove that the covariance matrix from the variational Bayes approximation is ‘too small’ compared with that for the MLE, so that resulting interval estimates for the parameters will be unrealistically narrow.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Attias, H.: Inferring parameters and structure of latent variable models by variational Bayes. In: Prade, H., Laskey, K. (eds.) Proc. 15th Conference on Uncertainty in Artificial Intelligence, Stockholm, Sweden, pp. 21–30. Morgan Kaufmann Publishers, San Francisco (1999)
Attias, H.: A variational Bayesian framework for graphical models. In: Solla, S., Leen, T., Muller, K.R. (eds.) Advances in Neural Information Processing Systems, vol. 12, pp. 209–215. MIT Press, Cambridge (2000)
Beal, M.J.: Variational Algorithms for Approximate Bayesian Inference. PhD thesis, University College London (2003)
Corduneanu, A., Bishop, C.M.: Variational Bayesian model selection for mixture distributions. In: Richardson, T., Jaakkola, T. (eds.) Proceedings Eighth International Conference on Artificial Intelligence and Statistics, pp. 27–34. Morgan Kaufmann, San Francisco (2001)
Ghahramani, Z., Beal, M.J.: Propagation algorithms for variational Bayesian learning. In: Leen, T., Dietterich, T., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 507–513. MIT Press, Cambridge (2001)
Humphreys, K., Titterington, D.M.: Approximate Bayesian inference for simple mixtures. In: Bethlehem, J.G., van der Heijden, P.G.M. (eds.) COMPSTAT 2000, pp. 331–336. Physica-Verlag, Heidelberg (2000)
Jaakkola, T.S., Jordan, M.I.: Bayesian logistic regression: a variational approach. Statistics and Computing 10, 25–37 (2000)
Jordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K.: An introduction to variational methods for graphical models. In: Jordan, M.I. (ed.) Learning in Graphical Models, pp. 105–162. MIT Press, Cambridge (1999)
MacKay, D.J.C.: Ensemble learning for hidden Markov models. Technical report, Cavendish Laboratory, University of Cambridge (1997)
Penny, W.D., Roberts, S.J.: Variational Bayes for 1-dimensional mixture models. Technical Report PARG-2000-01, Oxford University (2000)
Peters, B.C., Walker, H.F.: An iterative procedure for obtaining maximum-likelihood estimates of the parameters for a mixture of normal distributions. SIAM J. Appl. Math. 35, 362–378 (1978)
Redner, R.A., Walker, H.F.: Mixture densities, maximum likelihood and the EM algorithm. SIAM Review 26, 195–239 (1984)
Wang, B., Titterington, D.M.: Convergence and asymptotic normality of variational Bayesian approximations for exponential family models with missing values. In: Chickering, M., Halpern, J. (eds.) Proceedings of the twentieth conference on Uncertainty in Artificial Intelligence, Banff, Canada, pp. 577–584. AUAI Press (2004)
Walker, A.M.: On the asymptotic behaviour of posterior distributions. J. R. Statist. Soc. B 31, 80–88 (1969)
Heyde, C.C., Johnstone, I.M.: On asymptotic posterior normality for stochastic processes. J. R. Statist. Soc. B 41, 184–189 (1979)
Chen, C.F.: On asymptotic normality of limiting density functions with Bayesian implications. J. R. Statist. Soc. B 47, 540–546 (1985)
Bernardo, J.M., Smith, A.F.M.: Bayesian Theory. John Wiley & Sons, Inc., New York (1994)
Ghosal, S., Ghosh, J.K., Samanta, T.: On convergence of posterior distributions. The Annals of Statistics 23, 2145–2152 (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wang, B., Titterington, D.M. (2005). Variational Bayes Estimation of Mixing Coefficients. In: Winkler, J., Niranjan, M., Lawrence, N. (eds) Deterministic and Statistical Methods in Machine Learning. DSMML 2004. Lecture Notes in Computer Science(), vol 3635. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11559887_17
Download citation
DOI: https://doi.org/10.1007/11559887_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29073-5
Online ISBN: 978-3-540-31728-9
eBook Packages: Computer ScienceComputer Science (R0)