Variational Bayes for Generic Topic Models

  • Gregor Heinrich
  • Michael Goesele
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5803)


The article contributes a derivation of variational Bayes for a large class of topic models by generalising from the well-known model of latent Dirichlet allocation. For an abstraction of these models as systems of interconnected mixtures, variational update equations are obtained, leading to inference algorithms for models that so far have used Gibbs sampling exclusively.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Barnard, K., Duygulu, P., Forsyth, D., de Freitas, N., Blei, D., Jordan, M.: Matching words and pictures. JMLR 3(6), 1107–1136 (2003)MATHGoogle Scholar
  2. 2.
    Beal, M.J.: Variational Algorithms for Approximate Bayesian Inference. PhD thesis, Gatsby Computational Neuroscience Unit, University College London (2003)Google Scholar
  3. 3.
    Beal, M.J., Ghahramani, Z.: Variational bayesian learning of directed graphical models with hidden variables. Bayesian Analysis 1, 793–832 (2006)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Blei, D., Lafferty, J.: A correlated topic model of science. AOAS 1, 17–35 (2007)MathSciNetMATHGoogle Scholar
  5. 5.
    Blei, D., Ng, A., Jordan, M.: Hierarchical Bayesian models for applications in information retrieval. Bayesian Statistics 7, 25–44 (2003)MathSciNetGoogle Scholar
  6. 6.
    Blei, D., Ng, A., Jordan, M.: Latent Dirichlet allocation. JMLR 3, 993–1022 (2003)MATHGoogle Scholar
  7. 7.
    Heinrich, G.: A generic approach to topic models. In: ECML/PKDD (2009)Google Scholar
  8. 8.
    Li, W., Blei, D., McCallum, A.: Mixtures of hierarchical topics with pachinko allocation. In: ICML (2007)Google Scholar
  9. 9.
    Li, W., McCallum, A.: Pachinko allocation: DAG-structured mixture models of topic correlations. In: ICML (2006)Google Scholar
  10. 10.
    Minka, T.: Estimating a Dirichlet distribution. Web (2003)Google Scholar
  11. 11.
  12. 12.
    Steyvers, M., Smyth, P., Rosen-Zvi, M., Griffiths, T.: Probabilistic author-topic models for information discovery. In: ACM SIGKDD (2004)Google Scholar
  13. 13.
    Teh, Y.W., Newman, D., Welling, M.: A collapsed variational Bayesian inference algorithm for latent Dirichlet allocation. In: NIPS, vol. 19 (2007)Google Scholar
  14. 14.
    Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical Dirichlet processes. Technical Report 653, Department of Statistics, University of California at Berkeley (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Gregor Heinrich
    • 1
  • Michael Goesele
    • 2
  1. 1.Fraunhofer IGD and University of LeipzigGermany
  2. 2.TU DarmstadtGermany

Personalised recommendations