Free Energy of Stochastic Context Free Grammar on Variational Bayes

  • Tikara Hosino
  • Kazuho Watanabe
  • Sumio Watanabe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4232)


Variational Bayesian learning is proposed for approximation method of Bayesian learning. In spite of efficiency and experimental good performance, their mathematical property has not yet been clarified. In this paper we analyze variational Bayesian Stochastic Context Free Grammar which includes the true distribution thus the model is non-identifiable. We derive their asymptotic free energy. It is shown that in some prior conditions, the free energy is much smaller than identifiable models and satisfies eliminating redundant non-terminals.


Hide Markov Model Generalization Error Bayesian Learning Terminal Symbol Nonterminal Symbol 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Attias, H.: Inferring parameters and structure of latent variable models by variational Bayes. In: Proc. 15th Conference on Uncertainty in Artificial Intelligence, pp. 21–20 (1999)Google Scholar
  2. 2.
    Beal, M.J.: Variational Algorithms for Approximate Bayesian Inference, PhD thesis, University College London (2003)Google Scholar
  3. 3.
    Gassiat, E., Boucheron, S.: Optimal error exponents in hidden Markov models order estimation. IEEE Transactions on Information Theory 49(2), 964–980 (2003)MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Hosino, T., Watanabe, K., Watanabe, S.: Stochastic Complexity of Variational Bayesian Hidden Markov Models. In: International Joint Conference on Neural Networks (2005)Google Scholar
  5. 5.
    Kurihara, K., Sato, T.: An Application of the Variational Bayesian Approach to Probabilistic Context-Free Grammars. In: International Joint Conference on Natural Language Processing (2004)Google Scholar
  6. 6.
    Ito, H., Amari, S.-I., Kobayashi, K.: Identifiability of hidden Markov information sources and their minimum degrees of freedom. IEEE Transactions on Information Theory 38(2), 324–333 (1992)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Lari, K., Young, S.: The estimation of stochastic context-free grammars using the inside-outside algorithm. Computer Speech and Language 4, 33–56 (1990)CrossRefGoogle Scholar
  8. 8.
    Nakajima, S., Watanabe, S.: Generalization Error and Free Energy of Linear Neural Networks in Variational Bayes Approach. In: The 12th International Conference on Neural Information Processing (2005)Google Scholar
  9. 9.
    Schwarz, G.: Estimating the dimension of a model. Annals of Statistics 6(2), 461–464 (1978)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Watanabe, S.: Algebraic analysis for non-identifiable learning machines. Neural Computation 13(4), 899–933 (2001)MATHCrossRefGoogle Scholar
  11. 11.
    Watanabe, K., Watanabe, S.: Variational bayesian stochastic complexity of mixture models. In: Advances in Neural Information Processing Systems 18. MIT Press, Cambridge (2006) (to appear)Google Scholar
  12. 12.
    Yamazaki, K., Watanabe, S.: Generalization Errors in Estimating of Stochastic Context-Free Grammar. Artificial Intelligence and Soft Computing (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Tikara Hosino
    • 1
    • 2
  • Kazuho Watanabe
    • 1
  • Sumio Watanabe
    • 3
  1. 1.Computational Intelligence and System ScienceTokyo Institute of TechnologyYokohamaJapan
  2. 2.Nihon Unisys, Ltd.TokyoJapan
  3. 3.Precision and Intelligence LaboratoryTokyo Institute of TechnologyYokohamaJapan

Personalised recommendations