Analytic Solution of Hierarchical Variational Bayes in Linear Inverse Problem

  • Shinichi Nakajima
  • Sumio Watanabe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4132)


In singular models, the Bayes estimation, commonly, has the advantage of the generalization performance over the maximum likelihood estimation, however, its accurate approximation using Markov chain Monte Carlo methods requires huge computational costs. The variational Bayes (VB) approach, a tractable alternative, has recently shown good performance in the automatic relevance determination model (ARD), a kind of hierarchical Bayesian learning, in brain current estimation from magnetoencephalography (MEG) data, an ill-posed linear inverse problem. On the other hand, it has been proved that, in three-layer linear neural networks (LNNs), the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. In this paper, noting the similarity between the ARD in a linear problem and an LNN, we analyze a simplified version of the VB approach in the ARD. We discuss its relation to the shrinkage estimation and how ill-posedness affects learning. We also propose the algorithm that requires simpler computation than, and will provide similar performance to, the VB approach.


Linear Inverse Problem Shrinkage Estimation Huge Computational Cost Automatic Relevance Determination Hierarchical Variational 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hinton, G.E., van Camp, D.: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights. In: Proc. of COLT, pp. 5–13 (1993)Google Scholar
  2. 2.
    Attias, H.: Inferring Parameters and Structure of Latent Variable Models by Variational Bayes. In: Proc. of UAI (1999)Google Scholar
  3. 3.
    Neal, R.M.: Bayesian Learning for Neural Networks. Springer, Heidelberg (1996)MATHGoogle Scholar
  4. 4.
    Sato, M., Yoshioka, T., Kajihara, S., Toyama, K., Goda, N., Doya, K., Kawato, M.: Hierarchical Bayesian Estimation for MEG inverse problem. Neuro Image 23, 806–826 (2004)Google Scholar
  5. 5.
    James, W., Stein, C.: Estimation with Quadratic Loss. In: Proc. of the 4th Berkeley Symp. on Math. Stat. and Prob., pp. 361–379 (1961)Google Scholar
  6. 6.
    Nakajima, S., Watanabe, S.: Generalization Error and Free Energy of Variational Bayes Approach of Linear Neural Networks. In: Proc. of ICONIP, Taipei, Taiwan, pp. 55–60 (2005)Google Scholar
  7. 7.
    Callen, H.B.: Thermodynamics. Wiley, Chichester (1960)MATHGoogle Scholar
  8. 8.
    Hamalainen, M., Hari, R., Ilmoniemi, R.J., Knuutila, J., Lounasmaa, O.V.: Magnetoencephalography — Theory, Instrumentation, and Applications to Noninvasive Studies of the Working Human Brain. Rev. Modern Phys. 65, 413–497 (1993)CrossRefGoogle Scholar
  9. 9.
    Nakajima, S., Watanabe, S.: Generalization Performance of Subspace Bayes Approach in Linear Neural Networks. IEICE Trans. E89-D, 1128–1138 (2006)Google Scholar
  10. 10.
    Efron, B., Morris, C.: Stein’s Estimation Rule and its Competitors—an Empirical Bayes Approach. J. of Am. Stat. Assoc. 68, 117–130 (1973)MATHCrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Shinichi Nakajima
    • 1
  • Sumio Watanabe
    • 2
  1. 1.Nikon CorporationKumagayaJapan
  2. 2.Tokyo Institute of TechnologyYokohamaJapan

Personalised recommendations