Abstract
Recently, the variational Bayesian approximation was applied to probabilistic matrix factorization and shown to perform very well in experiments. However, its good performance was not completely understood beyond its experimental success. The purpose of this paper is to theoretically elucidate properties of a variational Bayesian matrix factorization method. In particular, its mechanism of avoiding overfitting is analyzed. Our analysis relies on the key fact that the matrix factorization model induces non-identifiability, i.e., the mapping between factorized matrices and the original matrix is not one-to-one. The positive-part James-Stein shrinkage operator and the Marcenko-Pastur law—the limiting distribution of eigenvalues of the central Wishart distribution—play important roles in our analysis.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Baldi, P.F., Hornik, K.: Learning in Linear Neural Networks: a Survey. IEEE Trans. on Neural Networks 6, 837–858 (1995)
Baldi, P., Brunak, S.: Bioinformatics. MIT Press, Cambridge (2001)
Konstan, J.A., Miller, B.N., Maltz, D., Herlocker, J.L., Gordon, L.R., Riedl, J.: Grouplens: applying collaborative filtering to usenet news. Commun. ACM 40, 77–87 (1997)
Funk, S.: Try this at home (2006), http://sifter.org/~simon/journal/20061211.html
Srebro, N., Jaakkola, T.: Weighted Low Rank Approximation. In: Proc. of ICML (2003)
Srebro, N., Rennie, J., Jaakkola, T.: Maximum Margin Matrix Factorization. In: Advances in NIPS, vol. 17 (2005)
Rennie, J.D.M., Srebro, N.: Fast Maximum Margin Matrix Factorization for Collaborative Prediction. In: Proc. of ICML (2005)
Salakhutdinov, R., Mnih, A.: Probabilistic Matrix Factorization. In: Advances in NIPS, vol. 20 (2008)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)
Attias, H.: Inferring Parameters and Structure of Latent Variable Models by Variational Bayes. In: Proc. of UAI (1999)
Lim, Y.J., Teh, T.W.: Variational Bayesian Approach to Movie Rating Prediction. In: Proc. of KDD Cup and Workshop (2007)
Raiko, T., Ilin, A., Karhunen, J.: Principal Component Analysis for Large Sale Problems with Lots of Missing Values. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS, vol. 4701, pp. 691–698. Springer, Heidelberg (2007)
James, W., Stein, C.: Estimation with Quadratic Loss. In: Proc. of the 4th Berkeley Symp. on Math. Stat. and Prob., pp. 361–379 (1961)
Watanabe, S.: Algebraic Analysis for Nonidentifiable Learning Machines. Neural Computation 13, 899–933 (2001)
Nakajima, S., Watanabe, S.: Variational Bayes Solution of Linear Neural Networks and its Generalization Performance. Neural Computation 19, 1112–1153 (2007)
Marcenko, V.A., Pastur, L.A.: Distribution of Eigenvalues for Some Sets of Random Matrices. Mathematics of the USSR-Sbornik 1, 457–483 (1967)
Wachter, K.W.: The Strong Limits of Random Matrix Spectra for Sample Matrices of Independent Elements. Annals of Probability 6, 1–18 (1978)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nakajima, S., Sugiyama, M. (2009). Analysis of Variational Bayesian Matrix Factorization. In: Theeramunkong, T., Kijsirikul, B., Cercone, N., Ho, TB. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2009. Lecture Notes in Computer Science(), vol 5476. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-01307-2_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-01307-2_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-01306-5
Online ISBN: 978-3-642-01307-2
eBook Packages: Computer ScienceComputer Science (R0)