Bayesian Methods for Low-Rank Matrix Estimation: Short Survey and Theoretical Study

  • Pierre Alquier
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8139)


The problem of low-rank matrix estimation recently received a lot of attention due to challenging applications. A lot of work has been done on rank-penalized methods [1] and convex relaxation [2], both on the theoretical and applied sides. However, only a few papers considered Bayesian estimation. In this paper, we review the different type of priors considered on matrices to favour low-rank. We also prove that the obtained Bayesian estimators, under suitable assumptions, enjoys the same optimality properties as the ones based on penalization.


Bayesian inference collaborative filtering reduced-rank regression matrix completion PAC-Bayesian bounds oracle inequalities 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bunea, F., She, Y., Wegkamp, M.H.: Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics 39(2), 1282–1309 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Candès, E., Tao, T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56(5), 2053–2080 (2009)CrossRefGoogle Scholar
  3. 3.
    Bennett, J., Lanning, S.: The netflix prize. In: Proceedings of KDD Cup and Workshop 2007 (2007)Google Scholar
  4. 4.
    Reinsel, G.C., Velu, R.P.: Multivariate reduced-rank regression: theory and applications. Springer Lecture Notes in Statistics, vol. 136 (1998)Google Scholar
  5. 5.
    Koltchinskii, V., Lounici, K., Tsybakov, A.B.: Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics 39(5), 2302–2329 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Alquier, P., Butucea, C., Hebiri, M., Meziani, K., Morimae, T.: Rank-penalized estimation of a quantum system. Preprint arXiv:1206.1711 (2012)Google Scholar
  7. 7.
    Geweke, J.: Bayesian reduced rank regression in econometrics. Journal of Econometrics 75, 121–146 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Lim, Y.J., Teh, Y.W.: Variational Bayesian approach to movie rating prediction. In: Proceedings of KDD Cup and Workshop 2007 (2007)Google Scholar
  9. 9.
    Lawrence, N.D., Urtasun, R.: Non-linear matrix factorization with Gaussian processes. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 601–608. ACM, New York (2009)Google Scholar
  10. 10.
    Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization. In: Platt, J.C., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in Neural Information Processing Systems 20, NIPS 2007. MIT Press, Cambridge (2008)Google Scholar
  11. 11.
    Anderson, T.: Estimating linear restrictions on regression coefficients for multivariate normal distributions. Annals of Mathematical Statistics 22, 327–351 (1951)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Izenman, A.: Reduced rank regression for the multivariate linear model. Journal of Multivariate Analysis 5(2), 248–264 (1975)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Yuan, M., Ekici, A., Lu, Z., Monteiro, R.: Dimension reduction and coefficient estimation in multivariate linear regression. Journal of the Royal Statistical Society - Series B 69, 329–346 (2007)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Candès, E., Plan, Y.: Matrix completion with noise. Proceedings of the IEEE 98(6), 625–636 (2009)Google Scholar
  15. 15.
    Candès, E., Recht, B.: Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9(6), 717–772 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Gross, D.: Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory 57, 1548–1566 (2011)CrossRefGoogle Scholar
  17. 17.
    Rohde, A., Tsybakov, A.B.: Estimation of high-dimensional low-rank matrices. The Annals of Statistics 39, 887–930 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Klopp, O.: Rank-penalized estimators for high-dimensionnal matrices. Electronic Journal of Statistics 5, 1161–1183 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Koltchinskii, V.: Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems. Springer Lecture Notes in Mathematics (2011)Google Scholar
  20. 20.
    Dreze, J.H.: Bayesian limited information analysis of the simultaneous equation model. Econometrica 44, 1045–1075 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Dreze, J.H., Richard, J.F.: Bayesian analysis of simultaneous equation models. In: Griliches, Z., Intriligater, J.F. (eds.) Handbook of Econometrics, vol. 1. North-Holland, Amsterdam (1983)Google Scholar
  22. 22.
    Zellner, A., Min, C., Dallaire, D.: Bayesian analysis of simultaenous equation and related models using the Gibbs sampler and convergence checks. H. G. B. Alexander Research Founsation working paper, University of Chicago (1993)Google Scholar
  23. 23.
    Kleibergen, F., van Dijk, H.K.: Bayesian simultaneous equation analysis using reduced rank structures. Econometric Theory 14, 699–744 (1998)CrossRefGoogle Scholar
  24. 24.
    Bauwens, L., Lubrano, M.: Identification restriction and posterior densities in cointegrated gaussian var systems. In: Fomby, T.M., Carter Hill, R. (eds.) Advances in Econometrics, vol. 11(B). JAI Press, Greenwich (1993)Google Scholar
  25. 25.
    Kleibergen, F., van Dijk, H.K.: On the shape of the likelihood-posterior in cointegration models. Econometric Theory 10, 514–551 (1994)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Kleibergen, F., Paap, R.: Priors, posteriors and Bayes factors for a Bayesian analysis of cointegration. Journal of Econometrics 111, 223–249 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Corander, J., Villani, M.: Bayesian assessment of dimensionality in reduced rank regression. Statistica Neerlandica 58(3), 255–270 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th Annual International Conference on Machine Learning, ICML 2008. ACM, New York (2008)Google Scholar
  29. 29.
    Zhou, M., Wang, C., Chen, M., Paisley, J., Dunson, D., Carin, L.: Nonparametric Bayesian matrix completion. In: IEEE Sensor Array and Multichannel Signal Processing Workshop (2010)Google Scholar
  30. 30.
    Babacan, S.D., Luessi, M., Molina, R., Katsaggelos, A.K.: Low-rank matrix completion by variational sparse Bayesian learning. In: IEEE International Conference on Audio, Speech and Signal Processing, Prague (Czech Republic), pp. 2188–2191 (2011)Google Scholar
  31. 31.
    Paisley, J., Carin, L.: A nonparametric Bayesian model for kernel matrix completion. In: Proceedings of ICASSP 2010, Dallas, USA (2010)Google Scholar
  32. 32.
    Yu, K., Tresp, V., Schwaighofer, A.: Learning Gaussian processes for multiple tasks. In: Proceedings of the 22th Annual International Conference on Machine Learning, ICML 2005 (2005)Google Scholar
  33. 33.
    Yu, K., Lafferty, J., Zhu, S., Gong, Y.: Large-scale collaborative prediction using a non-parametric random effects model. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009. ACM, New York (2009)Google Scholar
  34. 34.
    Aoyagi, M., Watanabe, S.: The generalization error of reduced rank regression in Bayesian estimation. In: International Symposium on Information Theory and its Applications, ISITA 2004, Parma, Italy (2004)Google Scholar
  35. 35.
    Aoyagi, M., Watanabe, S.: Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks 18, 924–933 (2005)CrossRefzbMATHGoogle Scholar
  36. 36.
    van der Vaart, A.W.: Asymptotic Statistics. Cambridge University Press (1998)Google Scholar
  37. 37.
    Shawe-Taylor, J., Williamson, R.: A PAC analysis of a Bayes estimator. In: Proceedings of the Tenth Annual Conference on Computational Learning Theory, pp. 2–9. ACM, New York (1997)CrossRefGoogle Scholar
  38. 38.
    McAllester, D.A.: Some pac-bayesian theorems. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, pp. 230–234. ACM (1998)Google Scholar
  39. 39.
    Catoni, O.: Statistical Learning Theory and Stochastic Optimization. Springer Lecture Notes in Mathematics (2004)Google Scholar
  40. 40.
    Catoni, O.: PAC-Bayesian Supervised Classification (The Thermodynamics of Statistical Learning). Lecture Notes-Monograph Series, vol. 56. IMS (2007)Google Scholar
  41. 41.
    Dalalyan, A.S., Tsybakov, A.B.: Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Machine Learning 72, 39–61 (2008)CrossRefGoogle Scholar
  42. 42.
    Dalalyan, A.S., Tsybakov, A.B.: Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78(5), 1423–1443 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Dalalyan, A.S., Salmon, J.: Sharp oracle inequalities for aggregation of affine estimators. The Annals of Statistics 40(4), 2327–2355 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Alquier, P., Lounici, K.: PAC-Bayesian bounds for sparse regression estimation with exponential weights. Electronic Journal of Statistics 5, 127–145 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Audibert, J.Y., Catoni, O.: Robust linear least squares regression. The Annals of Statistics 39, 2766–2794 (2011)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Pierre Alquier
    • 1
  1. 1.School of Mathematical SciencesUniversity College DublinDublin 4Ireland

Personalised recommendations