Skip to main content
SpringerLink
Log in

Bayesian Methods for Low-Rank Matrix Estimation: Short Survey and Theoretical Study

  • Conference paper
Algorithmic Learning Theory (ALT 2013)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 8139))

Included in the following conference series:

Abstract

The problem of low-rank matrix estimation recently received a lot of attention due to challenging applications. A lot of work has been done on rank-penalized methods [1] and convex relaxation [2], both on the theoretical and applied sides. However, only a few papers considered Bayesian estimation. In this paper, we review the different type of priors considered on matrices to favour low-rank. We also prove that the obtained Bayesian estimators, under suitable assumptions, enjoys the same optimality properties as the ones based on penalization.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bunea, F., She, Y., Wegkamp, M.H.: Optimal selection of reduced rank estimators of high-dimensional matrices. The Annals of Statistics 39(2), 1282–1309 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  2. Candès, E., Tao, T.: The power of convex relaxation: Near-optimal matrix completion. IEEE Transactions on Information Theory 56(5), 2053–2080 (2009)

    Article  Google Scholar 

  3. Bennett, J., Lanning, S.: The netflix prize. In: Proceedings of KDD Cup and Workshop 2007 (2007)

    Google Scholar 

  4. Reinsel, G.C., Velu, R.P.: Multivariate reduced-rank regression: theory and applications. Springer Lecture Notes in Statistics, vol. 136 (1998)

    Google Scholar 

  5. Koltchinskii, V., Lounici, K., Tsybakov, A.B.: Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion. The Annals of Statistics 39(5), 2302–2329 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  6. Alquier, P., Butucea, C., Hebiri, M., Meziani, K., Morimae, T.: Rank-penalized estimation of a quantum system. Preprint arXiv:1206.1711 (2012)

    Google Scholar 

  7. Geweke, J.: Bayesian reduced rank regression in econometrics. Journal of Econometrics 75, 121–146 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  8. Lim, Y.J., Teh, Y.W.: Variational Bayesian approach to movie rating prediction. In: Proceedings of KDD Cup and Workshop 2007 (2007)

    Google Scholar 

  9. Lawrence, N.D., Urtasun, R.: Non-linear matrix factorization with Gaussian processes. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, pp. 601–608. ACM, New York (2009)

    Google Scholar 

  10. Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization. In: Platt, J.C., Koller, D., Singer, Y., Roweis, S. (eds.) Advances in Neural Information Processing Systems 20, NIPS 2007. MIT Press, Cambridge (2008)

    Google Scholar 

  11. Anderson, T.: Estimating linear restrictions on regression coefficients for multivariate normal distributions. Annals of Mathematical Statistics 22, 327–351 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  12. Izenman, A.: Reduced rank regression for the multivariate linear model. Journal of Multivariate Analysis 5(2), 248–264 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  13. Yuan, M., Ekici, A., Lu, Z., Monteiro, R.: Dimension reduction and coefficient estimation in multivariate linear regression. Journal of the Royal Statistical Society - Series B 69, 329–346 (2007)

    Article  MathSciNet  Google Scholar 

  14. Candès, E., Plan, Y.: Matrix completion with noise. Proceedings of the IEEE 98(6), 625–636 (2009)

    Google Scholar 

  15. Candès, E., Recht, B.: Exact matrix completion via convex optimization. Foundations of Computational Mathematics 9(6), 717–772 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  16. Gross, D.: Recovering low-rank matrices from few coefficients in any basis. IEEE Transactions on Information Theory 57, 1548–1566 (2011)

    Article  Google Scholar 

  17. Rohde, A., Tsybakov, A.B.: Estimation of high-dimensional low-rank matrices. The Annals of Statistics 39, 887–930 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  18. Klopp, O.: Rank-penalized estimators for high-dimensionnal matrices. Electronic Journal of Statistics 5, 1161–1183 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  19. Koltchinskii, V.: Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems. Springer Lecture Notes in Mathematics (2011)

    Google Scholar 

  20. Dreze, J.H.: Bayesian limited information analysis of the simultaneous equation model. Econometrica 44, 1045–1075 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  21. Dreze, J.H., Richard, J.F.: Bayesian analysis of simultaneous equation models. In: Griliches, Z., Intriligater, J.F. (eds.) Handbook of Econometrics, vol. 1. North-Holland, Amsterdam (1983)

    Google Scholar 

  22. Zellner, A., Min, C., Dallaire, D.: Bayesian analysis of simultaenous equation and related models using the Gibbs sampler and convergence checks. H. G. B. Alexander Research Founsation working paper, University of Chicago (1993)

    Google Scholar 

  23. Kleibergen, F., van Dijk, H.K.: Bayesian simultaneous equation analysis using reduced rank structures. Econometric Theory 14, 699–744 (1998)

    Article  Google Scholar 

  24. Bauwens, L., Lubrano, M.: Identification restriction and posterior densities in cointegrated gaussian var systems. In: Fomby, T.M., Carter Hill, R. (eds.) Advances in Econometrics, vol. 11(B). JAI Press, Greenwich (1993)

    Google Scholar 

  25. Kleibergen, F., van Dijk, H.K.: On the shape of the likelihood-posterior in cointegration models. Econometric Theory 10, 514–551 (1994)

    Article  MathSciNet  Google Scholar 

  26. Kleibergen, F., Paap, R.: Priors, posteriors and Bayes factors for a Bayesian analysis of cointegration. Journal of Econometrics 111, 223–249 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  27. Corander, J., Villani, M.: Bayesian assessment of dimensionality in reduced rank regression. Statistica Neerlandica 58(3), 255–270 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  28. Salakhutdinov, R., Mnih, A.: Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th Annual International Conference on Machine Learning, ICML 2008. ACM, New York (2008)

    Google Scholar 

  29. Zhou, M., Wang, C., Chen, M., Paisley, J., Dunson, D., Carin, L.: Nonparametric Bayesian matrix completion. In: IEEE Sensor Array and Multichannel Signal Processing Workshop (2010)

    Google Scholar 

  30. Babacan, S.D., Luessi, M., Molina, R., Katsaggelos, A.K.: Low-rank matrix completion by variational sparse Bayesian learning. In: IEEE International Conference on Audio, Speech and Signal Processing, Prague (Czech Republic), pp. 2188–2191 (2011)

    Google Scholar 

  31. Paisley, J., Carin, L.: A nonparametric Bayesian model for kernel matrix completion. In: Proceedings of ICASSP 2010, Dallas, USA (2010)

    Google Scholar 

  32. Yu, K., Tresp, V., Schwaighofer, A.: Learning Gaussian processes for multiple tasks. In: Proceedings of the 22th Annual International Conference on Machine Learning, ICML 2005 (2005)

    Google Scholar 

  33. Yu, K., Lafferty, J., Zhu, S., Gong, Y.: Large-scale collaborative prediction using a non-parametric random effects model. In: Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009. ACM, New York (2009)

    Google Scholar 

  34. Aoyagi, M., Watanabe, S.: The generalization error of reduced rank regression in Bayesian estimation. In: International Symposium on Information Theory and its Applications, ISITA 2004, Parma, Italy (2004)

    Google Scholar 

  35. Aoyagi, M., Watanabe, S.: Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks 18, 924–933 (2005)

    Article  MATH  Google Scholar 

  36. van der Vaart, A.W.: Asymptotic Statistics. Cambridge University Press (1998)

    Google Scholar 

  37. Shawe-Taylor, J., Williamson, R.: A PAC analysis of a Bayes estimator. In: Proceedings of the Tenth Annual Conference on Computational Learning Theory, pp. 2–9. ACM, New York (1997)

    Chapter  Google Scholar 

  38. McAllester, D.A.: Some pac-bayesian theorems. In: Proceedings of the Eleventh Annual Conference on Computational Learning Theory, Madison, WI, pp. 230–234. ACM (1998)

    Google Scholar 

  39. Catoni, O.: Statistical Learning Theory and Stochastic Optimization. Springer Lecture Notes in Mathematics (2004)

    Google Scholar 

  40. Catoni, O.: PAC-Bayesian Supervised Classification (The Thermodynamics of Statistical Learning). Lecture Notes-Monograph Series, vol. 56. IMS (2007)

    Google Scholar 

  41. Dalalyan, A.S., Tsybakov, A.B.: Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity. Machine Learning 72, 39–61 (2008)

    Article  Google Scholar 

  42. Dalalyan, A.S., Tsybakov, A.B.: Sparse regression learning by aggregation and Langevin Monte-Carlo. J. Comput. System Sci. 78(5), 1423–1443 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  43. Dalalyan, A.S., Salmon, J.: Sharp oracle inequalities for aggregation of affine estimators. The Annals of Statistics 40(4), 2327–2355 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  44. Alquier, P., Lounici, K.: PAC-Bayesian bounds for sparse regression estimation with exponential weights. Electronic Journal of Statistics 5, 127–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  45. Audibert, J.Y., Catoni, O.: Robust linear least squares regression. The Annals of Statistics 39, 2766–2794 (2011)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematical Sciences, University College Dublin, 528 James Joyce Library, Belfield, Dublin 4, Ireland

    Pierre Alquier

Authors
  1. Pierre Alquier
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. National University of Singapore, Republic of Singapore

    Sanjay Jain  & Frank Stephan  & 

  2. Inria Lille - Nord Europe, Villeneuve d’Ascq, France

    RĂ©mi Munos

  3. Hokkaido University, Sapporo, Japan

    Thomas Zeugmann

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Alquier, P. (2013). Bayesian Methods for Low-Rank Matrix Estimation: Short Survey and Theoretical Study. In: Jain, S., Munos, R., Stephan, F., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2013. Lecture Notes in Computer Science(), vol 8139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40935-6_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-40935-6_22

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-40934-9

  • Online ISBN: 978-3-642-40935-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation