Skip to main content
SpringerLink
Log in

Fundamental limits of symmetric low-rank matrix estimation

  • Published:
Probability Theory and Related Fields Aims and scope Submit manuscript

Abstract

We consider the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise. Given a probabilistic model for the low-rank matrix, we compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean squared error, while the rank of the signal remains constant. We also show that our model extends beyond the particular case of additive Gaussian noise and we prove an universality result connecting the community detection problem to our Gaussian framework. We unify and generalize a number of recent works on PCA, sparse PCA, submatrix localization or community detection by computing the information-theoretic limits for these problems in the high noise regime. In addition, we show that the posterior distribution of the signal given the observations is characterized by a parameter of the same dimension as the square of the rank of the signal (i.e. scalar in the case of rank one). This allows to locate precisely the information-theoretic thresholds for the above mentioned problems. Finally, we connect our work with the hard but detectable conjecture in statistical physics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Abbe, E., Sandon, C.: Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic BP, and the information-computation gap. Preprint (2015). arXiv:1512.09080

  2. Aizenman, M., Sims, R., Starr, S.L.: Extended variational principle for the Sherrington–Kirkpatrick spin-glass model. Phys. Rev. B 68(21), 214403 (2003)

    Article  Google Scholar 

  3. Baik, J., Arous, G.B., Péché, S.: Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. 33, 1643–1697 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  4. Bandeira, A.S., Boumal, N., Singer, A.: Tightness of the maximum likelihood semidefinite relaxation for angular synchronization. Math. Program. 163, 1–23 (2016)

    MathSciNet  MATH  Google Scholar 

  5. Banks, J., Moore, C., Vershynin, R., Verzelen, N., Xu, J.: Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization. In: 2017 IEEE International Symposium on Information Theory (ISIT), pp. 1137–1141. IEEE (2017)

  6. Barbier, J., Dia, M., Macris, N., Krzakala, F., Lesieur, T., Zdeborová, L.: Mutual information for symmetric rank-one matrix estimation: a proof of the replica formula. In: Advances in Neural Information Processing Systems, pp. 424–432 (2016)

  7. Benaych-Georges, F., Nadakuditi, R.R.: The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Adv. Math. 227(1), 494–521 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Boucheron, S., Lugosi, G., Massart, P.: Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford (2013)

    Book  MATH  Google Scholar 

  9. Caltagirone, F., Lelarge, M., Miolane, L.: Recovering asymmetric communities in the stochastic block model. IEEE Trans. Netw. Sci. Eng. (2017). https://doi.org/10.1109/TNSE.2017.2758201

  10. Chatterjee, S., et al.: A generalization of the lindeberg principle. Ann. Probab. 34(6), 2061–2076 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  11. Coja-Oghlan, A., Krzakala, F., Perkins, W., Zdeborova, L.: Information-theoretic thresholds from the cavity method. Preprint (2016). arXiv:1611.00814

  12. Decelle, A., Krzakala, F., Moore, C., Zdeborová, L.: Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Phys. Rev. E 84(6), 066106 (2011)

    Article  Google Scholar 

  13. Deshpande, Y., Abbe, E., Montanari, A.: Asymptotic mutual information for the balanced binary stochastic block model. Inf. Inference: A J. IMA 6(2), 125–170 (2016)

    MathSciNet  MATH  Google Scholar 

  14. Deshpande, Y., Montanari, A.: Information-theoretically optimal sparse PCA. In: 2014 IEEE International Symposium on Information Theory, pp. 2197–2201. IEEE (2014)

  15. Féral, D., Péché, S.: The largest eigenvalue of rank one deformation of large wigner matrices. Commun. Math. Phys. 272(1), 185–228 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  16. Ghirlanda, S., Guerra, F.: General properties of overlap probability distributions in disordered spin systems. Towards parisi ultrametricity. J. Phys. A: Math. Gen. 31(46), 9149 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  17. Guerra, F.: Broken replica symmetry bounds in the mean field spin glass model. Commun. Math. Phys. 233(1), 1–12 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  18. Guo, D., Shamai, S., Verdú, S.: Mutual information and minimum mean-square error in Gaussian channels. IEEE Trans. Inf. Theory 51(4), 1261–1282 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  19. Hajek, B., Wu, Y., Xu, J.: Information limits for recovering a hidden community. In: 2016 IEEE International Symposium on Information Theory (ISIT), pp. 1894–1898. IEEE (2016)

  20. Iba, Y.: The Nishimori line and Bayesian statistics. J. Phys. A: Math. Gen. 32(21), 3875 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  21. Korada, S.B., Macris, N.: Exact solution of the gauge symmetric p-spin glass model on a complete graph. J. Stat. Phys. 136(2), 205–230 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  22. Korada, S.B., Macris, N.: Tight bounds on the capacity of binary input random cdma systems. IEEE Trans. Inf. Theory 56(11), 5590–5613 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. Korada, S.B., Montanari, A.: Applications of the lindeberg principle in communications and statistical learning. IEEE Trans. Inf. Theory 57(4), 2440–2450 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  24. Krzakala, F., Xu, J., Zdeborová, L.: Mutual information in rank-one matrix estimation. In: Information Theory Workshop (ITW), 2016 IEEE, pp. 71–75. IEEE (2016)

  25. Lesieur, T., Krzakala, F., Zdeborová, L.: MMSE of probabilistic low-rank matrix estimation: universality with respect to the output channel. In: 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015, Allerton Park and Retreat Center, Monticello, IL, USA, September 29–October 2, 2015, pp. 680–687 (2015)

  26. Lesieur, T., Krzakala, F., Zdeborová, L.: Phase transitions in sparse PCA. In: IEEE International Symposium on Information Theory, ISIT 2015, Hong Kong, China, June 14–19, 2015, pp. 1635–1639. IEEE (2015)

  27. Méasson, C., Montanari, A., Richardson, T.J., Urbanke, R.: The generalized area theorem and some of its consequences. IEEE Trans. Inf. Theory 55(11), 4793–4821 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  28. Mézard, M., Parisi, G., Virasoro, M.: Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, vol. 9. World Scientific, Singapore (1987)

    MATH  Google Scholar 

  29. Milgrom, P., Segal, I.: Envelope theorems for arbitrary choice sets. Econometrica 70(2), 583–601 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  30. Montanari, A.: Estimating random variables from random sparse observations. Eur. Trans. Telecommun. 19(4), 385–403 (2008)

    Article  Google Scholar 

  31. Montanari, A.: Finding one community in a sparse graph. J. Stat. Phys. 161(2), 273–299 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  32. Neeman, J., Netrapalli, P.: Non-reconstructability in the stochastic block model. Preprint (2014). arXiv:1404.6304

  33. Nishimori, H.: Statistical Physics of Spin Glasses and Information Processing: An Introduction, vol. 111. Clarendon Press, Oxford (2001)

    Book  MATH  Google Scholar 

  34. Panchenko, D.: The Sherrington–Kirkpatrick Model. Springer Science & Business Media, Berlin (2013)

    Book  MATH  Google Scholar 

  35. Perry, A., Wein, A.S., Bandeira, A.S., Moitra, A.: Optimality and sub-optimality of PCA for spiked random matrices and synchronization. Preprint (2016). arXiv:1609.05573

  36. Rangan, S., Fletcher, A.K.: Iterative estimation of constrained rank-one matrices in noise. In: 2012 IEEE International Symposium on Information Theory Proceedings (ISIT), pp. 1246–1250. IEEE (2012)

  37. Saade, A., Lelarge, M., Krzakala, F., Zdeborová, L.: Spectral detection in the censored block model. In: 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1184–1188. IEEE (2015)

  38. Saade, A., Lelarge, M., Krzakala, F., Zdeborová, L.: Clustering from sparse pairwise measurements. In: 2016 IEEE International Symposium on Information Theory (ISIT), pp. 780–784. IEEE (2016)

  39. Talagrand, M.: Mean Field Models for Spin Glasses: Volume I: Basic Examples, vol. 54. Springer Science & Business Media, Berlin (2010)

    MATH  Google Scholar 

  40. Talagrand, M.: Mean Field Models for Spin Glasses: Volume II: Advanced Replica-Symmetry and Low Temperature, vol. 55. Springer Science & Business Media, Berlin (2011)

    Book  MATH  Google Scholar 

  41. Wu, Y., Verdú, S.: Functional properties of minimum mean-square error and mutual information. IEEE Trans. Inf. Theory 58(3), 1289–1301 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  42. Zdeborová, L., Krzakala, F.: Statistical physics of inference: thresholds and algorithms. Adv. Phys. 65(5), 453–552 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Département d’informatique de l’ENS, École normale supérieure, CNRS, PSL Research University, 75005, Paris, France

    Marc Lelarge & Léo Miolane

  2. Inria, Paris, France

    Marc Lelarge & Léo Miolane

Authors
  1. Marc Lelarge
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Léo Miolane
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Léo Miolane.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lelarge, M., Miolane, L. Fundamental limits of symmetric low-rank matrix estimation. Probab. Theory Relat. Fields 173, 859–929 (2019). https://doi.org/10.1007/s00440-018-0845-x

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00440-018-0845-x

Mathematics Subject Classification

Search

Navigation