Advertisement

Fundamental limits of symmetric low-rank matrix estimation

  • Marc Lelarge
  • Léo Miolane
Article
  • 114 Downloads

Abstract

We consider the high-dimensional inference problem where the signal is a low-rank symmetric matrix which is corrupted by an additive Gaussian noise. Given a probabilistic model for the low-rank matrix, we compute the limit in the large dimension setting for the mutual information between the signal and the observations, as well as the matrix minimum mean squared error, while the rank of the signal remains constant. We also show that our model extends beyond the particular case of additive Gaussian noise and we prove an universality result connecting the community detection problem to our Gaussian framework. We unify and generalize a number of recent works on PCA, sparse PCA, submatrix localization or community detection by computing the information-theoretic limits for these problems in the high noise regime. In addition, we show that the posterior distribution of the signal given the observations is characterized by a parameter of the same dimension as the square of the rank of the signal (i.e. scalar in the case of rank one). This allows to locate precisely the information-theoretic thresholds for the above mentioned problems. Finally, we connect our work with the hard but detectable conjecture in statistical physics.

Mathematics Subject Classification

60B20 62B10 60G57 62M15 

References

  1. 1.
    Abbe, E., Sandon, C.: Detection in the stochastic block model with multiple clusters: proof of the achievability conjectures, acyclic BP, and the information-computation gap. Preprint (2015). arXiv:1512.09080
  2. 2.
    Aizenman, M., Sims, R., Starr, S.L.: Extended variational principle for the Sherrington–Kirkpatrick spin-glass model. Phys. Rev. B 68(21), 214403 (2003)CrossRefGoogle Scholar
  3. 3.
    Baik, J., Arous, G.B., Péché, S.: Phase transition of the largest eigenvalue for nonnull complex sample covariance matrices. Ann. Probab. 33, 1643–1697 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bandeira, A.S., Boumal, N., Singer, A.: Tightness of the maximum likelihood semidefinite relaxation for angular synchronization. Math. Program. 163, 1–23 (2016)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Banks, J., Moore, C., Vershynin, R., Verzelen, N., Xu, J.: Information-theoretic bounds and phase transitions in clustering, sparse PCA, and submatrix localization. In: 2017 IEEE International Symposium on Information Theory (ISIT), pp. 1137–1141. IEEE (2017)Google Scholar
  6. 6.
    Barbier, J., Dia, M., Macris, N., Krzakala, F., Lesieur, T., Zdeborová, L.: Mutual information for symmetric rank-one matrix estimation: a proof of the replica formula. In: Advances in Neural Information Processing Systems, pp. 424–432 (2016)Google Scholar
  7. 7.
    Benaych-Georges, F., Nadakuditi, R.R.: The eigenvalues and eigenvectors of finite, low rank perturbations of large random matrices. Adv. Math. 227(1), 494–521 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Boucheron, S., Lugosi, G., Massart, P.: Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford (2013)CrossRefzbMATHGoogle Scholar
  9. 9.
    Caltagirone, F., Lelarge, M., Miolane, L.: Recovering asymmetric communities in the stochastic block model. IEEE Trans. Netw. Sci. Eng. (2017).  https://doi.org/10.1109/TNSE.2017.2758201
  10. 10.
    Chatterjee, S., et al.: A generalization of the lindeberg principle. Ann. Probab. 34(6), 2061–2076 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Coja-Oghlan, A., Krzakala, F., Perkins, W., Zdeborova, L.: Information-theoretic thresholds from the cavity method. Preprint (2016). arXiv:1611.00814
  12. 12.
    Decelle, A., Krzakala, F., Moore, C., Zdeborová, L.: Asymptotic analysis of the stochastic block model for modular networks and its algorithmic applications. Phys. Rev. E 84(6), 066106 (2011)CrossRefGoogle Scholar
  13. 13.
    Deshpande, Y., Abbe, E., Montanari, A.: Asymptotic mutual information for the balanced binary stochastic block model. Inf. Inference: A J. IMA 6(2), 125–170 (2016)MathSciNetzbMATHGoogle Scholar
  14. 14.
    Deshpande, Y., Montanari, A.: Information-theoretically optimal sparse PCA. In: 2014 IEEE International Symposium on Information Theory, pp. 2197–2201. IEEE (2014)Google Scholar
  15. 15.
    Féral, D., Péché, S.: The largest eigenvalue of rank one deformation of large wigner matrices. Commun. Math. Phys. 272(1), 185–228 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Ghirlanda, S., Guerra, F.: General properties of overlap probability distributions in disordered spin systems. Towards parisi ultrametricity. J. Phys. A: Math. Gen. 31(46), 9149 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Guerra, F.: Broken replica symmetry bounds in the mean field spin glass model. Commun. Math. Phys. 233(1), 1–12 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Guo, D., Shamai, S., Verdú, S.: Mutual information and minimum mean-square error in Gaussian channels. IEEE Trans. Inf. Theory 51(4), 1261–1282 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Hajek, B., Wu, Y., Xu, J.: Information limits for recovering a hidden community. In: 2016 IEEE International Symposium on Information Theory (ISIT), pp. 1894–1898. IEEE (2016)Google Scholar
  20. 20.
    Iba, Y.: The Nishimori line and Bayesian statistics. J. Phys. A: Math. Gen. 32(21), 3875 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Korada, S.B., Macris, N.: Exact solution of the gauge symmetric p-spin glass model on a complete graph. J. Stat. Phys. 136(2), 205–230 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Korada, S.B., Macris, N.: Tight bounds on the capacity of binary input random cdma systems. IEEE Trans. Inf. Theory 56(11), 5590–5613 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Korada, S.B., Montanari, A.: Applications of the lindeberg principle in communications and statistical learning. IEEE Trans. Inf. Theory 57(4), 2440–2450 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Krzakala, F., Xu, J., Zdeborová, L.: Mutual information in rank-one matrix estimation. In: Information Theory Workshop (ITW), 2016 IEEE, pp. 71–75. IEEE (2016)Google Scholar
  25. 25.
    Lesieur, T., Krzakala, F., Zdeborová, L.: MMSE of probabilistic low-rank matrix estimation: universality with respect to the output channel. In: 53rd Annual Allerton Conference on Communication, Control, and Computing, Allerton 2015, Allerton Park and Retreat Center, Monticello, IL, USA, September 29–October 2, 2015, pp. 680–687 (2015)Google Scholar
  26. 26.
    Lesieur, T., Krzakala, F., Zdeborová, L.: Phase transitions in sparse PCA. In: IEEE International Symposium on Information Theory, ISIT 2015, Hong Kong, China, June 14–19, 2015, pp. 1635–1639. IEEE (2015)Google Scholar
  27. 27.
    Méasson, C., Montanari, A., Richardson, T.J., Urbanke, R.: The generalized area theorem and some of its consequences. IEEE Trans. Inf. Theory 55(11), 4793–4821 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Mézard, M., Parisi, G., Virasoro, M.: Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications, vol. 9. World Scientific, Singapore (1987)zbMATHGoogle Scholar
  29. 29.
    Milgrom, P., Segal, I.: Envelope theorems for arbitrary choice sets. Econometrica 70(2), 583–601 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Montanari, A.: Estimating random variables from random sparse observations. Eur. Trans. Telecommun. 19(4), 385–403 (2008)CrossRefGoogle Scholar
  31. 31.
    Montanari, A.: Finding one community in a sparse graph. J. Stat. Phys. 161(2), 273–299 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Neeman, J., Netrapalli, P.: Non-reconstructability in the stochastic block model. Preprint (2014). arXiv:1404.6304
  33. 33.
    Nishimori, H.: Statistical Physics of Spin Glasses and Information Processing: An Introduction, vol. 111. Clarendon Press, Oxford (2001)CrossRefzbMATHGoogle Scholar
  34. 34.
    Panchenko, D.: The Sherrington–Kirkpatrick Model. Springer Science & Business Media, Berlin (2013)CrossRefzbMATHGoogle Scholar
  35. 35.
    Perry, A., Wein, A.S., Bandeira, A.S., Moitra, A.: Optimality and sub-optimality of PCA for spiked random matrices and synchronization. Preprint (2016). arXiv:1609.05573
  36. 36.
    Rangan, S., Fletcher, A.K.: Iterative estimation of constrained rank-one matrices in noise. In: 2012 IEEE International Symposium on Information Theory Proceedings (ISIT), pp. 1246–1250. IEEE (2012)Google Scholar
  37. 37.
    Saade, A., Lelarge, M., Krzakala, F., Zdeborová, L.: Spectral detection in the censored block model. In: 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1184–1188. IEEE (2015)Google Scholar
  38. 38.
    Saade, A., Lelarge, M., Krzakala, F., Zdeborová, L.: Clustering from sparse pairwise measurements. In: 2016 IEEE International Symposium on Information Theory (ISIT), pp. 780–784. IEEE (2016)Google Scholar
  39. 39.
    Talagrand, M.: Mean Field Models for Spin Glasses: Volume I: Basic Examples, vol. 54. Springer Science & Business Media, Berlin (2010)zbMATHGoogle Scholar
  40. 40.
    Talagrand, M.: Mean Field Models for Spin Glasses: Volume II: Advanced Replica-Symmetry and Low Temperature, vol. 55. Springer Science & Business Media, Berlin (2011)CrossRefzbMATHGoogle Scholar
  41. 41.
    Wu, Y., Verdú, S.: Functional properties of minimum mean-square error and mutual information. IEEE Trans. Inf. Theory 58(3), 1289–1301 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Zdeborová, L., Krzakala, F.: Statistical physics of inference: thresholds and algorithms. Adv. Phys. 65(5), 453–552 (2016)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Département d’informatique de l’ENS, École normale supérieure, CNRSPSL Research UniversityParisFrance
  2. 2.InriaParisFrance

Personalised recommendations