Skip to main content

Advertisement

SpringerLink
Log in

Free Energy of Multi-Layer Generalized Linear Models

  • Published:
Communications in Mathematical Physics Aims and scope Submit manuscript

Abstract

We compute the high-dimensional limit of the free energy associated with a multi-layer generalized linear model. Under certain technical assumptions, we identify the limit in terms of a variational formula. The approach is to first show that the limit is a solution to a Hamilton–Jacobi equation whose initial condition is related to the limiting free energy of a model with one fewer layer. Then, we conclude with an iteration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Barbier, J., Krzakala, F., Macris, N., Miolane, L., Zdeborová, L.: Optimal errors and phase transitions in high-dimensional generalized linear models. Proc. Natl. Acad. Sci. 116(12), 5451–5460 (2019)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  2. Barbier, J., Macris, N.: The adaptive interpolation method: a simple scheme to prove replica formulas in Bayesian inference. Probab. Theory Relat. Fields 174(3–4), 1133–1185 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  3. Barbier, J., Macris, N.: The adaptive interpolation method for proving replica formulas. Applications to the Curie–Weiss and Wigner spike models. J. Phys. A Math. Theor. 52(29), 294002 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  4. Barbier, J., Macris, N., Miolane, L.: The layered structure of tensor estimation and its mutual information. In: 55th Annual Allerton Conference on Communication, Control, and Computing, pp. 1056–1063. IEEE, (2017)

  5. Bardi, M., Evans, L.C.: On Hopf’s formulas for solutions of Hamilton–Jacobi equations. Nonlinear Anal. Theory Methods Appl. 8(11), 1373–1381 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  6. Barra, A., Dal Ferraro, G., Tantari, D.: Mean field spin glasses treated with PDE techniques. Eur. Phys. J. B 86(7), 1–10 (2013)

    Article  MathSciNet  Google Scholar 

  7. Barra, A., Di Biasio, A., Guerra, F.: Replica symmetry breaking in mean-field spin glasses through the Hamilton–Jacobi technique. J. Stat. Mech. Theory Exp. 2010(09), P09006 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  8. Boucheron, S., Lugosi, G., Massart, P.: Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford University Press, Oxford (2013)

    Book  MATH  Google Scholar 

  9. Chen, H.-B.: Hamilton–Jacobi equations for nonsymmetric matrix inference. arXiv preprint arXiv:2006.05328 (2020)

  10. Chen, H.-B., Mourrat, J.-C., Xia, J.: Statistical inference of finite-rank tensors. arXiv preprint arXiv:2104.05360 (2021)

  11. Chen, H.-B., Xia, J.: Fenchel–Moreau identities on self-dual cones. arXiv preprint arXiv:2011.06979 (2020)

  12. Chen, H.-B., Xia, J.: Hamilton–Jacobi equations for inference of matrix tensor products. arXiv preprint arXiv:2009.01678 (2020)

  13. Dia, M., Macris, N., Krzakala, F., Lesieur, T., Zdeborová, L., et al.: Mutual information for symmetric rank-one matrix estimation: a proof of the replica formula. Adv. Neural Inf. Process. Syst. 29, 424–432 (2016)

    Google Scholar 

  14. Evans, L.C.: Partial Differential Equations, vol. 19. American Mathematical Society, Providence (2010)

    MATH  Google Scholar 

  15. Gabrié, M., Manoel, A., Luneau, C., Barbier, J., Macris, N., Krzakala, F., Zdeborová, L.: Entropy and mutual information in models of deep neural networks. J. Stat. Mech. Theory Exp. 2019(12), 124014 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  16. Genovese, G., Barra, A.: A mechanical approach to mean field spin models. J. Math. Phys. 50(5), 053303 (2009)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  17. Guerra, F.: Sum rules for the free energy in the mean field spin glass model. Fields Inst. Commun. 30(11), 161–170 (2001)

    MathSciNet  MATH  Google Scholar 

  18. Lions, P.-L., Rochet, J.-C.: Hopf formula and multitime Hamilton–Jacobi equations. Proc. Am. Math. Soc. 96(1), 79–84 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  19. Luneau, C., Barbier, J., Macris, N.: Mutual information for low-rank even-order symmetric tensor estimation. Inf. Inference A J. IMA 10(4), 1167–1207. arXiv:1904.04565 (2020). https://doi.org/10.1093/imaiai/iaaa022

  20. Luneau, C., Macris, N., Barbier, J.: High-dimensional rank-one nonsymmetric matrix decomposition: the spherical case. arXiv preprint arXiv:2004.06975 (2020)

  21. Mourrat, J.-C.: Free energy upper bound for mean-field vector spin glasses. arXiv preprint arXiv:2010.09114 (2020)

  22. Mourrat, J.-C.: Hamilton–Jacobi equations for mean-field disordered systems. Ann. Henri Lebesgue 4, 453–484 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  23. Mourrat, J.-C.: Nonconvex interactions in mean-field spin glasses. Probab. Math. Phys. 2(2), 61–119 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  24. Mourrat, J.-C.: The Parisi formula is a Hamilton–Jacobi equation in Wasserstein space. Can. J. Math. 3, 1–23 (2021)

    Google Scholar 

  25. Mourrat, J.-C.: Hamilton–Jacobi equations for finite-rank matrix inference. Ann. Appl. Probab. 30(5), 2234–2260 (2020). https://doi.org/10.1214/19-AAP1556. https://projecteuclid.org/journals/annals-of-applied-probability/volume-30/issue-5/HamiltonJacobi-equations-for-finite-rank-matrix-inference/10.1214/19-AAP1556.short@article%7B10.1214/19-AAP1556

  26. Mourrat, J.-C., Panchenko, D.: Extending the Parisi formula along a Hamilton–Jacobi equation. Electron. J. Probab. 25, 1–17 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  27. Reeves, G.: Information-theoretic limits for the matrix tensor product. IEEE J. Sel. Areas Inf. Theory 1(3), 777–798 (2020)

    Article  Google Scholar 

  28. Rockafellar, R.T.: Convex Analysis, vol. 36. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

Download references

Acknowledgements

We warmly thank Jean-Christophe Mourrat for many helpful comments and discussions. We thank the referees for the comments that help us improve the paper considerably.

Author information

Authors and Affiliations

  1. Courant Institute of Mathematical Sciences, New York University, New York, NY, USA

    Hong-Bin Chen

  2. Department of Mathematics, University of Pennsylvania, Philadelphia, PA, USA

    Jiaming Xia

Authors
  1. Hong-Bin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiaming Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiaming Xia.

Additional information

Communicated by S. Chatterjee.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, HB., Xia, J. Free Energy of Multi-Layer Generalized Linear Models. Commun. Math. Phys. 400, 1861–1913 (2023). https://doi.org/10.1007/s00220-022-04630-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00220-022-04630-4

Search

Navigation