Abstract
Lateral predictive coding is a recurrent neural network that creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here, we analytically investigate the trade-off between information robustness and energy in a linear model of lateral predictive coding and numerically minimize a free energy quantity. We observed several phase transitions in the synaptic weight matrix, particularly a continuous transition that breaks reciprocity and permutation symmetry and builds cyclic dominance and a discontinuous transition with the associated sudden emergence of tight balance between excitatory and inhibitory interactions. The optimal network follows an ideal gas law over an extended temperature range and saturates the efficiency upper bound of energy use. These results provide theoretical insights into the emergence and evolution of complex internal models in predictive processing systems.
Article PDF
Avoid common mistakes on your manuscript.
References
L. R. Swanson, Front. Syst. Neurosci. 10, (2016).
N. J. Wade, J. History Neurosci. 30, 405 (2021).
H. B. Barlow, in Possible principles underlying the transformations of sensory messages: Sensory Communication, edited by W. A. Rosenblith (MIT Press, New York, 1961), pp. 217–234.
Y. Huang, and R. P. N. Rao, WIRES Cogn. Sci. 2, 580 (2011).
C. S. Kim, Biol. Cybern. 115, 87 (2021).
K. Friston, J. Kilner, and L. Harrison, J. Physiol.-Paris 100, 70 (2006).
M. Aguilera, B. Millidge, A. Tschantz, and C. L. Buckley, Phys. Life Rev. 40, 24 (2022), arXiv: 2105.11203.
V. Jirsa, and H. Sheheitli, J. Phys. Complex. 3, 015007 (2022).
M. V. Srinivasan, S. B. Laughlin, and A. Dubs, Proc. R. Soc. Lond. B. 216, 427 (1982).
H. Tang, M. Schrimpf, W. Lotter, C. Moerman, A. Paredes, J. Ortega Caro, W. Hardesty, D. Cox, and G. Kreiman, Proc. Natl. Acad. Sci. USA 115, 8835 (2018), arXiv: 1706.02240.
Z. Pang, C. B. O’May, B. Choksi, and R. VanRullen, Neural Netw. 144, 164 (2021).
F. Fumarola, B. Hein, and K. D. Miller, Phys. Rev. X 12, 031024 (2022), arXiv: 2109.02048.
H. Zhao, Sci. China-Phys. Mech. Astron. 64, 270511 (2021), arXiv: 1905.08313.
D. D. Clarke, and L. Sokoloff, in Circulation and energy metabolism in the brain: Basic Neurochemistry: Molecular, Cellular and Medical Aspects, 6th ed., edited by G. J. Siegel, B. W. Agranoff, R. Wayne Albers, S. K. Fisher, and M. D. Uhler (Lippincott-Raven, Philadelphia, 1999), pp. 637–669.
P. Lennie, Curr. Biol. 13, 493 (2003).
Y. Chen, S. Wang, C. C. Hilgetag, and C. Zhou, PLoS Comput. Biol. 9, e1002937 (2013).
J. E. Niven, Curr. Opin. Neurobiol. 41, 129 (2016).
L. Yu, and Y. Yu, J Neurosci. Res. 95, 2253 (2017).
C. Metzner, M. E. Yamakou, D. Voelkl, A. Schilling, and P. Krauss, arXiv: 2301.12892.
C. Jutten, and J. Herault, Signal Process. 24, 1 (1991).
B. A. Olshausen, and D. J. Field, Nature 381, 607 (1996).
G. F. Harpur, and R. W. Prager, Network-Computat. Neural Syst. 7, 277 (1996).
A. Ali, N. Ahmad, E. de Groot, M. A. Johannes van Gerven, and T. C. Kietzmann, Patterns 3, 100639 (2022).
L. Yu, and L. Liu, Phys. Rev. E 89, 032725 (2014), arXiv: 1308.4122.
Z. Padamsey, D. Katsanevaki, N. Dupuy, and N. L. Rochefort, Neuron 110, 280 (2022).
L. Weninger, P. Srivastava, D. Zhou, J. Z. Kim, E. J. Cornblath, M. A. Bertolero, U. Habel, D. Merhof, and D. S. Bassett, Phys. Rev. E 106, 014401 (2022), arXiv: 2110.13781.
H. Huang, Statistical Mechanics of Neural Networks (Higher Education Press, Beijing, 2022).
Z. Y. Huang, X. Y. Fan, J. Zhou, and H. J. Zhou, Commun. Theor. Phys. 74, 095601 (2022), arXiv: 2207.09047.
A. J. Bell, and T. J. Sejnowski, Neural Computat. 7, 1129 (1995).
N. Tishby, F. C. Pereira, and W. Bialek, in The information bottleneck method: Proceedings of the 37th Annual Allerton Conference on Communication, Control and Computing (IEEE Press, Piscataway, 1999), pp. 368–377.
B. He, and Y. Wang, Chin. Phys. B 26, 030506 (2017).
Z.-Y. Huang, R. Zhou, M. Huang, and H.-J. Zhou, arXiv: 2302.11681.
A. Hyvärinen, and E. Oja, Neural Netw. 13, 411 (2000).
E. Schneidman, M. J. Berry Ii, R. Segev, and W. Bialek, Nature 440, 1007 (2006), arXiv: q-bio/0512013.
C. van Vreeswijk, and H. Sompolinsky, Science 274, 1724 (1996).
D. P. Yang, H. J. Zhou, and C. Zhou, PLoS Comput. Biol. 13, e1005384 (2017).
L. Yu, Z. Shen, C. Wang, and Y. Yu, Front. Cell. Neurosci. 12, 123 (2018).
J. Liang, S. J. Wang, and C. Zhou, Natl. Sci. Rev. 9, nwab102 (2022).
F. J. Pineda, Phys. Rev. Lett. 59, 2229 (1987).
M. W. Spratling, Brain Cognition 112, 92 (2017).
C. Curto, J. Geneson, and K. Morrison, Neural Computat. 31, 94 (2019).
H. Bauke, and S. Mertens, Phys. Rev. E 75, 066701 (2007), arXiv: cond-mat/0609584.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest The authors declare that they have no conflict of interest.
Additional information
This work was supported by the National Natural Science Foundation of China (Grant Nos. 12047503, 11747601 and 12247104), the National Innovation Institute of Defense Technology (Grant No. 22TQ0904ZT01025). The Numerical Simulations were carried out at the HPC cluster of ITP-CAS and also at the BSCC-A3 platform of the National Supercomputer Center in Beijing with the help of the TRNG random number generators [42].
Supporting Information
The supporting information is available online at http://phys.scichina.com and https://link.springer.com. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.
Electronic supplementary material
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Huang, ZY., Zhou, R., Huang, M. et al. Energy-information trade-off induces continuous and discontinuous phase transitions in lateral predictive coding. Sci. China Phys. Mech. Astron. 67, 260511 (2024). https://doi.org/10.1007/s11433-024-2341-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11433-024-2341-2