Abstract:
Equilibrium states of large layered neural networks with differentiable activation function and a single, linear output unit are investigated using the replica formalism. The quenched free energy of a student network with a very large number of hidden units learning a rule of perfectly matching complexity is calculated analytically. The system undergoes a first order phase transition from unspecialized to specialized student configurations at a critical size of the training set. Computer simulations of learning by stochastic gradient descent from a fixed training set demonstrate that the equilibrium results describe quantitatively the plateau states which occur in practical training procedures at sufficiently small but finite learning rates.
Similar content being viewed by others
Author information
Authors and Affiliations
Additional information
Received 16 December 1998
Rights and permissions
About this article
Cite this article
Ahr, M., Biehl, M. & Urbanczik, R. Statistical physics and practical training of soft-committee machines. Eur. Phys. J. B 10, 583–588 (1999). https://doi.org/10.1007/s100510050889
Issue Date:
DOI: https://doi.org/10.1007/s100510050889