Skip to main content
SpringerLink
Log in

Learning in Ultrametric Committee Machines

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

The problem of learning by examples in ultrametric committee machines (UCMs) is studied within the framework of statistical mechanics. Using the replica formalism we calculate the average generalization error in UCMs with L hidden layers and for a large enough number of units. In most of the regimes studied we find that the generalization error, as a function of the number of examples presented, develops a discontinuous drop at a critical value of the load parameter. We also find that when L>1 a number of teacher networks with the same number of hidden layers and different overlaps induce learning processes with the same critical points.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Variano, E., AMcCoy, J.H., Lipson, H.: Networks, dynamics, and modularity. Phys. Rev. Lett. 92, 188701 (2004)

    Article  ADS  Google Scholar 

  2. Huerta, R., Rabinovich, M.: Reproducible sequence generation in random neural ensembles. Phys. Rev. Lett. 93, 238104 (2004)

    Article  ADS  Google Scholar 

  3. Yoshioka, M.: Learning of spatiotemporal patterns in Ising-spin neural networks: analysis of storage capacity by path integral methods. Phys. Rev. Lett. 102, 158102 (2009)

    Article  ADS  Google Scholar 

  4. Lin, I.-H., Wu, R.-K., Chen, C.-M.: Synchronization in a noise-driven developing neural network. Phys. Rev. E 84, 051923 (2011)

    Article  ADS  Google Scholar 

  5. Saito, A., Taiji, M., Ikegami, T.: Inaccessibility in online learning of recurrent neural networks. Phys. Rev. Lett. 93, 168101 (2004)

    Article  ADS  Google Scholar 

  6. Fiete, I.R., Seung, H.S.: Gradient learning in spiking neural networks by dynamic perturbation of conductances. Phys. Rev. Lett. 97, 048104 (2006)

    Article  ADS  Google Scholar 

  7. Saito, H., Katahira, K., Okanoya, K., Okada, M.: Statistical mechanics of structural and temporal credit assignment effects on learning in neural networks. Phys. Rev. E 83, 051125 (2011)

    Article  ADS  Google Scholar 

  8. Braunstein, A., Ramezanpour, A., Zecchina, R., Zhang, P.: Inference and learning in sparse systems with multiple states. Phys. Rev. E 83, 056114 (2011)

    Article  ADS  Google Scholar 

  9. Koralek, A.C., Jin, X., Long, J.D. II, Costa, R.M., Carmena, J.M.: Corticostriatal plasticity is necessary for learning intentional neuroprosthetic skills. Nature 483, 331 (2012)

    Article  ADS  Google Scholar 

  10. Bardin, J.: Neuroscience: Making connections. Nature 483, 394 (2012)

    Article  ADS  Google Scholar 

  11. Mesgarani, N., Chang, E.F.: Selective cortical representation of attended speaker in multi-talker speech perception. Nature 485, 233 (2012)

    Article  ADS  Google Scholar 

  12. Seung, H.S., Sompolinsky, H., Tishby, N.: Statistical mechanics of learning by examples. Phys. Rev. A 45, 6056–6091 (1992)

    Article  MathSciNet  ADS  Google Scholar 

  13. Neirotti, J.P.: Can a student learn optimally from two different teachers? J. Phys. A 43, 015101 (2010)

    Article  MathSciNet  ADS  Google Scholar 

  14. Neirotti, J.P.: Parallel strategy for optimal learning in perceptrons. J. Phys. A 43, 125101 (2010)

    Article  MathSciNet  ADS  Google Scholar 

  15. Neirotti, J.P., Franco, L.: Computational capabilities of multilayer committee machines. J. Phys. A 43, 445103 (2010)

    Article  MathSciNet  ADS  Google Scholar 

  16. Franco, L., Anthony, M.: The influence of oppositely classified examples on the generalization complexity of Boolean functions. IEEE Trans. Neural Netw. 17, 578–590 (2006)

    Article  Google Scholar 

  17. Rammal, R., Toulouse, G., Virasoro, M.A.: Ultrametricity for physicists. Rev. Mod. Phys. 58, 765–788 (1986)

    Article  MathSciNet  ADS  Google Scholar 

  18. Engel, A., Köhler, H.M., Tschepke, F., Vollmayr, H., Zippelius, A.: Storage capacity and learning algorithms for two-layer neural networks. Phys. Rev. A 45, 7590–7609 (1992)

    Article  ADS  Google Scholar 

  19. Monasson, R., O’Kane, D.: Domain of solutions and replica symmetry breaking in multilayer neural networks. Europhys. Lett. 27, 85–90 (1994)

    Article  ADS  Google Scholar 

  20. Monasson, R., Zecchina, R.: Weight space structure and internal representations: a direct approach to learning and generalization in multilayer neural networks. Phys. Rev. Lett. 75, 2432–2435 (1995)

    Article  ADS  Google Scholar 

  21. Schwarz, H.: Learning rule in a multilayer neural network. J. Phys. A 26, 5781–5794 (1993)

    Article  MathSciNet  ADS  Google Scholar 

Download references

Acknowledgements

The author would like to acknowledge the friendly criticisms from Dr. Roberto C. Alamino and Dr Laura Rebollo-Neira.

Author information

Authors and Affiliations

  1. NCRG, Aston University, Birmingham, B4 7ET, UK

    J. P. Neirotti

Authors
  1. J. P. Neirotti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. P. Neirotti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Neirotti, J.P. Learning in Ultrametric Committee Machines. J Stat Phys 149, 887–897 (2012). https://doi.org/10.1007/s10955-012-0636-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-012-0636-1

Keywords

Search

Navigation