Skip to main content
SpringerLink
Log in

Generalization Learning in a Perceptron with Binary Synapses

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

We consider the generalization problem for a perceptron with binary synapses, implementing the Stochastic Belief-Propagation-Inspired (SBPI) learning algorithm which we proposed earlier, and perform a mean-field calculation to obtain a differential equation which describes the behaviour of the device in the limit of a large number of synapses N. We show that the solving time of SBPI is of order \(N\sqrt{\log N}\) , while the similar, well-known clipped perceptron (CP) algorithm does not converge to a solution at all in the time frame we considered. The analysis gives some insight into the ongoing process and shows that, in this context, the SBPI algorithm is equivalent to a new, simpler algorithm, which only differs from the CP algorithm by the addition of a stochastic, unsupervised meta-plastic reinforcement process, whose rate of application must be less than \(\sqrt{2/(\pi N)}\) for the learning to be achieved effectively. The analytical results are confirmed by simulations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baldassi, C., Braunstein, A., Brunel, N., Zecchina, R.: Efficient supervised learning in networks with binary synapses. Proc. Natl. Acad. Sci. USA 104, 2079–2084 (2007)

    Article  Google Scholar 

  2. Bhalla, U.S., Iyengar, R.: Emergent properties of networks of biological signaling pathways. Science 283, 381–387 (1999)

    Article  ADS  Google Scholar 

  3. Bialek, W.: Stability and noise in biochemical switches. Adv. Neural Inf. Process. Syst. 13, 103–109 (2000)

    Google Scholar 

  4. Blum, A.L., Rivest, R.L.: Training a 3-node network is np-complete. Neural Netw. 5, 117–127 (1992)

    Article  Google Scholar 

  5. Braunstein, A., Zecchina, R.: Learning by message-passing in networks of discrete synapses. Phys. Rev. Lett. 96, 030,201 (2006)

    Article  MathSciNet  Google Scholar 

  6. Engel, A., van den Broeck, C.: Statistical Mechanics of Learning. Cambridge University Press, Cambridge (2001)

    MATH  Google Scholar 

  7. Fusi, S., Drew, P.J., Abbott, L.F.: Cascade models of synaptically stored memories. Neuron 45(4), 599–611 (2005)

    Article  Google Scholar 

  8. Golea, M., Marchand, M.: On learning perceptrons with binary weights. Neural Comput. 78, 333–342 (1993)

    Google Scholar 

  9. Gutfreund, H., Stein, Y.: Capacity of neural networks with discrete synaptic couplings. J. Phys. A Math. Gen. 23, 2613–2630 (1990)

    Article  MATH  ADS  MathSciNet  Google Scholar 

  10. Hayer, A., Bhalla, U.S.: Molecular switches at the synapse emerge from receptor and kinase traffic. PLoS Comput. Biol. 1(2), e20 (2005)

    Article  Google Scholar 

  11. Kabashima, Y., Uda, S.: A BP-based algorithm for performing Bayesian inference in large perceptron-type networks. In: Algorithmic Learning Theory. Lecture Notes in Computer Science, vol. 3244, pp. 479–493. Springer, Berlin (2004)

    Google Scholar 

  12. Kinouchi, O., Caticha, N.: Optimal generalization in perceptrons. J. Phys. A 25, 6243 (1992)

    Article  MATH  ADS  MathSciNet  Google Scholar 

  13. Krauth, W., Mézard, M.: Storage capacity of memory networks with binary couplings. J. Phys. 50, 3057 (1989)

    Google Scholar 

  14. Miller, P., Zhabotinsky, A.M., Lisman, J.E., Wang, X.J.: The stability of a stochastic camkii switch: dependence on the number of enzyme molecules and protein turnover. PLoS Biology 3(4), e107 (2005)

    Article  Google Scholar 

  15. O’Connor, D.H., Wittenberg, G.M., Wang, S.S.H.: Graded bidirectional synaptic plasticity is composed of switch-like unitary events. Proc. Natl. Acad. Sci. USA 102, 9679–9684 (2005)

    Article  ADS  Google Scholar 

  16. Petersen, C.C., Malenka, R.C., Nicoll, R.A., Hopfield, J.J.: All-or-none potentiation at CA3-CA1 synapses. Proc. Natl. Acad. Sci. USA 95, 4732–4737 (1998)

    Article  ADS  Google Scholar 

  17. Rosenblatt, F.: Principles of Neurodynamics. Spartan Books, New York (1962)

    MATH  Google Scholar 

  18. Rosen-Zvi, M.: On-line learning in the ising perceptron. J. Phys. A 33, 7277–7287 (2000)

    Article  MATH  ADS  MathSciNet  Google Scholar 

  19. Solla, S.A., Winther, O.: Optimal perceptron learning: an online Bayesian approach. In: On-Line Learning in Neural Networks. Cambridge University Press, Cambridge (1998)

    Google Scholar 

  20. Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. In: Exploring Artificial Intelligence in the New Millennium, pp. 236–239. Morgan Kaufman, San Mateo (2003). Chap. 8

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Fisica, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129, Torino, Italy

    Carlo Baldassi

  2. Institute for Scientific Interchange Foundation, Viale Settimio Severo 65, 10133, Torino, Italy

    Carlo Baldassi

Authors
  1. Carlo Baldassi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlo Baldassi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Baldassi, C. Generalization Learning in a Perceptron with Binary Synapses. J Stat Phys 136, 902–916 (2009). https://doi.org/10.1007/s10955-009-9822-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-009-9822-1

Keywords

Search

Navigation