Adaptive Thresholds for Layered Neural Networks with Synaptic Noise

  • D. Bollé
  • R. Heylen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4131)


The inclusion of a macroscopic adaptive threshold is studied for the retrieval dynamics of layered feedforward neural network models with synaptic noise. It is shown that if the threshold is chosen appropriately as a function of the cross-talk noise and of the activity of the stored patterns, adapting itself automatically in the course of the recall process, an autonomous functioning of the network is guaranteed. This self-control mechanism considerably improves the quality of retrieval, in particular the storage capacity, the basins of attraction and the mutual information content.


Mutual Information Optimal Threshold Associative Memory Sparse Code Adaptive Threshold 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Willshaw, D.J., Buneman, O.P., Longuet-Higgins, H.C.: Nonholographic associative memory. Nature 222, 960 (1969)CrossRefGoogle Scholar
  2. 2.
    Palm, G.: On the storage capacity of an associative memory with random distributed storage elements. Biol. Cyber. 39, 125 (1981)MATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Gardner, E.: The space of interactions in neural network models. J. Phys. A: Math. Gen. 21, 257 (1988)CrossRefGoogle Scholar
  4. 4.
    Okada, M.: Notions of associative memory and sparse coding. Neural Networks 9, 1429 (1996)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Dominguez, D.R.C., Bollé, D.: Self-control in sparsely coded networks. Phys. Rev. Lett. 80, 2961 (1998)CrossRefGoogle Scholar
  6. 6.
    Bollé, D., Dominguez, D.R.C., Amari, S.: Mutual information of sparsely coded associative memory with self-control and ternary neurons. Neural Networks 13, 455 (2000)CrossRefGoogle Scholar
  7. 7.
    Bollé, D., Heylen, R.: Self-control dynamics for sparsely coded networks with synaptic noise. In: 2004 Proceedings of the IEEE International Joint Conference on Neural Networks, p. 3195 (2004)Google Scholar
  8. 8.
    Dominguez, D.R.C., Korutcheva, E., Theumann, W.K., Erichsen Jr., R.: Flow diagrams of the quadratic neural network. In: Dorronsoro, J.R. (ed.) ICANN 2002. LNCS, vol. 2415, p. 129. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  9. 9.
    Bollé, D., Massolo, G.: Thresholds in layered neural networks with variable activity. J. Phys. A: Math. Gen. 33, 2597 (2000)MATHCrossRefGoogle Scholar
  10. 10.
    Hertz, J., Krogh, A., Palmer, R.G.: Introduction to the Theory of Neural Computation. Addison-Wesley, Redwood City (1991)Google Scholar
  11. 11.
    Domany, E., Kinzel, W., Meir, R.: Layered Neural Networks. J.Phys. A: Math. Gen. 22, 2081 (1989)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Bollé, D.: Multi-state neural networks based upon spin-glasses: a biased overview. In: Korutcheva, E., Cuerno, R. (eds.) Advances in Condensed Matter and Statistical Mechanics, pp. 321–349. Nova Science Publishers, New-York (2004)Google Scholar
  13. 13.
    Nadal, J.-P., Brunel, N., Parga, N.: Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction. Network: Computation in Neural Systems 9, 207 (1998)MATHCrossRefGoogle Scholar
  14. 14.
    Schultz, S., Treves, A.: Stability of the replica-symmetric solution for the information conveyed by a neural network. Phys. Rev. E 57, 3302 (1998)CrossRefGoogle Scholar
  15. 15.
    Blahut, R.E.: Principles and Practice of Information Theory. Addison-Wesley, Reading (1990)Google Scholar
  16. 16.
    Shannon, C.E.: A mathematical theory for communication. Bell Systems Technical Journal 27, 379 (1948)MATHMathSciNetGoogle Scholar
  17. 17.
    Schwenker, F., Sommer, F.T., Palm, G.: Iterative retrieval of sparsely coded associative memory patterns. Neural Networks 9, 445 (1996)CrossRefGoogle Scholar
  18. 18.
    Amari, S.: Neural theory and association of concept information. Biol. Cyber. 26, 175 (1977)MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Amari, S., Maginu, K.: Statistical neurodynamics of associative memory. Neural Networks 1, 63 (1988)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • D. Bollé
    • 1
  • R. Heylen
    • 1
  1. 1.Institute for Theoretical PhysicsKatholieke Universiteit LeuvenLeuvenBelgium

Personalised recommendations