Decrementing hamming and Bayesian neural networks: Analog implementations and relative performance

  • Vernon G. Dobson
  • José Ma Salinas
Neural Network Architectures And Algorithms
Part of the Lecture Notes in Computer Science book series (LNCS, volume 540)


A review of recent patent applications indicates that neural networks using Hamming-type algorithms with minimum-mismatch selection provide an optimal combination of implementational simplicity, information storage capacity and signal-noise characteristics. These networks can be adapted to implement Bayes' rule, by setting link gains to the negative logarithm of conditional or a priori probabilities. Where probability distributions and noise are not uniform or random, the performance of Bayesian classifiers may be significantly better than that of the corresponding Hamming network on the same vector set. We demonstrate this for the noisy digit classification task. We also generate biologically plausible curvature detectors for character recognition and compare the performances of Bayesian and Hamming networks at classifying the resultant vectors. Preliminary results suggest that Hamming networks may provide good approximations to the Bayes optimum for sparse natural vector sets under some conditions.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

10 References

  1. Baum E.B., Moody J., Wilczec F., 1988 Biological Cybernetics, 59, 217–228.Google Scholar
  2. Blake A. 1983 The least disturbance principle and weak constraints. Pattern Recognition Letters, 1, 393–399.Google Scholar
  3. Burton G.J., Haig N.D., Moorhead I.R. 1986 A Self-similar stack model for human and machine vision. Biological Cybernetics, 53, 397–403.Google Scholar
  4. Denker J.S. 1988 Highly parallel computation network employing a binary-valued T matrix and single output amplifiers. United States Patent Number 4,737,929 (filed Apr. 14, 1986).Google Scholar
  5. Denker J.S. 1986 Neural network models of learning and adaptation. Physica D, vol. 22, 216–243.Google Scholar
  6. Devijver P., Kittler J., 1982 Pattern Recognition: a Statistical Approach. (Prentice-Hall:N.J.).Google Scholar
  7. Dobson V.G., 1975 Pattern learning and the control of behaviour in inhibitory neural network hierarchies. Perception 4, 35–50.Google Scholar
  8. Dobson V.G., 1988 Signal Processing. International Patent Application. WO 88/01079 (filed Aug. 8, 1986)Google Scholar
  9. Dobson V.G., 1987 Superior accuracy of Decrementing over Incrementing associative networks when operating in initially random connectivities. Journal of Intelligent Systems. 1, 43–78.Google Scholar
  10. Dobson V.G., 1989 Decrementing Networks, In L. Personnaz, G. Dreyfus, Eds. Neural Networks: from Models to Applications (IDSET:Paris) 316–325.Google Scholar
  11. Dobson V.G., Salinas R.M. Decrementing Hamming and Bayesian neural networks. Journal of Intelligent Systems. In press.Google Scholar
  12. Duda R.O., Hart P.E., 1973 Pattern Classification and Scene Analysis. (J. Wiley:New York).Google Scholar
  13. Gardner E. 1988 The space of interactions in neural network models. Journal of physiology, A, MathematicsGoogle Scholar
  14. General 21 271–279. Goldstein M. Dillon W.R. 1978 Discrete Discriminant Analysis. (J.Wiley: New York)Google Scholar
  15. Graf H.P., Jackel L.D., Hubbard W.E., 1988 VLSI implementation of a neural network model. Computer: Institute of Electrical and Electronic Engineers, March, 41–49.Google Scholar
  16. Hopfield J.J., 1982 Neural networks and physical systems with emergent collective computational abilities. Proceedings of National Academy of Science, U.S.A. 79, 2554–2558.Google Scholar
  17. Hopfield J.J., 1984 Neurons with graded responses have collective computational properties like those of two-state neurons. Proc. Natl. Acad. Sci. U.S.A. 81, 3088.Google Scholar
  18. Hopfield J.J., Tank D.W., 1986 Computing with neural circuits: a model. Science 233, 625.Google Scholar
  19. Hopfield J.J., 1987 Electronic network for collective decision based on large number of connections between signals. United States Patent Number 4,660,166 (filed Jan. 22, 1985).Google Scholar
  20. Kohonen T., Barna, G., Chrisley R., 1988 Statistical pattern recognition with neural networks: benchmarking studies. Proceedings of 2nd International Conference on Neural networks; Institute of Electrical and Electronic Engineers, San Diego I 61–69.Google Scholar
  21. Kononenko I, 1989 Bayesian Neural Networks. Biological Cybernetics 61, 361–370.Google Scholar
  22. Lansner A., Ekeberg O., 1985 Reliability and speed of recall in an associative network. Institute of Electrical and Electronic Engineers: Transactions on Pattern Analysis and Machine Intelligence 7(4) 490–498.Google Scholar
  23. Linsker R. 1988 Self-organization in a perceptual network. Computer: Institute of Electrical and Electronic Engineers, March, 105–117.Google Scholar
  24. Lippmann R.P., 1987 An introduction to computing with neural nets. Institute of Electrical and Electronic Engineers ASSP Magazine. April, 4–22.Google Scholar
  25. Moopenn A., Thakoor A.P., 1989 Hybrid analog-digital associative neural networks. United States Patent Number 4,807,168 (filed Jun. 10, 1987).Google Scholar
  26. Moopenn A., Thakoor A.P., Duong T. 1988 A neural network for Euclidean distance minimisation. Proceedings of 2nd International Conference on Neural networks; Institute of Electrical and Electronic Engineers, San Diego I, 349–356.Google Scholar
  27. Nadal J. Toulouse G 1990 Information storage in sparsely coded memory nets. Network 1 61–74.Google Scholar
  28. Tarassenko L., Seifert B.G., Toombs J.N., 1989 Neural network architectures for associative memory Proceedings 1st. International Conference on Analogue Neural Networks. Institute of Electrical Engineers Publications. 313, 17–22.Google Scholar
  29. Thorpe S.J., Imbert M. 1989 Biological constraints on connectionist modelling. In Pfeiffer, R. Connectionism in Perspective. (J.Wiley: New York) 316–325. Willshaw D.J., Buneman O.P., Longuet-Higgins C.H., 1969 Non-holographic associative memory. Nature, 222, 960–962.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1991

Authors and Affiliations

  • Vernon G. Dobson
    • 1
  • José Ma Salinas
    • 2
  1. 1.Department of Experimental PsychologyOxfordUK
  2. 2.Departamento de Sociologia y PsicologiaUniversidad to GranadaEspaña

Personalised recommendations