Reducing Communication for Distributed Learning in Neural Networks

  • Peter Auer
  • Harald Burgsteiner
  • Wolfgang Maass
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2415)

Abstract

A learning algorithm is presented for circuits consisting of a single layer of perceptrons. We refer to such circuits as parallel perceptrons. In spite of their simplicity, these circuits are universal approximators for arbitrary boolean and continuous functions. In contrast to backprop for multi-layer perceptrons, our new learning algorithm - the parallel delta rule (p-delta rule) - only has to tune a single layer of weights, and it does not require the computation and communication of analog values with high precision. Reduced communication also distinguishes our new learning rule from other learning rules for such circuits such as those traditionally used for MADALINE. A theoretical analysis shows that the p-delta rule does in fact implement gradient descent — with regard to a suitable error measure — although it does not require to compute derivatives. Furthermore it is shown through experiments on common real-world benchmark datasets that its performance is competitive with that of other learning approaches from neural networks and machine learning. Thus our algorithm also provides an interesting new hypothesis for the organization of learning in biological neural systems.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Auer, P., Burgsteiner, H. & Maass, W. The p-delta Learning Rule, submitted for publication, http://www.igi.TUGraz.at/maass/p_delta_learning.pdf.
  2. 2.
    Blake, C.L. & Merz, C.J. (1998). UCI Repository of machine learning databases, http://www.ics.uci.edu/~mlearn/MLRepository.html. Irvine, CA: University of California, Department of Information and Computer Science.Google Scholar
  3. 3.
    N. Cristianini, N., Shawe-Taylor J. (2000). An Introduction to Support Vector Machines, Cambridge University Press.Google Scholar
  4. 4.
    Freund, Y., and Schapire, R. E. (1999). Large margin classification using the Perceptron algorithm, Machine Learning 37(3):277–296.MATHCrossRefGoogle Scholar
  5. 5.
    Guyon I., Boser B., and Vapnik V. (1993). Automatic capacity tuning of very large VC-dimension classifiers, Advances in Neural Information Processing Systems, volume 5, Morgan Kaufmann (San Mateo) 147–155.Google Scholar
  6. 6.
    Maass, W., Natschlaeger, T., and Markram, H. (2001). Real-time computing without stable status: a new framework for neural computation based on perturbations, http://www.igi.TUGraz.at/maass/liquid_state_machines.pdf.
  7. 7.
    Nilsson, N. J. (1990). The Mathematical Foundations of Learning Machines, Morgan Kauffmann Publishers, San Mateo (USA).Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Peter Auer
    • 1
  • Harald Burgsteiner
    • 1
  • Wolfgang Maass
    • 1
  1. 1.Institute for Theoretical Computer ScienceTechnische Universität GrazGrazAustria

Personalised recommendations