Advertisement

The influence of the sigmoid function parameters on the speed of backpropagation learning

  • Jun Han
  • Claudio Moraga
Computational Models of Neurons and Neural Nets
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 930)

Abstract

Sigmoid function is the most commonly known function used in feed forward neural networks because of its nonlinearity and the computational simplicity of its derivative. In this paper we discuss a variant sigmoid function with three parameters that denote the dynamic range, symmetry and slope of the function respectively. We illustrate how these parameters influence the speed of backpropagation learning and introduce a hybrid sigmoidal network with different parameter configuration in different layers. By regulating and modifying the sigmoid function parameter configuration in different layers the error signal problem, oscillation problem and asymmetrical input problem can be reduced. To compare the learning capabilities and the learning rate of the hybrid sigmoidal networks with the conventional networks we have tested the two-spirals benchmark that is known to be a very difficult task for backpropagation and their relatives.

Keywords

Hide Layer Sigmoid Function Hide Unit Parameter Configuration Training Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Hornik, K., Stinchcombe, M., and White, H.: Multilayer Feedforward Networks are Universal Approximators. Neural Networks, Vol. 2, no. 5, 359–366, (1989)Google Scholar
  2. [2]
    Kosko, B.: Neural Networks and Fuzzy Systems. Prentice-Hall, INC. (1992)Google Scholar
  3. [3]
    Fahlman, S.E.: An Empirical Study of Learning Speed in Back-Propagation Networks. Technical Report CMU-CS-88-162, CMU, (1988)Google Scholar
  4. [4]
    Fahlman, S.E., Lebiere, C.: The Cascade-Correlation Learning Architecture. in Touretzky (ed.) Advances in Neural Information Processing Systems 2, Morgan-Kaufmann, (1990)Google Scholar
  5. [5]
    Stornetta, W.S., Huberman, B.A.: An improved Three-Layer, Back Propagation Algorithm. IEEE First Int. Conf. on Neural Networks, (1987)Google Scholar
  6. [6]
    Little, W.A.: The Existence of Persistent States in the Brain. Math. Biosci. 19, 101, (1974)Google Scholar
  7. [7]
    Little, W.A., Shaw, G.L.: Analytic Study of the Memory Capacity of a Neural Network. Math. Biosci. 39, 281, (1978)Google Scholar
  8. [8]
    Müller, B., Reinhardt. J.: Neural Networks. Springer-Verlag, (1990)Google Scholar
  9. [9]
    Hertz, J., Krogh, A., Palmer, R.G.: Introduction to the theory of neural computation. Addison-Wesley publishing Company, (1991)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1995

Authors and Affiliations

  • Jun Han
    • 1
  • Claudio Moraga
    • 1
  1. 1.Research Group Computational Intelligence Dept. of Computer ScienceUniversity of DortmundDortmundGermany

Personalised recommendations

Cite paper

Buy options