Advertisement

Minimally disturbing learning

  • V. Ruiz de Angulo
  • Carme Torras
Neural Network Architectures And Algorithms
Part of the Lecture Notes in Computer Science book series (LNCS, volume 540)

Abstract

We have undertaken the study of methods for avoiding catastrophic forgetting in feedforward neural networks, without sacrifying the benefits of distributed representations. We formalize the problem as the minimization of the error over the previously learned input-output (i–o) patterns, subject to the constraint of perfect encoding of the new pattern. Then we transform this constrained optimization problem into an unconstrained one. This new formulation naturally leads to an algorithm for solving the problem, wihch we call Minimally Disturbing Learning (MDL). Some experimental comparisons of the performance of MDL with back-propagation are provided which, besides showing the advantages of using MDL, reveal the dependence of forgetting on the learning rate in back-propagation.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [Becker et al, 1989]
    S. Becker, Y. Le Cun. Improving the convergence of Back-propagation learning with second order methods. In Touretzky, D., Hinton, F., and Sejnowski, T., editors. Proc. of the 1988 Connectionist Models Summer School, pp. 29–37, San Mateo. Morgan Kuffman.Google Scholar
  2. [French, 1991]
    R.M. French. Using semi-distributed representations to overcome catastrophic forgetting in connectionist networks. CRCC Technical Report 51-1991. Center for Research on Concepts and Cognition. Indiana University.Google Scholar
  3. [Grossman et al. 1988]
    T. Grossman, R. Meir, E. Domany. Learning by choice of internal representations. Complex Systems 2 (1988) 555–575.Google Scholar
  4. [Hinton and Sejnowski 1988]
    G.E. Hinton and T.J. Sejnowski. Learning and Relearning in Boltzman machines. In D.E. Rumelhart & J.L. McCLelland, Parallel distributed processing: Explorations in the microstructure of cognition. Vol 1: Foundations. Cambridge, MA: MIT press.Google Scholar
  5. [Karnin, 1990]
    E. D. Karnin. A simple procedure for pruning Back-propagation trained Neural Networks. IEEE Transactions on Neural Networks Vol 1. No 2, June 1990. Neural Information Processing Systems. Ed. by David S. Touretzky, 1990 Morgan Kauffman Publishers.Google Scholar
  6. [Krogh et al. 1990]
    A. Krogh, C.J. Thorbergsson and J.A. Hertz. A cost function for internal representations.Google Scholar
  7. [Le Cun et al, 1990]
    Y. Le Cun, J. S. Denker and S. A. Solla. Optimal Brain Damage. In Advances in Neural Information Processing Systems. Ed. by David S. Touretzky, 1990 Morgan Kauffman Publishers.Google Scholar
  8. [Ratcliff, 1990]
    R. Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and for getting functions. Psychological Review 1990 Vol. 97 No. 2, 235–308.Google Scholar
  9. [Rohwer 1990]
    R. Rohwer.The moving target training algorithm. In Advances in Neural Information Processing Systems. Ed. by David S. Touretzky, 1990 Morgan Kauffman Publishers.Google Scholar
  10. [Rumelhart et al, 1986]
    D. E. Rumelhart, G. E. Hinton, R.J. Williams. Learning internal representations by error propagation. In D.E. Rumelhart & J.L. McCLelland, Parallel distributed processing: Explorations in the microstructure of cognition. Vol 1: Foundations. Cambridge, MA: MIT press.Google Scholar
  11. [Torras 1989]
    Relaxation and Neural Learning:Points of Convergence and Divergence. Journal of Parallel and Distributed Computing 6, pp. 217–244.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1991

Authors and Affiliations

  • V. Ruiz de Angulo
    • 1
  • Carme Torras
    • 2
  1. 1.Commission of the European Communities Joint Research Centre - Ispra SiteInstitute for Systems Engineering and Informatics Neural Networks LaboratoryIspra (Va)Italy
  2. 2.Institut de Cibernetica (CSIC-UPC)BarcelonaSpain

Personalised recommendations