A Derivation of Learning Rules for Dynamic Recurrent Neural Networks

Abstract

This chapter brings together mathematical and physical tools which lie at the root of a derivation of the learning rules for dynamic recurrent neural networks. It starts with the basic problem in the calculus of variations of how to determine a set of functions y _i /(x), i = 1,2,…, n, which minimize (or maximize) the integral of some functional of these functions. This problem is generalized by the introduction of constraints such that the solutions also have to satisfy the constraint equations. This results in the method of the ‘Lagrange undetermined multiplier functions’. Application of these methods in physics will be discussed after the introduction of Hamilton’s Principle. The Lagrangian and Hamiltonian energy functions together with the canonical equations of motion of Hamilton are introduced. The way in which Pontryagin applied these methods in the construction of his ‘Minimum Principle of Pontryagin’ for the solution of general optimal control problems is presented. The chapter ends with the derivation of the learning rules.