Continuous Function Optimisation via Gradient Descent on a Neural Network Approxmiation Function

  • Kate A. Smith
  • Jatinder N. D. Gupta
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2084)


Existing neural network approaches to optimisation problems are quite limited in the types of optimisation problems that can be solved. Convergence theorems that utilise Liapunov functions limit the applicability of these techniques to minimising usually quadratic functions only. This paper proposes a new neural network approach that can be used to solve a broad variety of continuous optimisation problems since it makes no assumptions about the nature of the objective function. The approach comprises two stages: first a feedforward neural network is used to approximate the optimisation function based on a sample of evaluated data points; then a feedback neural network is used to perform gradient descent on this approximation function. The final solution is a local minima of the approximated function, which should coincide with true local minima if the learning has been accurate. The proposed method is evaluated on the De Jong test suite: a collection of continuous optimisation problems featuring various characteristics such as saddlepoints, discontinuities, and noise.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    K. A. Smith, “Neural Networks for Combinatorial Optimisation: a review of more than a decade of research”, INFORMS Journal on Computing, vol. 11, no. 1, pp. 15–34, 1999.MATHMathSciNetCrossRefGoogle Scholar
  2. [2]
    J. J. Hopfield and D. W. Tank, “ ‘Neural’ Computation of Decisions in Optimization Problems', Biological Cybernetics, vol. 52, pp. 141–152, 1985.MATHMathSciNetGoogle Scholar
  3. [3]
    R. Durbin and D. Willshaw, “An analogue approach to the travelling salesman problem using an elastic net method”, Nature, vol. 326, pp. 689–691, 1987.CrossRefGoogle Scholar
  4. [4]
    K. Smith, M. Palaniswami and M. Krishnamoorthy, “Neural Techniques for Combinatorial Optimisation with Applications”, IEEE Transactions on Neural Networks, vol. 9, no. 6, pp. 1301–1318, 1998.CrossRefGoogle Scholar
  5. [5]
    K. A. De Jong, “An analysis of behavior of a class of genetic adaptive systems”, Doctoral Dissertation, University of Michigan, Dissertation Abstracts International, vol. 36, no. 10, 5140B, 1975.Google Scholar
  6. [6]
    K. Hornik, M. Stinchcombe and H. White, “Multilayer feedforward networks are universal approximators”, Neural Networks, vol. 2, pp. 359–366, 1989.CrossRefGoogle Scholar
  7. [7]
    R. Barron, “Universal approximation bounds for superpositions of a sigmoidal function”, IEEE Transactions on Information Theory, vol. 39, pp. 359–366, 1989.MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Kate A. Smith
    • 1
  • Jatinder N. D. Gupta
    • 2
  1. 1.School of Business SystemsMonash University, ClaytonVictoriaAustralia
  2. 2.Department of ManagementBall State University, MuncieIndianaUSA

Personalised recommendations