Find out how to access previewonly content
Neurodynamical Optimization
 LiZhi Liao,
 Houduo Qi,
 Liqun Qi
 … show all 3 hide
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Get AccessAbstract
Dynamical (or ode) system and neural network approaches for optimization have been coexisted for two decades. The main feature of the two approaches is that a continuous path starting from the initial point can be generated and eventually the path will converge to the solution. This feature is quite different from conventional optimization methods where a sequence of points, or a discrete path, is generated. Even dynamical system and neural network approaches share many common features and structures, yet a complete comparison for the two approaches has not been available. In this paper, based on a detailed study on the two approaches, a new approach, termed neurodynamical approach, is introduced. The new neurodynamical approach combines the attractive features in both dynamical (or ode) system and neural network approaches. In addition, the new approach suggests a systematic procedure and framework on how to construct a neurodynamical system for both unconstrained and constrained problems. In analyzing the stability issues of the underlying dynamical (or ode) system, the neurodynamical approach adopts a new strategy, which avoids the Lyapunov function. Under the framework of this neurodynamical approach, strong theoretical results as well as promising numerical results are obtained.
References
1.
AluffiPentini, F., Parisi, V. and Zirilli, F. (1984), A differentialequations algorithm for nonlinear equations, ACM Trans. on Math. Software 10(3), 299–316.
2.
AluffiPentini, F., Parisi, V. and Zirilli, F. (1984) Algorithm 617 DAFNE: A differentialequations algorithm for nonlinear equations, ACM Trans. on Math. Software, 10(3), 317–324.
3.
Anstreicher, K. M. (1988), Linear programming and the Newton barrier flow, Math. Prog. 41, 367–373.
4.
Arrow, K. J., Hurwicz, L. and Uzawa, H. (1958), Studies in Linear and Nonlinear Programming, Stanford University Press, Stanford, CA.
5.
Barron, A. R. (1993), Universal approximation bounds for superpositions of a sigmoidal function, IEEE Trans. Inform. Theory 39(3), 930–945.
6.
Boggs, P. T. (1971), The solution of nonlinear systems of equations by Astable integration techniques, SIAM J. Numer. Anal. 8(4), 767–785.
7.
Botsaris, C. A. and Jacobson, D. H. (1976), A Newtontype curvilinear search method for optimization, JMAA, 54, 217–229.
8.
Botsaris, C. A. (1978), Differential gradient methods, JMAA 63, 177–198.
9.
Botsaris, C. A. (1978), A curvilinear optimization method based upon iterative estimation of the eigensystem of the Hessian matrix, JMAA 63, 396–411.
10.
Botsaris, C. A. (1978), A class of methods for unconstrained minimization based on stable numerical integration techniques, JMAA 63, 729–749.
11.
Bouzerdoum, A. and Pattison, T. R. (1993), Neural network for quadratic optimization with bound constrains, IEEE Trans. Neural Networks 4, 293–304.
12.
Branin, Jr. F. H., (1972), Widely convergent method for finding multiple solutions of simultaneous nonlinear equations, IBM Journal of Research and Development 16, 504–522.
13.
Brown, A. A. and BartholomewBiggs, M. C. (1989), Some effective methods for unconstrained optimization based on the solution of systems of ordinary differential equations, JOTA 62(2), 211–224.
14.
Brown, A. A. and BartholomewBiggs, M. C. (1989), ODE versus SQP methods for constrained optimization, JOTA 62(3), 371–386.
15.
Chen, Y. H. and Fang, S. C. (1998), Solving convex programming problem with equality constraints by neural networks, Computers Math. Appl. 36, 41–68.
16.
Chu, M. T. (1988), On the continuous realization of iterative processes, SIAM Review 30(3), 375–387.
17.
Cichocki, A. and Unbehauen, R. (1993), Neural Networks for Optimization and Signal Processing. Wiley, Chichester.
18.
Cichocki, A., Unbehauen, R., Weinzierl, K. and Holzel, R., (1996), A new neural network for solving linear programming problems, European J. Operational Res., 93, 244–256.
19.
Chua, L. O. and Lin, G. N. (1984), Nonlinear programming without computation, IEEE Trans. Circuits Syst., 31, 182–188.
20.
Cybenko, G. (1989), Approximation by superpositions of a sigmoidal function, Math. Control Signals Systems, 2, 303–314.
21.
Evtushenko, Yu. G. and Zhadan, V. G., (1978), A relaxation method for solving problems of nonlinear programming, U.S.S.R. Comput. Math. Math. Phys. 17(4), 73–87.
22.
Diener, I. and Schaback, R., (1990), An extended continuous Newton method, JOTA 67(1), 57–77.
23.
Glazos, M. P., Hui, S. and Żak, S., (1998), Sliding modes in solving convex programming problems, SIAM J. Control Optim. 36, 680–697.
24.
Goldstein, A. A. (1964), Convex programming in Hilbert space, Bulletin of American Mathematical Society 70, 709–710.
25.
Han, Q., Liao, L.Z., Qi, H. and Qi, L., (2001), Stability analysis of gradientbased neural networks for optimization problems, J. Global Optim. 19(4), 363–381.
26.
Hassan, N. and Rzymowski, W., (1990), An ordinary differential equation in nonlinear programming, Nonlinear Analysis, Theory, Method & Applications 15(7), 597–599.
27.
Haykin, S. S., (1994), Neural Networks: A Comprehensive Foundation, PrenticeHall, Englewood Cliffs, NJ.
28.
He, B. S., (1994), Solving a class of linear projection equations, Numerische Mathematik 68, 71–80.
29.
He, B. S., (1997), A class of projection and contraction methods for monotone variational inequalities, Applied Mathematics and Optimization 35, 69–76.
30.
He, B. S., (1999), Inexact implicit methods for monotone general variational inequalities, Mathematical Programming, 86(1), 199–217.
31.
He, B. S. and Yang H., (2000) A neural network model for monotone linear asymmetric variational inequalities, IEEE Trans. Neural Networks, 11, 3–16.
32.
Hopfield, J. J., (1982), Neural networks and physical systems with emergent collective computational ability, Proc. Natl. Acad. Sci. USA, 79, 2554–2558.
33.
Hopfield, J. J., (1984), Neurons with graded response have collective computational properties like those of twostate neurons, Proc. Natl. Acad. Sci., 81, 3088–3092.
34.
Hopfield, J. J. and Tank, D. W., (1985), Neural computation of decisions in optimization problems, Biolog. Cybernetics, 52, 141–152.
35.
Hornik, K., (1991), Approximation capabilities of multilayer feedforward networks, Neural Networks, 4, 251–257.
36.
Hornik, K., Stinchcombe, M. and White, H., (1989), Multilayer feedforward networks are universal approximators, Neural Networks, 2, 359–366.
37.
Hou, Z.G., Wu, C.P. and Bao, P., (1998), A neural network for hierarchical optimization of nonlinear largescale systems, International Journal of Systems Science 29(2), 159–166.
38.
Incerti, S., Parisi, V. and Zirilli, F., (1979), A new method for solving nonlinear simultaneous equations, SIAM J. Numer. Anal. 16, 779–789.
39.
Kennedy, M. P. and Chua, L. O., (1988), Neural networks for nonlinear programming, IEEE Trans. Circuits Syst. 35, 554–562.
40.
Liao, L.Z. and Qi, H., (1999), A neural network for the linear complementarity problem, Math. Comput. Modelling 29(3), 9–18.
41.
Liao, L.Z., Qi, H. and Qi, L., (2001), Solving nonlinear complementarity problems with neural networks: a reformulation method approach, JCAM 131(12), 343–359.
42.
Lillo, W. E., Loh, M. H., Hui S. and Zak, S. H., (1993), On solving constrained optimization problems with neural networks: a penalty method approach, IEEE Trans. Neural Networks, 4, 931–940.
43.
Maa, C. Y. and Shanblatt, M. A., (1992), Linear and quadratic programming neural network analysis, IEEE Trans. Neural Networks 3, 580–594.
44.
Maa, C. Y. and Shanblatt, M. A., (1992), A twophase optimization neural network, IEEE Trans. Neural Networks 3, 1003–1009.
45.
Mangasarian, O. L., (1993), Mathematical programming in neural networks, ORSA J. Comput. 5(4), 349–360.
46.
Moré, J. J., Garbow, B. S. and Hillstrom, K. E., (1981), Testing unconstrained optimization software, ACM Trans. Math. Software 7(1), 17–41.
47.
Novaković, Z. R., (1990), Solving systems of nonlinear equations using the Lyapunov direct method, Computers Math. Applic. 20(12), 19–23.
48.
Pan, P.Q., (1992), New ODE methods for equality constrained optimization (1) — equations, JCM 10(1), 77–92.
49.
Pan, P.Q., (1992), New ODE methods for equality constrained optimization (2) — algorithms, JCM 10(2), 129–146.
50.
Polyak, B. T., (1966), Constrained minimization problems, USSR Computational Mathematics and Mathematical Physics 6, 1–50.
51.
RodríguezVázquez, A., DomínguezCastro, R., Rueda, A., Huertas J. L. and SánchezSinencio, E., (1990), Nonlinear switchcapacitor ‘neural’ networks for optimization problems, IEEE Trans. Circuits Syst. 37, 384–398.
52.
Schäffler, S. and Warsitz, H., (1990), A trajectoryfollowing method for unconstrained optimization, JOTA 67(1), 133–140.
53.
Schnabel, R. B. and Eskow, E., (1990), A new modified Cholesky factorization, SIAM J. Sci. Stat. Comput. 11, 1136–1158.
54.
Slotine, J.J. E. and Li, W., (1991), Applied Nonlinear Control, PrenticeHall, Englewood Cliffs, NJ.
55.
Snyman, J. A., (1982), A new and dynamic method for unconstrained minimization, Appl. Math. Modelling 6, 449–462.
56.
Solodov, M. V. and Tseng, P., (1996), Modified projectiontype methods for monotone variational inequalities, SIAM J. Control and Optimization 34, 1814–1830.
57.
Sudharsanan, S. and Sundareshan, M., (1991), Exponential stability and a systematic synthesis of a neural network for quadratic minimization, Neural Networks 4, 599–613.
58.
Tanabe, K., (1980), A geometric method in nonlinear programming, JOTA 30(2), 181–210.
59.
Tank, D. W. and Hopfield, J. J., (1986), Simple neural optimization networks: An A/D convert, signal decision circuit, and a linear programming circuit, IEEE Trans. Circuits Syst. 33, 533–541.
60.
Teo, K. L., Wong, K. H. and Yan, W. Y., (1995), Gradientflow approach for computing a nonlinearquadratic optimaloutput feedback gain matrix, JOTA 85, 75–96.
61.
Vincent, T. L., Goh, B. S. and Teo, K. L., (1992), Trajectoryfollowing algorithms for minmax optimization problems, JOTA 75, 501–519.
62.
Wilde, N. G., (1969), A note on a differential equation approach to nonlinear programming, Management Science 15(11), 739–739.
63.
Williems, J. L., (1970), Stability Theory of Dynamical Systems, Nelson.
64.
Wu, X., Xia, Y., Li, J. and Chen, W. K., (1996), A high performance neural network for solving linear and quadratic programming problems, IEEE Trans. Neural Networks, 7, 643–651.
65.
Xia, Y., (1996), A new neural network for solving linear programming problems and its applications, IEEE Trans. Neural Networks 7, 525–529.
66.
Xia, Y., (1996) A new neural network for solving linear and quadratic programming problems, IEEE Trans. Neural Networks 7, 1544–1547.
67.
Xia, Y. and Wang, J., (1998), A general methodology for designing globally convergent optimization neural networks, IEEE Trans. Neural Networks 9, 1331–1343.
68.
Yamashita, H., (1980), A differential equation approach to nonlinear programming, Math. Prog. 18, 155–168.
69.
Zabczyk, J., (1992), Mathematical Control Theory: An Introduction, Birkhauser, Boston.
70.
Żak, S. H., Upatising, V. and Hui, S., (1995) Solving linear programming problems with neural networks: a comparative study, IEEE Trans. Neural Networks 6, 94–104.
71.
Żak, S. H., Upatising, V., Lillo, W. E. and Hui, S., (1994), A dynamical systems approach to solving linear programming problems. In: K. D. Elworthy, W. N. Everitt and E. B. Lee (eds.), Differential Equations, Dynamical Systems, and Control Science, Marcel Dekker, New York.
72.
Zhang, S. and Constantinides, A. G., (1992), Lagrange programming neural network, IEEE Trans. Circuits Syst. 39, 441–452.
73.
Zhang, X.S., (2000), Neural Network in Optimization, Kluwer Academic Publishers, Dordrecht.
74.
Zhou, Z. and Shi, Y., (1997), An ODE method of solving nonlinear programming, Computers Math. Applic. 34(1), 97–102.
75.
Zhou, Z. and Shi, Y., (1998), A convergence of ODE method in constrained optimization, JMAA 218(1), 297–307.
76.
Zirilli, F., (1982), The use of ordinary differential equations in the solution of nonlinear systems of equations, Powell, M. J. D., (ed.) Nonlinear Optimization 1981, Academic Press, London.
 Title
 Neurodynamical Optimization
 Journal

Journal of Global Optimization
Volume 28, Issue 2 , pp 175195
 Cover Date
 20040201
 DOI
 10.1023/B:JOGO.0000015310.27011.02
 Print ISSN
 09255001
 Online ISSN
 15732916
 Publisher
 Kluwer Academic Publishers
 Additional Links
 Topics
 Keywords

 Dynamical system
 Neural network
 Neurodynamical
 Ode System
 Optimization
 Industry Sectors
 Authors

 LiZhi Liao ^{(1)}
 Houduo Qi ^{(2)}
 Liqun Qi ^{(2)}
 Author Affiliations

 1. Department of Mathematics, Hong Kong Baptist University, Kowloon Tong, Kowloon, Hong Kong; Email
 2. Department of Applied Mathematics, The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong; Email