The neuro-dynamic scheme for solving general form of discrete time optimal control problems

  • Alireza Nazemi
  • Samira Sukhtsaraie
  • Marzieh Mortezaee


In this paper, we show that recently developed neural network methods for quadratic programming can be put to use in solving discrete time optimal control problems, with general pointwise constraints on states and controls. We describe a high performance recurrent neural network for a discrete time linear quadratic regulator problem with mixed state–control constraints. The equilibrium point of the proposed model is proved to be equivalent to the optimal solution of the discrete time problem. It is also shown that the proposed network model is stable in the Lyapunov sense and it is globally convergent to an exact optimal solution of the original problem. Several practical examples are provided to show the feasibility and the efficiency of the scheme.


Discrete time optimal control Neural network Convex quadratic programming Convergent Stability 


  1. 1.
    Toan NT, Thuy LQ Second-order necessary optimality conditions for a discrete optimal control problem with mixed constraints, Journal of Global Optimization,
  2. 2.
    Toan NT, Ansari QH, Yao J-C Second-Order Necessary Optimality Conditions for a Discrete Optimal Control Problem, Journal of Optimization Theory and Applications, DOI
  3. 3.
    Marchand PA, Lawrencet PD, Cherchas DB (1989) A discrete time optimal control law for a robot arm. Opt Control Appl Methods 10:1–20MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Leyendecker S, Ober-Blobaum S, Marsden JE, Ortiz M (2010) Discrete mechanics and optimal control for constrained systems. Opt Control Appl Methods 31:505–528MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Sandblom C-L, Eiselt HA, Jornasten K (1987) Discrete time optimal contriol of an economic system using differtent objective functions. Opt Control Appl Methods 8:253–269CrossRefGoogle Scholar
  6. 6.
    Tan F, Luo B, Guan X (2015) Finite-horizon 𝜖-optimal tracking control of discrete-time linear systems using iterative approximate dynamic programming. Asian J Control 17(1):176–189MathSciNetCrossRefMATHGoogle Scholar
  7. 7.
    Al-Tamimi A, Lewis FL, Abu-Khalaf M (2008) Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof. IEEE Trans Syst Man Cybern 38:943–949CrossRefGoogle Scholar
  8. 8.
    Bemporad A, Borrelli F, Morari M (2002) Model predictive control based on linear programming – the explicit solution. IEEE Trans Autom Control 47:1974–1985MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Bemporad A, Borrelli F, Morari M (2003) Min-max control of constrained uncertain discrete-time linear systems. IEEE Trans Autom Control 48:1600–1606MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Boltyanskii VG (1978) Optimal control of discrete systems. Wiley, New YorkMATHGoogle Scholar
  11. 11.
    Ioffe AD, Tikhomirov VM (1979) Theory of extremal problems. North-Holland, AmsterdamGoogle Scholar
  12. 12.
    Propoi AI (1973) Elements of the theory of optimal discrete processes. Moscow, Nauka. (in Russian)MATHGoogle Scholar
  13. 13.
    Kalman RE (1960) Contributions to the theory of optimal control. Bullet Soc Mex 5:102–119MathSciNetMATHGoogle Scholar
  14. 14.
    Chyung DH (1966) Discrete 1I IWM- optimal control system with essentially quadratic cost functionals. IEEE Trans Autom Control 11:404–413MathSciNetCrossRefGoogle Scholar
  15. 15.
    Deley GW, Franklin GF (1965) Optimal bounded control of linear sampled-data systems with quadratic loss. J Basic Eng 57:135–141CrossRefGoogle Scholar
  16. 16.
    Lee EB (1963) Recurrence equations and the control of their evolution. J Math Anal Appl 7:118–126MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Eaton JH (1963) An online solution to sampled-data time optimal control. J Electron Control 15:333–341CrossRefGoogle Scholar
  18. 18.
    Koepcke RW (1963) A solution to the sampled minimum-time problem. J Basic Eng 86:145–150CrossRefGoogle Scholar
  19. 19.
    Ogata K (1995) Discrete-time control systems, 2nd edn. Prentice-Hall, New JerseyGoogle Scholar
  20. 20.
    Pokoski J (1965) An analysis scheme for suboptimal minimum-time sampled-data systems. Joint Autom Control Conf. 15:270–257Google Scholar
  21. 21.
    Itoh U (1971) Optimal control of the discrete linear system with the bounded controller and the quadratic cost functional (in Japanese). J Inst Electr Eng Jpn 91:521–530Google Scholar
  22. 22.
    Halkin H (1964). In: Leondes CT (ed) Optimal control for systems described by difference equations, in Advances in Control Systems. Academic Press, New YorkGoogle Scholar
  23. 23.
    Butkovskii AG (1963) The necessary and sufficient conditions for optimality of discrete control systems. Autom Remote Control 24:963–970Google Scholar
  24. 24.
    Jordan BW, Polak E (1964) Theory of a class of discrete optimal systems. Theory Class Discret Opt Control Syst 17:697–711MathSciNetGoogle Scholar
  25. 25.
    Katz S, Kranc GM (1969) On the least time control problem with interior output constraints. IEEE Trans Autom Control 14:255–261MathSciNetCrossRefGoogle Scholar
  26. 26.
    Kranc GM, Shilman MB (1970) An application of functional analysis to time optimal control of linear discrete systems with output constraints. J Frankl Inst 290:137–147MathSciNetCrossRefMATHGoogle Scholar
  27. 27.
    Pantoja JFAD, Mayne DQ (1991) Sequential quadratic programming algorithm for discrete optimal control problems with control inequality constraints. Int J Control 53:823–836MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Wright SJ (1990) Solution of discrete-time optimal control problems on parallel computers. Parallel Comput 16:221–238MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Rockafellar RT, Wets RJ (1990) Generalized linear-quadratic problems of deterministic and stochastic optimal control in discrete time. SIAM J Control Optim 28:810–822MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Ohno K (1978) A new approach to differential dynamic programming for discrete- time systems. IEEE Trans Autom Control AC-23:37–47MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Sage AP, White III CC (1977) Prentice-Hall, New JerseyGoogle Scholar
  32. 32.
    Liu X, Li Y, Zhang W (2014) Stochastic linear quadratic optimal control with constraint for discrete-time systems. Appl Math Comput 228:264–270MathSciNetMATHGoogle Scholar
  33. 33.
    wright SJ (1993) Interior point methods for optimal control of discrete time systems. J Optim Theory Appl 77:161–187MathSciNetCrossRefMATHGoogle Scholar
  34. 34.
    Borrelli F, Baoti M, Bemporad A, Morari M (2005) Dynamic programming for constrained optimal control of discrete-time linear hybrid systems. Automatica 41:1709–1721MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Sontag ED (1981) Nonlinear regulation: The piecewise linear approach. IEEE Trans Autom Control 26:346–358MathSciNetCrossRefMATHGoogle Scholar
  36. 36.
    Mayne DQ (2001) Constrained optimal control. In: European control conference. Plenary lecture, PortoGoogle Scholar
  37. 37.
    Baotic M, Vasak M, Morari M, Peric N (2003) Hybrid theory based optimal control of electronic throttle. In: Proceeding American Control Conference, DenverGoogle Scholar
  38. 38.
    Branicky MS, Borkar VS, Mitter SK (1998) A unified framework for hybrid control: model and optimal control theory. IEEE Trans Autom Control 43:31–45MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Xu X, Antsaklis PJ (2003) Results and perspectives on computational methods for optimal control of switched systems. In: Maler O, Pnueli A (eds) Hybrid Systems: Computation and Control, HSCC 2003, volume 2623 ofLecture Notes in Computer Science. Springer Verlag, pp 540–556Google Scholar
  40. 40.
    Bemporad A, Borodani P, Mannelli M (2003) Hybrid control ofan automotive robotized gearbox for reduction ofconsumptions and emissions. In: Maler O, Pnueli A (eds) Hybrid systems: Computation and control, HSCC 2003, Lecture notes in computer science, vol 2623. Springer, Berlin, pp 81—96Google Scholar
  41. 41.
    Bemporad A, Giorgetti N, Kolmanovsky IV, Hrovat D (2002) Hybrid modeling and control ofa direct injection stratified charge engine. In: Symposium on advanced automotive technologies, ASME international mechanical engineering congress and exposition, New OrleansGoogle Scholar
  42. 42.
    Bemporad A, Morari M (1999) Control ofsystems integrating logic, dynamics, and constraints. Automatica 35(3):407–427MathSciNetCrossRefMATHGoogle Scholar
  43. 43.
    F Borrelli A, Bemporad M, Fodor D, Hrovat D (2001) A hybrid approach to traction control. In: Sangiovanni-Vincentelli A, Di Benedetto MD (eds) Hybrid systems: Computation and control, Lecture notes in computer science, vol 2034. Springer, Berlin, pp 162–174Google Scholar
  44. 44.
    Mignone D (2002) Control and estimation of hybrid systems via mathematical optimization. Dr. sc. tech. Thesis, Automatic Control Laboratory - ETH, Zurich. Google Scholar
  45. 45.
    Möbus R, Baotic M, Morari M (2003) Multi-object adaptive cruise control. In: Maler O, Pnueli A (eds) Hybrid systems: Computation and control, HSCC 2003, Lecture notes in computer science, vol 2623. Springer, Berlin, pp 359–374Google Scholar
  46. 46.
    Torrisi FD, Bemporad A (2004) HYSDELA tool for generating computational hybrid models. IEEE Trans Control Syst Technol 12(2):235–249CrossRefGoogle Scholar
  47. 47.
    Borrelli F, Baotic M, Bemporad A, Morari M (2003) Constrained optimal control of discrete-time linear hybrid systems. Technical Report AUT03-05, Automatic Control Laboratory. ETH Zurich, SwitzerlandGoogle Scholar
  48. 48.
    Tank DW, Hopfield JJ (1986) Simple neural optimization networks: An A/D converter, signal decision circuit, and a linear programming pircuit. IEEE Trans Circ Syst 33:533–541CrossRefGoogle Scholar
  49. 49.
    Effati S, Nazemi AR (2006) Neural network models and its application for solving linear and quadratic programming problems. Appl Math Comput 172:305–331MathSciNetMATHGoogle Scholar
  50. 50.
    Effati S, Ghomashi A, Nazemi AR (2007) Application of projection neural network in solving convex programming problems. Appl Math Comput 188:1103–1114MathSciNetMATHGoogle Scholar
  51. 51.
    Forti M, Nistri P, Quincampoix M (2006) Convergence of neural networks for programming problems via a nonsmooth Lojasiewicz inequality. IEEE Trans Neural Netw 17:1471–1486CrossRefGoogle Scholar
  52. 52.
    Gao XB, Liao L-Z, Qi LQ (2005) A novel neural network for variational inequalities with linear and nonlinear constraints. IEEE Trans Neural Netw 16:1305–1317CrossRefGoogle Scholar
  53. 53.
    Hu X (2009) Applications of the general projection neural network in solving extended linear-quadratic programming problems with linear constraints. Neurocomputing 72:1131–1137CrossRefGoogle Scholar
  54. 54.
    Hu X, Wang J (2007) Design of general projection neural networks for solving monotone linear variational inequalities and linear and quadratic optimization problems. IEEE Trans Syst Man Cybern Part B 37:1414–1421CrossRefGoogle Scholar
  55. 55.
    Liu QS, Wang J (2008) A one-layer recurrent neural network with a discontinuous hard-limiting activation function for quadratic programming. IEEE Trans Neural Netw 19:558–570CrossRefGoogle Scholar
  56. 56.
    Malek A, Hosseinipour-Mahani N, Ezazipour S (2010) Efficient recurrent neural network model for the solution of general nonlinear optimization problems. Optim Methods Softw 25:1–18MathSciNetCrossRefMATHGoogle Scholar
  57. 57.
    Nazemi AR (2012) A dynamic system model for solving convex nonlinear optimization problems. Commun Nonlinear Sci Numer Simul 17:1696–1705MathSciNetCrossRefMATHGoogle Scholar
  58. 58.
    Nazemi AR (2014) A neural network model for solving convex quadratic programming problems with some applications. Eng Appl Artif Intell 32:54–62CrossRefGoogle Scholar
  59. 59.
    Nazemi AR, Dehghan M (2015) A neural network method for solving support vector classification problems. Neurocomputing 152:369–376CrossRefGoogle Scholar
  60. 60.
    Wu H, Shi R, Qin L, Tao F, He L (2010) A nonlinear projection neural network for solving interval quadratic programming problems and its stability analysis. Math Probl Eng 2010:1–13MathSciNetMATHGoogle Scholar
  61. 61.
    Xia Y, Feng G (2005) An improved network for convex quadratic optimization with application to real-time beamforming. Neurocomputing 64:359–374CrossRefGoogle Scholar
  62. 62.
    Xue X, Bian W (2007) A project neural network for solving degenerate convex quadratic program. Neurocomputing 70:2449–2459CrossRefGoogle Scholar
  63. 63.
    Yang Y, Cao J (2008) A feedback neural network for solving convex constraint optimization problems. Appl Math Comput 201:340–350MathSciNetMATHGoogle Scholar
  64. 64.
    Monteiro RDC, Adler I (1989) Interior path-following primal–dual algorithms, Part 2: convex quadratic programming. Math Programm 44:43–66CrossRefMATHGoogle Scholar
  65. 65.
    Mangasarian OL (1969) Nonlinear programming. McGraw-Hill, New YorkMATHGoogle Scholar
  66. 66.
    Ferreira JAS, Vidal RVV (1984) Optimization of a pump-pipe system by dynamic programming. Eng Optim 7:241–251CrossRefGoogle Scholar
  67. 67.
    Ritch PS (1973) Discrete optimal control with multiple constraints I: constraint separation and transformation technique. Automatica 9:415–429MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Alireza Nazemi
    • 1
  • Samira Sukhtsaraie
    • 1
  • Marzieh Mortezaee
    • 1
  1. 1.Faculty of Mathematical ScienceShahrood University of TechnologyShahroodIran

Personalised recommendations