Robust Optimal Control of Nonlinear Systems with Matched Uncertainties

  • Ding WangEmail author
  • Chaoxu Mu
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 167)


In this chapter , we study the robust optimal control of nonlinear systems with matched uncertainties. In the first part, the infinite-horizon robust optimal control problem for a class of continuous-time uncertain nonlinear systems is investigated by using data-based adaptive critic designs. The neural network identification scheme is combined with the traditional adaptive critic technique, in order to design the nonlinear robust optimal control under uncertain environment. The robust optimal controller of the original uncertain system with a specified cost function is established by adding a feedback gain to the optimal controller of the nominal system. Then, a neural network identifier is employed to reconstruct the unknown dynamics of the nominal system with stability analysis. Hence, the data-based adaptive critic designs can be developed to solve the HJB equation corresponding to the transformed optimal control problem. The uniform ultimate boundedness of the closed-loop system is also proved by using the Lyapunov approach. In the second part, the robust optimal control design is revisited by using a data-based integral policy iteration approach. Here, the actor-critic technique based on neural networks and least squares implementation method are employed to facilitate deriving the optimal control law iteratively, so that the closed-form expression of the robust optimal controller is available. Four simulation examples with application backgrounds are also presented to illustrate the effectiveness of the established robust optimal control scheme. In summary, it is important to note that the results developed in this chapter broaden the application scope of ADP-based optimal control approach to more general nonlinear systems possessing dynamical uncertainties.


  1. 1.
    Abu-Khalaf, M., Lewis, F.L.: Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach. Automatica 41(5), 779–791 (2005)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Al-Tamimi, A., Lewis, F.L., Abu-Khalaf, M.: Discrete-time nonlinear HJB solution using approximate dynamic programming: convergence proof. IEEE Trans. Syst. Man Cybern.-Part B: Cybern. 38(4), 943–949 (2008)CrossRefGoogle Scholar
  3. 3.
    Beard, R.W., Saridis, G.N., Wen, J.T.: Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation. Automatica 33(12), 2159–2177 (1997)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bertsekas, D.P., Homer, M.L., Logan, D.A., Patek, S.D., Sandell, N.R.: Missile defense and interceptor allocation by neuro-dynamic programming. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 30(1), 42–51 (2000)CrossRefGoogle Scholar
  5. 5.
    Bhasin, S., Kamalapurkar, R., Johnson, M., Vamvoudakis, K.G., Lewis, F.L., Dixon, W.E.: A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica 49(1), 82–92 (2013)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Bian, T., Jiang, Y., Jiang, Z.P.: Adaptive dynamic programming and optimal control of nonlinear nonaffine systems. Automatica 50(10), 2624–2632 (2014)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Dierks, T., Jagannathan, S.: Online optimal control of affine nonlinear discrete-time systems with unknown internal dynamics by using time-based policy update. IEEE Trans. Neural Netw. Learn. Syst. 23(7), 1118–1129 (2012)CrossRefGoogle Scholar
  8. 8.
    Fu, J., He, H., Zhou, X.: Adaptive learning and control for MIMO system based on adaptive dynamic programming. IEEE Trans. Neural Netw. 22(7), 1133–1148 (2011)CrossRefGoogle Scholar
  9. 9.
    Gao, H., Meng, X., Chen, T.: A new design of robust \(H_{2}\) filters for uncertain systems. Syst. Control Lett. 57(7), 585–593 (2008)CrossRefGoogle Scholar
  10. 10.
    Guo, G., Wang, Y., Hill, D.J.: Nonlinear output stabilization control for multimachine power systems. IEEE Trans. Circuits Syst.-I, Fund. Theory Appl. 47(1), 46–53 (2000)CrossRefGoogle Scholar
  11. 11.
    Heydari, A., Balakrishnan, S.N.: Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Trans. Neural Netw. Learn. Syst. 24(1), 145–157 (2013)CrossRefGoogle Scholar
  12. 12.
    Hussain, S., Xie, S.Q., Jamwal, P.K.: Robust nonlinear control of an intrinsically compliant robotic gait tranning orthosis. IEEE Trans. Syst. Man Cybern.: Syst. 43(3), 655–665 (2013)CrossRefGoogle Scholar
  13. 13.
    Jiang, Y., Jiang, Z.P.: Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics. Automatica 48(10), 2699–2704 (2012)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Jiang, Y., Jiang, Z.P.: Robust adaptive dynamic programming for large-scale systems with an application to multimachine power systems. New York University, Brooklyn, NY, Technical Report. (2012)
  15. 15.
    Jiang, Y., Jiang, Z.P.: Robust adaptive dynamic programming for large-scale systems with an application to multimachine power systems. IEEE Trans. Circuits Syst.-II: Express Briefs 59(10), 693–697 (2012)CrossRefGoogle Scholar
  16. 16.
    Jiang, Y., Jiang, Z.P.: Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE Trans. Neural Netw. Learn. Syst. 25(5), 882–893 (2014)CrossRefGoogle Scholar
  17. 17.
    Jiang, Z.P., Jiang, Y.: Robust adaptive dynamic programming for linear and nonlinear systems: an overview. Eur. J. Control 19(5), 417–425 (2013)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Kundur, P., Balu, N.J., Lauby, M.G.: Power System Stability and Control. McGraw-Hill, New York (1994)Google Scholar
  19. 19.
    Lee, J.Y., Park, J.B., Choi, Y.H.: Integral Q-learning and explorized policy iteration for adaptive optimal control of continuous-time linear systems. Automatica 48(11), 2850–2859 (2012)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Lee, J.Y., Park, J.B., Choi, Y.H.: Integral reinforcement learning for continuous-time input-affine nonlinear systems with simultaneous invariant explorations. IEEE Trans. Neural Netw. Learn. Syst. 26(5), 916–932 (2015)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Lewis, F.L., Liu, D.: Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Wiley, New Jersey (2013)Google Scholar
  22. 22.
    Lewis, F.L., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits Syst. Mag. 9(3), 32–50 (2009)CrossRefGoogle Scholar
  23. 23.
    Lewis, F.L., Vrabie, D., Syrmos, V.: Optimal Control. Wiley, Hoboken, New Jersey (2012)CrossRefGoogle Scholar
  24. 24.
    Liang, J., Venayagamoorthy, G.K., Harley, R.G.: Wide-area measurement based dynamic stochastic optimal power flow control for smart grids with high variability and uncertainty. IEEE Trans. Smart Grid 3(1), 59–69 (2012)CrossRefGoogle Scholar
  25. 25.
    Lin, F., Brandt, R.D.: An optimal control approach to robust control of robot manipulators. IEEE Trans. Robot. Autom. 14(1), 69–77 (1998)CrossRefGoogle Scholar
  26. 26.
    Lin, F., Brand, R.D., Sun, J.: Robust control of nonlinear systems: Compensating for uncertainty. Int. J. Control 56(6), 1453–1459 (1992)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Liu, D., Li, C., Li, H., Wang, D., Ma, H.: Neural-network-based decentralized control of continuous-time nonlinear interconnected systems with unknown dynamics. Neurocomputing 165, 90–98 (2015)CrossRefGoogle Scholar
  28. 28.
    Liu, D., Li, H., Wang, D.: Online synchronous approximate optimal learning algorithm for multiplayer nonzero-sum games with unknown dynamics. IEEE Trans. Syst. Man. Cybern.: Syst. 44(8), 1015–1027 (2014)CrossRefGoogle Scholar
  29. 29.
    Liu, D., Li, H., Wang, D.: Error bounds of adaptive dynamic programming algorithms for solving undiscounted optimal control problems. IEEE Trans. Neural Netw. Learn. Syst. 26(6), 1323–1334 (2015)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Liu, D., Wang, D., Li, H.: Decentralized stabilization for a class of continuous-time nonlinear interconnected systems using online learning optimal control approach. IEEE Trans. Neural Netw. Learn. Syst. 25(2), 418–428 (2014)CrossRefGoogle Scholar
  31. 31.
    Liu, D., Wang, D., Wang, F.Y., Li, H., Yang, X.: Neural-network-based online HJB solution for optimal robust guaranteed cost control of continuous-time uncertain nonlinear systems. IEEE Trans. Cybern. 44(12), 2834–2847 (2014)CrossRefGoogle Scholar
  32. 32.
    Liu, D., Wei, Q.: Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems. IEEE Trans. Cybern. 43(2), 779–789 (2013)CrossRefGoogle Scholar
  33. 33.
    Liu, D., Yang, X., Wang, D., Wei, Q.: Reinforcement-learning-based robust controller design for continuous-time uncertain nonlinear systems subject to input constraints. IEEE Trans. Cybern. 45(7), 1372–1385 (2015)CrossRefGoogle Scholar
  34. 34.
    Luo, B., Wu, H.N., Huang, T., Liu, D.: Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design. Automatica 50(12), 3281–3290 (2014)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Luo, B., Wu, H.N., Li, H.X.: Adaptive optimal control of highly dissipative nonlinear spatially distributed processes with neuro-dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 26(4), 684–696 (2015)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Luo, Y., Sun, Q., Zhang, H., Cui, L.: Adaptive critic design-based robust neural network control for nonlinear distributed parameter systems with unknown dynamics. Neurocomputing 148, 200–208 (2015)CrossRefGoogle Scholar
  37. 37.
    Modares, H., Lewis, F.L., Naghibi-Sistani, M.B.: Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Trans. Neural Netw. Learn. Syst. 24(10), 1513–1525 (2013)CrossRefGoogle Scholar
  38. 38.
    Modares, H., Naghibi-Sistani, M.B., Lewis, F.L.: A policy iteration approach to online optimal control of continuous-time constrained-input systems. ISA Trans. 52(5), 611–621 (2013)CrossRefGoogle Scholar
  39. 39.
    Mu, C., Ni, Z., Sun, C., He, H.: Air-breathing hypersonic vehicle tracking control based on adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 28(3), 584–598 (2017)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Na, J., Herrmann, G.: Online adaptive approximate optimal tracking control with simplified dual approximation structure for continuoustime unknown nonlinear systems. IEEE/CAA J. Autom. Sinica 1(4), 412–422 (2014)CrossRefGoogle Scholar
  41. 41.
    Naidu, D.S.: Optimal Control Systems. CRC Press, Florida (2003)Google Scholar
  42. 42.
    Ni, Z., He, H., Wen, J.: Adaptive learning in tracking control based on the dual critic network design. IEEE Trans. Neural Netw. Learn. Syst. 24(6), 913–928 (2013)CrossRefGoogle Scholar
  43. 43.
    Ni, Z., He, H., Zhong, X., Prokhorov, D.V.: Model-free dual heuristic dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 26(8), 1834–1839 (2015)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Nodland, D., Zargarzadeh, H., Jagannathan, S.: Neural network-based optimal adaptive output feedback control of a helicopter UAV. IEEE Trans. Neural Netw. Learn. Syst. 24(7), 1061–1073 (2013)CrossRefGoogle Scholar
  45. 45.
    Palanisamy, M., Modares, H., Lewis, F.L., Aurangzeb, M.: Continuous-time Q-learning for infinite-horizon discounted cost linear quadratic regulator problems. IEEE Trans. Cybern. 45(2), 165–176 (2015)CrossRefGoogle Scholar
  46. 46.
    Prokhorov, D.V., Wunsch, D.C.: Adaptive critic designs. IEEE Trans. Neural Netw. 8(5), 997–1007 (1997)CrossRefGoogle Scholar
  47. 47.
    Saridis, G.N., Lee, C.G.: An approximation theory of optimal control for trainable manipulators. IEEE Trans. Syst. Man Cybern.-Part B: Cybern. 9(3), 152–159 (1979)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Song, R., Xiao, W., Sun, C.: A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture. Sci. China: Inf. Sci. 57(6), 1–10 (2014)zbMATHGoogle Scholar
  49. 49.
    Vamvoudakis, K.G., Lewis, F.L.: Online actor-critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5), 878–888 (2010)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Vrabie, D., Lewis, F.L.: Neural network approach to continuous-time direct adaptive optimal control for partially unknown nonlinear systems. Neural Netw. 22(3), 237–246 (2009)CrossRefGoogle Scholar
  51. 51.
    Wang, D., Li, C., Liu, D., Mu, C.: Data-based robust optimal control of continuous-time affine nonlinear systems with matched uncertainties. Inf. Sci. 366, 121–133 (2016)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Wang, D., Liu, D.: Neuro-optimal control for a class of unknown nonlinear dynamic systems using SN-DHP technique. Neurocomputing 121, 218–225 (2013)CrossRefGoogle Scholar
  53. 53.
    Wang, D., Liu, D., Li, H.: Policy iteration algorithm for online design of robust control for a class of continuous-time nonlinear systems. IEEE Trans. Autom. Sci. Eng. 11(2), 627–632 (2014)CrossRefGoogle Scholar
  54. 54.
    Wang, D., Liu, D., Li, H., Ma, H.: Neural-network-based robust optimal control design for a class of uncertain nonlinear systems via adaptive dynamic programming. Inf. Sci. 282, 167–179 (2014)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Wang, D., Liu, D., Li, H., Ma, H., Li, C.: A neural-network-based online optimal control approach for nonlinear robust decentralized stabilization. Soft. Comput. 20(2), 707–716 (2016)CrossRefGoogle Scholar
  56. 56.
    Wang, D., Liu, D., Wei, Q., Zhao, D., Jin, N.: Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming. Automatica 48(8), 1825–1832 (2012)MathSciNetCrossRefGoogle Scholar
  57. 57.
    Wang, D., Liu, D., Zhang, Q., Zhao, D.: Data-based adaptive critic designs for nonlinear robust optimal control with uncertain dynamics. IEEE Trans. Syst. Man Cybern.: Syst. 46(11), 1544–1555 (2016)CrossRefGoogle Scholar
  58. 58.
    Wang, D., Liu, D., Zhao, D., Huang, Y., Zhang, D.: A neural-network-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints. Neural Comput. Appl. 22(2), 219–227 (2013)CrossRefGoogle Scholar
  59. 59.
    Wang, Z., Chan, F.T.S.: A robust replenishment and production control policy for a single-stage production/inventory system with inventory inaccuracy. IEEE Trans. Syst. Man Cybern.: Syst. 45(2), 326–337 (2015)CrossRefGoogle Scholar
  60. 60.
    Werbos, P.J.: Approximate dynamic programming for real-time control and neural modeling. Neural, Fuzzy, and Adaptive Approaches, Handbook of Intelligent Control, pp. 493–526 (1992)Google Scholar
  61. 61.
    Xu, X., Hou, Z., Lian, C., He, H.: Online learning control using adaptive critic designs with sparse kernel machines. IEEE Trans. Neural Netw. Learn. Syst. 24(5), 762–775 (2013)CrossRefGoogle Scholar
  62. 62.
    Yang, X., Liu, D., Huang, Y.: Neural-network-based online optimal control for uncertain nonlinear continuous-time systems with control constraints. IET Control Theory Appl. 7(17), 2037–2047 (2013)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Yang, X., Liu, D., Wang, D.: Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints. Int. J. Control 87(3), 553–566 (2014)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Zhang, H., Cui, L., Zhang, X., Luo, Y.: Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Trans. Neural Netw. 22(12), 2226–2236 (2011)CrossRefGoogle Scholar
  65. 65.
    Zhang, H., Liu, D., Luo, Y., Wang, D.: Adaptive Dynamic Programming for Control: Algorithms and Stability. Springer, London (2013)CrossRefGoogle Scholar
  66. 66.
    Zhong, X., He, H., Zhang, H., Wang, Z.: Optimal control for unknown discrete-time nonlinear markov jump systems using adaptive dynamic programming. IEEE Trans. Neural Netw. Learn. Syst. 25(12), 2141–2155 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.The State Key Laboratory of Management and Control for Complex SystemsInstitute of Automation, Chinese Academy of SciencesBeijingChina
  2. 2.School of Electrical and Information EngineeringTianjin UniversityTianjinChina

Personalised recommendations