Dynamic Programming Viscosity Solution Approach and Its Applications to Optimal Control Problems

  • Bing SunEmail author
  • Zhen-Zhen Tao
  • Yang-Yang Wang
Part of the Studies in Systems, Decision and Control book series (SSDC, volume 200)


This chapter is concerned with optimal control problems of dynamical systems described by partial differential equations (PDEs). Firstly, using the Dubovitskii-Milyutin approach, we obtain the necessary condition of optimality, i.e., the Pontryagin maximum principle for optimal control problem of an age-structured population dynamics for spread of universally fatal diseases. Secondly, for an optimal birth control problem of a McKendrick type age-structured population dynamics, we establish the optimal feedback control laws by the dynamic programming viscosity solution (DPVS) approach. Finally, for a well-adapted upwind finite-difference numerical scheme for the HJB equation arising in optimal control, we prove its convergence and show that the solution from this finite-difference scheme converges to the value function of the associated optimal control problem.


Optimal feedback control Viscosity solution Dynamic programming approach Numerical solution Convergence 


  1. 1.
    Sussmann, H.J., Willems, J.C.: 300 years of optimal control: from the brachystochrone to the maximum principle. IEEE Control Syst. 17, 32–44 (1997)CrossRefGoogle Scholar
  2. 2.
    Ho, Y.C.: On centralized optimal control. IEEE Trans. Autom. Control 60, 537–538 (2005)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Fattorini, H.O.: Infinite-Dimensional Optimization and Control Theory. Encyclopedia of Mathematics and Its Applications. Cambridge University Press, Cambridge (1999)Google Scholar
  4. 4.
    Guo, B.Z., Chai, S.G.: Control Theory of Infinite-Dimensional Systems. Science Press, Beijing (2012). (in Chinese)Google Scholar
  5. 5.
    Rubio, J.E.: Control and Optimization: The Linear Treatment of Nonlinear Problems. Manchester University Press, Manchester (1986)zbMATHGoogle Scholar
  6. 6.
    Bryson Jr., A.E.: Optimal Control—1950 to 1985. IEEE Control Syst. 13, 26–33 (1996)CrossRefGoogle Scholar
  7. 7.
    von Stryk, O., Bulirsch, R.: Direct and indirect methods for trajectory optimization. Ann. Oper. Res. 37, 357–373 (1992)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Teo, K.L., Goh, C.J., Wong, K.H.: A Unified Computational Approach to Optimal Control Problems. Longman Scientific & Technical, New York (1991)zbMATHGoogle Scholar
  9. 9.
    Rehbock, V., Wang, S., Teo, K.L.: Computing optimal control with hyperbolic partial differential equation. J. Aust. Math. Soc. Ser. B 40, 266–287 (1998)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Teo, K.L., Jennings, L.S., Lee, H.W.J., Rehbock, V.: The control parametrization enhancing transform for constrained optimal control problems. J. Aust. Math. Soc. Ser. B 40, 314–335 (1999)CrossRefGoogle Scholar
  11. 11.
    Jennings, L.S., Teo, K.L., Goh, C.J.: MISER3.2—Optimal Control Software: Theory and User Manual. Department of Mathematics, The University of Western Australia, Australia (1997)Google Scholar
  12. 12.
    Rehbock, V., Teo, K.L., Jennings, L.S.: Suboptimal feedback control for a class of nonlinear systems using spline interpolation. Discrete Continuous Dyn. Syst. 1, 223–236 (1995)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Augustin, D., Maurer, H.: Computational sensitivity analysis for state constrained optimal control problems. Ann. Oper. Res. 101, 75–99 (2001)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Liu, Y., Ito, S., Lee, H.W.J., Teo, K.L.: Semi-infinite programming approach to continuously-constrained linear-quadratic optimal control problems. J. Optim. Theory Appl. 108, 617–632 (2001)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Malanowski, K., Maurer, H., Pickenhain, S.: Second-order sufficient conditions for state-constrained optimal control problems. J. Optim. Theory Appl. 123, 595–617 (2004)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Maurer, H., Oberle, H.J.: Second order sufficient conditions for optimal control problems with free final time: the Riccati approach. SIAM J. Control Optim. 41, 380–403 (2002)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Maurer, H., Osmolovskii, N.P.: Second order sufficient conditions for time-optimal bang-bang control. SIAM J. Control Optim. 42, 2239–2263 (2004)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Crandall, M.G.: Viscosity solutions: a primer. In: Capuzzo-Dolcetta, I., Lions, P.L. (eds.) Viscosity Solutions and Applications. Lecture Notes in Mathematics, vol. 1660, pp. 1–43. Springer, Berlin (1997)Google Scholar
  19. 19.
    Bardi, M., Capuzzo-Dolcetta, I.: Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations. Maurizio Falcone and Pierpaolo Soravia. Birkhäuser, Boston (1997)CrossRefGoogle Scholar
  20. 20.
    Barron, E.N.: Application of viscosity solutions of infinite-dimensional Hamilton-Jacobi-Bellman equations to some problems in distributed optimal control. J. Optim. Theory Appl. 64, 245–268 (1990)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Cannarsa, P., Gozzi, F., Soner, H.M.: A dynamic programming approach to nonlinear boundary control problems of parabolic type. J. Funct. Anal. 117, 25–61 (1993)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Gozzi, F.S., Sritharan, S., Świȩch, A.: Viscosity solutions of dynamic-programming equations for the optimal control of the two-dimensional Navier-Stokes equations. Arch. Ration. Mech. Anal. 163, 295–327 (2002)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Kocan, M., Soravia, P.: A viscosity approach to infinite-dimensional Hamilton-Jacobi equations arising in optimal control with state constraints. SIAM J. Control Optim. 36, 1348–1375 (1998)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Yong, J.M., Zhou, X.Y.: Stochastic Controls: Hamiltonian Systems and HJB Equations. In: Applications of Mathematics (New York), vol. 43. Springer, New York (1999)Google Scholar
  25. 25.
    Fleming, W.H., Soner, H.M.: Controlled Markov Processes and Viscosity Solutions. In: Applications of Mathematics, vol. 25. Springer, New York (1993)Google Scholar
  26. 26.
    Sun, B., Guo, B.Z.: Convergence of an upwind finite-difference scheme for Hamilton-Jacobi-Bellman equation in optimal control. IEEE Trans. Autom. Control. 60(11), 3012–3017 (2015)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Guo, B.Z., Sun, B.: Dynamic programming approach to the numerical solution of optimal control with paradigm by a mathematical model for drug therapies of HIV/AIDS. Optim. Eng. 15(1), 119–136 (2014)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Guo, B.Z., Sun, B.: A new algorithm for finding numerical solutions of optimal feedback control. IMA J. Math. Control Inf. 26(1), 95–104 (2009)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Guo, B.Z., Sun, B.: Numerical solution to the optimal feedback control of continuous casting process. J. Global Optim. 39(2), 171–195 (2007)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Guo, B.Z., Sun, B.: Numerical solution to the optimal birth feedback control of a population dynamics: viscosity solution approach. Optimal Control Appl. Methods 26(5), 229–254 (2005)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Huang, C.S., Wang, S., Teo, K.L.: On application of an alternating direction method to Hamilton-Jacobi-Bellman equations. J. Comput. Appl. Math. 166, 153–166 (2004)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Wang, S., Jennings, L.S., Teo, K.L.: Numerical solution of Hamilton-Jacobi-Bellman equations by an upwind finite volume method. J. Global Optim. 27, 177–192 (2003)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Wang, S., Gao, F., Teo, K.L.: An upwind finite-difference method for the approximation of viscosity solutions to Hamilton-Jacobi-Bellman equations. IMA J. Math. Control Inf. 17, 167–178 (2000)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Sun, B., Wu, M.X.: Optimal control of age-structured population dynamics for spread of universally fatal diseases. Appl. Anal. 92, 901–921 (2013)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Sun, B.: Optimal control of age-structured population dynamics for spread of universally fatal diseases II. Appl. Anal. 93, 1730–1744 (2014)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Lin, F., Muthuraman, K., Lawley, M.: An optimal control theory approach to non-pharmaceutical interventions. BMC Infect. Dis. 10, 32 (2010)CrossRefGoogle Scholar
  37. 37.
    Weisstein, E.W.: SIR Model. MathWorld—A Wolfram Web Resource.
  38. 38.
    Brauer, F.: Models for the spread of universally fatal diseases. J. Math. Biol. 28, 451–462 (1990)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Hethcote, H.W.: The mathematics of infectious diseases. SIAM Rev. 42, 599–653 (2000)MathSciNetCrossRefGoogle Scholar
  40. 40.
    Iannelli, M.: Mathematical modelling of epidemics. In: Summer School on Mathematical Models in Life Science: Theory and Simulation, Dobbiaco (2005)Google Scholar
  41. 41.
    Iannelli, M., Martcheva, M.: Homogeneous dynamical systems and the age-structured SIR model with proportionate mixing incidence. Evolution Equations: Applications to Physics, Industry, Life Sciences and Economics. Progress Nonlinear Differential Equations and Applications, vol. 55, pp. 227–251. Birkhäuser, Basel (2003)Google Scholar
  42. 42.
    Rowthorn, R.E., Laxminarayan, R., Gilligan, C.A.: Optimal control of epidemics in metapopulations. J. R. Soc. Interface 6, 1135–1144 (2009)CrossRefGoogle Scholar
  43. 43.
    Li, M.Y., Wang, L.C.: Global stability in some SEIR epidemic models. Mathematical Approaches for Emerging and Reemerging Infectious Diseases: Models, Methods, and Theory. IMA Volumes in Mathematics and its Applications, vol. 126, pp. 295–311. Springer, New York (2002)Google Scholar
  44. 44.
    Behncke, H.: Optimal control of deterministic epidemic. Optim. Control Appl. Meth. 21, 269–285 (2000)MathSciNetCrossRefGoogle Scholar
  45. 45.
    Iacoviello, D., Liuzzi, G.: Optimal control for SIR epidemic model: a two treatments strategy. In: The 16th Mediterranean Conference on Control and Automation, pp. 842–847. Congress Centre, Ajaccio (2008)Google Scholar
  46. 46.
    Clancy, D.: Optimal intervention for epidemic models with general infection and removal rate functions. J. Math. Biol. 39, 309–331 (1999)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Morton, R., Wickwire, K.H.: On the optimal control of a deterministic epidemic. Adv. Appl. Probab. 6, 622–635 (1974)MathSciNetCrossRefGoogle Scholar
  48. 48.
    Sethi, S.P.: Optimal quarantine programmes for controlling an epidemic spread. J. Oper. Res. Soc. 29, 265–268 (1978)MathSciNetCrossRefGoogle Scholar
  49. 49.
    Sethi, S.P., Staats, P.W.: Optimal control of some simple deterministic epidemic models. J. Oper. Res. Soc. 29, 129–136 (1978)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Wickwire, K.: Optimal immunization rules for an epidemic with recovery. Adv. Appl. Probab. 9, 222–223 (1977)CrossRefGoogle Scholar
  51. 51.
    Guo, B.Z.: A controlled age-structured population model for the spread of universally fatal diseases. J. Syst. Sci. Math. Sci. 15, 269–278 (1995)MathSciNetzbMATHGoogle Scholar
  52. 52.
    Jacob, B., Omrane, A.: Optimal control for age-structured population dynamics of incomplete data. J. Math. Anal. Appl. 370, 42–48 (2010)MathSciNetCrossRefGoogle Scholar
  53. 53.
    Chan, W.L., Guo, B.Z.: Optimal birth control of population dynamics. J. Math. Anal. Appl. 144, 532–552 (1989)MathSciNetCrossRefGoogle Scholar
  54. 54.
    Chan, W.L., Guo, B.Z.: Optimal birth control of population dynamics. II. Problems with free final time, phase constraints, and mini-max costs. J. Math. Anal. Appl. 146, 523–539 (1990)MathSciNetCrossRefGoogle Scholar
  55. 55.
    Sun, B.: Maximum principle for optimal boundary control of the Kuramoto-Sivashinsky equation. J. Franklin Inst. 347, 467–482 (2010)MathSciNetCrossRefGoogle Scholar
  56. 56.
    Dubovitskii, A.Y., Milyutin, A.A.: Extremum problems in the presence of constraints. Zh. Vychisl. Mat. i Mat. Fiz. 5, 395–453 (1965) (in Russian). English Translation. USSR Comput. Math. Math. Phys. 5, 1–80 (1965)Google Scholar
  57. 57.
    Girsanov, I.V.: Lectures on Mathematical Theory of Extremum Problems. Economics and Mathematical Systems. Springer, Berlin (1972)CrossRefGoogle Scholar
  58. 58.
    Barbu, V., Iannelli, M.: Optimal control of population dynamics. J. Optim. Theory Appl. 102, 1–14 (1999)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Aniţa, S.: Optimal harvesting for a nonlinear age-dependent population dynamics. J. Math. Anal. Appl. 226, 6–22 (1998)MathSciNetCrossRefGoogle Scholar
  60. 60.
    Aniţa, S., Iannelli, M., Kim, M.-Y., Park, E.-J.: Optimal harvesting for periodic age-dependent population dynamics. SIAM J. Appl. Math. 58, 1648–1666 (1998)MathSciNetCrossRefGoogle Scholar
  61. 61.
    Song, J., Yu, J.Y.: Population System Control. Springer, Berlin (1988)CrossRefGoogle Scholar
  62. 62.
    Department of Demographic Census in National Bureau of Statistics of China, China Population Statistics Yearbook, Scientific and Technical Documents Publishing House, Beijing (1990) (in Chinese)Google Scholar
  63. 63.
    Guo, B.Z., Yao, C.Z.: New results on the exponential stability of non-stationary population dynamics. Acta Math. Sci. (English Ed.) 16, 330–337 (1996)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Rudin, W.: Functional Analysis, 2nd edn. McGraw-Hill Inc., New York (1991)zbMATHGoogle Scholar
  65. 65.
    Yung, S.P.: Multiprocess optimal control problem on Hilbert space and related Hamilton-Jacobi equation. Nonlinear Anal. 29, 1319–1342 (1997)MathSciNetCrossRefGoogle Scholar
  66. 66.
    Gozzi, F., Świȩch, A., Zhou, X.Y.: A corrected proof of the stochastic verification theorem within the framework of viscosity solutions. SIAM J. Control Optim. 43, 2009–2019 (2005)MathSciNetCrossRefGoogle Scholar
  67. 67.
    Zhou, X.Y.: Verification theorems within the framework of viscosity solutions. J. Math. Anal. Appl. 177, 208–225 (1993)MathSciNetCrossRefGoogle Scholar
  68. 68.
    Li, R.H., Feng, G.C.: Numerical Solution of Differential Equations. Higher Education Press, Shanghai (1980). (in Chinese)Google Scholar
  69. 69.
    Crandall, M.G., Lions, P.L.: Viscosity solutions of Hamilton-Jacobi equations. Trans. Am. Math. Soc. 277, 1–42 (1983)MathSciNetCrossRefGoogle Scholar
  70. 70.
    Crandall, M.G., Lions, P.L.: Two approximations of solutions of Hamilton-Jacobi equations. Math. Comp. 43, 1–19 (1984)MathSciNetCrossRefGoogle Scholar
  71. 71.
    Falcone, M., Lanucara, P., Seghini, A.: A splitting algorithm for Hamilton-Jacobi-Bellman equations. Appl. Numer. Math. 15, 207–218 (1994)MathSciNetCrossRefGoogle Scholar
  72. 72.
    McEneaney, W.M.: Max-Plus Methods for Nonlinear Control and Estimation. Birkhäuser, Boston (2006)zbMATHGoogle Scholar
  73. 73.
    McEneaney, W.M.: Convergence rate for a curse-of-dimensionality-free method for Hamilton-Jacobi-Bellman PDEs represented as maxima of quadratic forms. SIAM J. Control Optim. 48, 2651–2685 (2009)MathSciNetCrossRefGoogle Scholar
  74. 74.
    Horowitz, M.B., Damle, A., Burdick, J.W.: Linear Hamilton Jacobi Bellman equations in high dimensions. In: The 53rd IEEE Conference on Decision and Control, pp. 5880–5887. IEEE Press, Los Angeles (2014)Google Scholar
  75. 75.
    Gombao, S.: Approximation of optimal controls for semilinear parabolic PDE by solving Hamilton-Jacobi-Bellman equations. In: Gilliam, D.S., Rosenthal, J. (eds.) Electronic Proceedings of 15th International Symposium on the Mathematical Theory of Networks and Systems, University of Notre Dame, South Bend (2002)Google Scholar
  76. 76.
    Guo, B.Z., Wu, T.T.: Approximation of optimal feedback control: a dynamic programming approach. J. Global Optim. 46, 395–422 (2010)MathSciNetCrossRefGoogle Scholar
  77. 77.
    Kunisch, K., Xie, L.: POD-based feedback control of the Burgers equation by solving the evolutionary HJB equation. Comput. Math. Appl. 49, 1113–1126 (2005)MathSciNetCrossRefGoogle Scholar
  78. 78.
    Lantoine, G., Epenoy, R.: Quadratically constrained linear-quadratic regulator approach for finite-thrust orbital rendezvous. J. Guid. Control Dyn. 35, 1787–1797 (2012)CrossRefGoogle Scholar
  79. 79.
    Tang, G.Y., Wang, H.H.: Successive approximation approach of optimal control for nonlinear discrete-time system. Int. J. Syst. Sci. 36, 153–161 (2005)MathSciNetCrossRefGoogle Scholar
  80. 80.
    Bokanowski, O., Megdich, N., Zidani, H.: Convergence of a non-monotone scheme for Hamilton-Jacobi-Bellman equations with discontinuous initial data. Numer. Math. 115, 1–44 (2010)MathSciNetCrossRefGoogle Scholar
  81. 81.
    Jensen, M., Smears, I.: On the convergence of finite element methods for Hamilton-Jacobi-Bellman equations. SIAM J. Numer. Anal. 51, 137–162 (2013)MathSciNetCrossRefGoogle Scholar
  82. 82.
    Souganidis, P.E.: Approximation schemes for viscosity solution of Hamilton-Jacobi equations. J. Differ. Equ. 59, 1–43 (1985)MathSciNetCrossRefGoogle Scholar
  83. 83.
    Barles, G., Souganidis, P.E.: Convergence of approximation schemes for fully nonlinear second order equations. Asymptotic Anal. 4, 271–283 (1991)MathSciNetzbMATHGoogle Scholar
  84. 84.
    Strikwerda, J.C.: Finite difference methods for the Stokes and Navier-Stokes equations. SIAM. J. Sci. Stat. Comput. 5, 56–68 (1984)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.School of Mathematics and StatisticsBeijing Institute of TechnologyBeijingChina
  2. 2.Beijing Key Laboratory on MCAACIBeijing Institute of TechnologyBeijingChina

Personalised recommendations