Skip to main content
SpringerLink
Log in

Control problems on infinite horizon subject to time-dependent pure state constraints

  • Original Article
  • Published:
Mathematics of Control, Signals, and Systems Aims and scope Submit manuscript

Abstract

In the last decades, control problems with infinite horizons and discount factors have become increasingly central not only for economics but also for applications in artificial intelligence and machine learning. The strong links between reinforcement learning and control theory have led to major efforts toward the development of algorithms to learn how to solve constrained control problems. In particular, discount plays a role in addressing the challenges that come with models that have unbounded disturbances. Although algorithms have been extensively explored, few results take into account time-dependent state constraints, which are imposed in most real-world control applications. For this purpose, here we investigate feasibility and sufficient conditions for Lipschitz regularity of the value function for a class of discounted infinite horizon optimal control problems subject to time-dependent constraints. We focus on problems with data that allow nonautonomous dynamics, and Lagrangian and state constraints that can be unbounded with possibly nonsmooth boundaries.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. In the sense of set-valued maps, see e.g. [1]-Section 1.4.

  2. Here \(W^{1,1}(a,b)\) stands for the space of all absolutely continuous functions on [a, b] endowed with the norm \(\left\| g \right\| =g(a)+\int _a^b g'(s)ds\).

  3. We recall that for a function \(q\in L^1_{\textrm{loc}}([t_0,+\infty [;{\mathbb {R}})\) the integral

    $$\begin{aligned} \int _{t_0}^\infty q (t) \,dt:=\lim _{T \rightarrow \infty }\int _{t_0}^T q (t) \,dt, \end{aligned}$$

    provided this limit exists.

References

  1. Aubin J-P, Frankowska H (2009) Set-valued analysis. Modern Birkhäuser Classics. Birkhäuser Boston Inc, Boston, MA

  2. Baird L (1995) Residual algorithms: reinforcement learning with function approximation. In: Machine learning proceedings. Elsevier, pp 30–37

  3. Basco V (2022) Weak epigraphical solutions to Hamilton–Jacobi–Bellman equations on infinite horizon. J Math Anal Appl 515(2):126452

    Article  MathSciNet  MATH  Google Scholar 

  4. Basco V, Cannarsa P, Frankowska H (2018) Necessary conditions for infinite horizon optimal control problems with state constraints. Math Control Relat Fields 8(3–4):535–555

    Article  MathSciNet  MATH  Google Scholar 

  5. Basco V, Frankowska H (2019) Hamilton–Jacobi–Bellman equations with time-measurable data and infinite horizon. Nonlinear Differ Equ Appl 26(1):7

    Article  MathSciNet  MATH  Google Scholar 

  6. Basco V, Frankowska H (2019) Lipschitz continuity of the value function for the infinite horizon optimal control problem under state constraints. In: Alabau-Boussouira F, et al (eds) Trends in control theory and partial differential equations, vol 32 of Springer INdAM Series. Springer International Publishing, pp 15 – 52

  7. Bertsekas D (2022) Dynamic programming and optimal control, volume 1. Athena scientific

  8. Bertsekas D (2019) Reinforcement learning and optimal control. Athena Scientific

  9. Blackwell D (1965) Discounted dynamic programming. Ann Math Stat 36(1):226–235

    Article  MathSciNet  MATH  Google Scholar 

  10. Boyan J, Moore A (1994) Generalization in reinforcement learning: safely approximating the value function. Adv Neural Inf Process Syst 7

  11. Calafiore GC, Fagiano L (2012) Robust model predictive control via scenario optimization. IEEE Trans Autom Control 58(1):219–224

    Article  MathSciNet  MATH  Google Scholar 

  12. Cannon M, Kouvaritakis B, Raković SV, Cheng Q (2010) Stochastic tubes in model predictive control with probabilistic constraints. IEEE Trans Autom Control 56(1):194–200

    Article  MathSciNet  MATH  Google Scholar 

  13. Crandall MG, Evans LC, Lions P-L (1984) Some properties of viscosity solutions of Hamilton–Jacobi equations. Trans Amer Math Soc 282(2):487–502

    Article  MathSciNet  MATH  Google Scholar 

  14. Crandall MG, Lions P-L (1983) Viscosity solutions of Hamilton–Jacobi equations. Trans Amer Math Soc 277(1):1–42

    Article  MathSciNet  MATH  Google Scholar 

  15. De Jager B, Van Keulen T (2013) Optimal control of hybrid vehicles. Springer, Kessels

  16. De Pinho MR, Foroozandeh Z, Matos A (2016) Optimal control problems for path planing of AUV using simplified models. In: 2016 IEEE 55th conference on decision and control (CDC), pp 210–215. IEEE

  17. Farina M, Giulioni L, Magni L, Scattolini R (2013) A probabilistic approach to model predictive control. In: 52nd IEEE conference on decision and control. IEEE, pp 7734–7739

  18. Feichtinger G, Kovacevic RM, Tragler G (2018) Control systems and mathematical methods in economics, volume 687. Lecture Notes in Economics and Mathematical Systems. Springer

  19. Frankel A (2016) Discounted quotas. J Econ Theory 166:396–444

    Article  MathSciNet  MATH  Google Scholar 

  20. Gordon GJ (1995) Stable function approximation in dynamic programming. In: Machine learning proceedings. Elsevier, pp 261–268

  21. Hashimoto T (2013) Probabilistic constrained model predictive control for linear discrete-time systems with additive stochastic disturbances. In: 52nd IEEE conference on decision and control. IEEE, pp 6434–6439

  22. Hewing L, Zeilinger MN (2018) Stochastic model predictive control for linear systems using probabilistic reachable sets. In: 2018 IEEE conference on decision and control (CDC), pp 5182–5188. IEEE

  23. Kamgarpour M, Summers T (2017) On infinite dimensional linear programming approach to stochastic control. IFAC-PapersOnLine 50(1):6148–6153

    Article  Google Scholar 

  24. Kouvaritakis B, Cannon M, Couchman P (2006) Mpc as a tool for sustainable development integrated policy assessment. IEEE Trans Autom Control 51(1):145–149

    Article  MathSciNet  MATH  Google Scholar 

  25. Kouvaritakis B, Cannon M, Raković SV, Cheng Q (2010) Explicit use of probabilistic distributions in linear predictive control. Automatica 46(10):1719–1724

    Article  MathSciNet  MATH  Google Scholar 

  26. Margellos K, Goulart P, Lygeros J (2014) On the road between robust optimization and the scenario approach for chance constrained optimization problems. IEEE Trans Autom Control 59(8):2258–2263

    Article  MathSciNet  MATH  Google Scholar 

  27. Menon PKA, Briggs MM (1990) Near-optimal midcourse guidance for air-to-air missiles. J Guid Control Dyn 13(4):596–602

    Article  Google Scholar 

  28. Munos R (1998) A general convergence method for reinforcement learning in the continuous case. In: European conference on machine learning. Springer, pp 394–405

  29. Munos R (2000) A study of reinforcement learning in the continuous case by the means of viscosity solutions. Mach Learn 40:265–299

    Article  MATH  Google Scholar 

  30. Nystrup P, Boyd S, Lindström E, Madsen H (2019) Multi-period portfolio selection with drawdown control. Ann Oper Res 282(1):245–271

    Article  MathSciNet  MATH  Google Scholar 

  31. Postoyan R, Buşoniu L, Nešić D, Daafouz J (2016) Stability analysis of discrete-time infinite-horizon optimal control with discounted cost. IEEE Trans Autom Control 62(6):2736–2749

    Article  MathSciNet  MATH  Google Scholar 

  32. Schildbach G, Goulart P, Morari M (2015) Linear controller design for chance constrained systems. Automatica 51:278–284

    Article  MathSciNet  MATH  Google Scholar 

  33. Van Parys BPG, Goulart PJ, Morari M (2013) Infinite horizon performance bounds for uncertain constrained systems. IEEE Trans Autom Control 58(11):2803–2817

    Article  MathSciNet  MATH  Google Scholar 

  34. Vinter RB (2000) Optimal Control. Birkhäuser, Boston, MA

    MATH  Google Scholar 

  35. Weston J, Tolić D, Palunko I (2022) Mixed use of Pontryagin’s principle and the Hamilton–Jacobi–Bellman equation in infinite-and finite-horizon constrained optimal control. In: International conference on intelligent autonomous systems. Springer, pp 167–185

Download references

Author information

Authors and Affiliations

  1. Thales Alenia Space, Via Saccomuro, 24, Rome, Italy

    Vincenzo Basco

Authors
  1. Vincenzo Basco
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vincenzo Basco.

Ethics declarations

Conflict of interest

The author declares no conflicts of interest in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Basco, V. Control problems on infinite horizon subject to time-dependent pure state constraints. Math. Control Signals Syst. (2023). https://doi.org/10.1007/s00498-023-00372-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00498-023-00372-3

Keywords

Search

Navigation