Skip to main content
SpringerLink
Log in

Necessary First-Order and Second-Order Optimality Conditions in Discrete-Time Stochastic Systems

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

In this paper, first-order and second-order necessary conditions for optimality for discrete-time stochastic optimal control problems governed by discrete-time Itô equations are established. A new discrete-time backward stochastic equation and discrete-time backward stochastic matrix equation are introduced. Based on the discrete-time backward stochastic Itô equation, a discrete-time stochastic maximum principle for the stochastic discrete optimal control problems is obtained. Moreover, using the discrete-time backward stochastic matrix equation the second-order necessary conditions for optimality are obtained.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Pontryagin, L.S., Boltyanskii, V.G., Gamkrelidze, R.V., Mishchenko, E.F.: Matematicheskaya teoriya optimal’nykh protsessov (Mathematical Theory of Optimal Processes). Nauka, Moscow (1969)

    Google Scholar 

  2. Halkin, H.: On the necessary conditions for the optimal control of nonlinear systems. J. Math. Anal. 12, 1–82 (1964)

    Article  MATH  Google Scholar 

  3. Holtzman, J.M.: Convexity and the maximum principle for discrete systems. IEEE Trans. Automat. Control 11, 30–35 (1966)

    Article  MathSciNet  MATH  Google Scholar 

  4. Vinter, R.B.: Optimality and sensitivity of discrete time processes. Control Cybern. 17, 191–211 (1988)

    MathSciNet  MATH  Google Scholar 

  5. Mahmudov, N.I.: General necessary optimality conditions for stochastic systems with controllable diffusion. (Russian) Statistics and control of random processes (Russian) (Preila, 1987), pp. 135–138. “Nauka”, Moscow (1989)

  6. Gabasov, R.F., Kirillova, F.M.: The Qualitative Theory of Optimal Processes. Nauka, Moscow (1971). (in Russian)

    MATH  Google Scholar 

  7. Mahmudov, N.I., Bashirov, A.E.: First Order and Second Order Necessary Conditions of Optimality for Stochastic Systems. Statistics and Control of Stochastic Processes (Moscow, 1995/1996), pp. 283–295. World Science Publications, River Edge (1997)

  8. Yong, J., Zhou, X.: Stochastic Controls: Hamiltonian Systems and HJB Equations. Springer, Berlin (1999)

    Book  MATH  Google Scholar 

  9. Rami, M., Chen, X., Zhou, X.: Discrete-time indefinite LQ control with state and control dependent noises. J. Glob. Opt. 23, 245–265 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  10. Arkin, V., Evstigneev, I., Medova-Dempster, E., Medova, E., Dempster, M.: Stochastic Models of Control and Economic Dynamics. Academic Press, London (1987)

    Google Scholar 

  11. Muller, H.: Maximum principles for discrete-time nonlinear stochastic control systems. Optimization 27, 343–354 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  12. Blot, J.: An infinite-horizon stochastic discrete-time Pontryagin principle. Nonlinear Anal. Theory Methods Appl. 71, 999–1004 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  13. Zhang, H., Duan, G., Xie, L.: Linear quadratic regulation for linear time-varying systems with multiple input delays. Automatica 42, 1465–1476 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Wang, H., Zhang, H., Wang, X.: Optimal control for stochastic discrete-time systems with multiple input-delays. In: Proceedings of the 10th World Congress on Intelligent Control and Automation, Beijing, July 6–8, pp. 1529–1534 (2012)

  15. Duncan, T.E., Pasik-Duncan, B.: Discrete time linear quadratic control with arbitrary correlated noise. IEEE Trans. Automat. Control 58, 1290–1293 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  16. Lin, X., Zhang, W.: A maximum principle for optimal control of discrete-time stochastic systems with multiplicative noise. IEEE Trans. Automat. Control 60, 1121–1126 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  17. Mingaleeva, Z.T., Shvartsman, I.A.: Second-order necessary optimality conditions for a discrete optimal control problem. Transl. Differ. Uravn. 50, 1640–1646 (2014)

    MathSciNet  MATH  Google Scholar 

  18. Mordukhovich, B.S., Shvartsman, I.: Discrete maximum principle for nonsmooth optimal control problems with delays. Cybern. Syst. Anal. 38, 255–264 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  19. Mordukhovich, B.S.: Approximation Methods in Problems of Optimization and Control. Nauka, Moscow (1988). (in Russian)

    MATH  Google Scholar 

  20. Sorokin, S.P.: Necessary feedback optimality conditions and nonstandard duality in problems of discrete system optimization. Automat. Remote Control 75, 1556–1564 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  21. Mahmudov, N.I.: Maximum Principle for Stochastic Discrete-Time Itô Equations, Brownian Motion: Elements, Dynamics and Applications, Chapter 6, pp. 1–22. Nova (2015)

  22. Dykhta, V.A.: Variational necessary optimality conditions with feedback descent controls for optimal control problems. Doklady Math. 91, 394–396 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  23. Dykhta, V.A.: Positional strengthenings of the maximum principle and sufficient optimality conditions. Proc. Steklov Inst. Math. 293, 43–57 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  24. Bismut, J.: Linear quadratic optimal stochastic control with random coefficients. SIAM J. Control Opt. 14, 419–444 (1976)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The author is grateful to the relevant editors and two anonymous reviewers for their detailed and insightful comments and suggestions, which have led to an improvement of the paper in both content and style.

Author information

Authors and Affiliations

  1. Department of Mathematics, Eastern Mediterranean University, Gazimagusa, T. R. North Cyprus, Mersin 10, Turkey

    Nazim I. Mahmudov

Authors
  1. Nazim I. Mahmudov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nazim I. Mahmudov.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mahmudov, N.I. Necessary First-Order and Second-Order Optimality Conditions in Discrete-Time Stochastic Systems. J Optim Theory Appl 182, 1001–1018 (2019). https://doi.org/10.1007/s10957-019-01478-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-019-01478-y

Keywords

Mathematics Subject Classification

Search

Navigation