Skip to main content
Log in

Solving Stochastic Optimal Control Problems by a Wiener Chaos Approach

  • Published:
Vietnam Journal of Mathematics Aims and scope Submit manuscript

Abstract

We introduce a novel generic methodology to solve continuous finite-horizon stochastic optimal control problems (SOCPs). We treat controlled stochastic differential equations (SDEs) within the Wiener chaos framework by utilizing Malliavin calculus and developing innovative ideas to preserve the feedback character of optimal Markov decision rules.

This allows a direct reformulation of SOCPs into deterministic ones. Hence, it facilitates using Bock’s direct multiple shooting method for solving SOCPs and pioneers the extension of sophisticated methods for deterministic control to the broad context of SDEs.

Numerical examples validate this new framework with huge computational advantages compared to standard ideas in SOC.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Augustin, F., Gilg, A., Paffrath, M., Rentrop, P., Wever, U.: Polynomial chaos for the approximation of uncertainties: chances and limits. Eur. J. Appl. Math. 19, 149–190 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  2. Augustin, F., Rentrop, P.: Stochastic Galerkin techniques for random ordinary differential equations. Numer. Math. 122, 399–419 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  3. Azzato, J.D., Krawczyk, J.B.: SOCSol4L: an improved MATLAB package for approximating the solution to a continuous-time stochastic optimal control problem. Working paper of the Victoria University of Wellington (2006)

  4. Barczy, M., Iglói, E.: Karhunen–Loève expansions of α-Wiener bridges. Cent. Eur. J. Math. 9, 65–84 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  5. Blatman, G., Sudret, B.: Sparse polynomial chaos expansions based on an adaptive least angle regression algorithm. In: 19ème Congrès Français de Mécanique, Marseille (2009)

    Google Scholar 

  6. Bock, H.G., Plitt, K.J.: A multiple shooting algorithm for direct solution of optimal control problems. In: Proceedings of the 9th IFACWorld Congress, Budapest, pp. 242–247. Pergamon, New York (1984)

    Google Scholar 

  7. Bulirsch, R.: Die Mehrzielmethode zur numerischen Lösung von nichtlinearen Randwertproblemen und Aufgaben der optimalen Steuerung. Technical report, Carl-Cranz-Gesellschaft, Oberpfaffenhofen (1971)

  8. Cameron, R.H., Martin, W.T.: The orthogonal development of non-linear functionals in series of Fourier–Hermite functionals. Ann. Math. 48, 385–392 (1947)

    Article  MATH  MathSciNet  Google Scholar 

  9. Fleming, W.H., Rishel, R.W.: Deterministic and Stochastic Optimal Control. Springer, New York (1982)

    Book  Google Scholar 

  10. Frauenfelder, P., Schwab, C., Todor, R.A.: Finite elements for elliptic problems with stochastic coefficients. Comput. Methods Appl. Mech. Eng. 194, 205–228 (2005)

    Article  MATH  MathSciNet  Google Scholar 

  11. Ghanem, R.G., Spanos, P.D.: Stochastic Finite Elements. Springer, Heidelberg (1991)

    Book  MATH  Google Scholar 

  12. Hanson, F.B.: Stochastic digital control system techniques. In: Control and Dynamic Systems. Advances in Theory and Applications, vol. 76, pp. 103–162. Academic Press, New York (1996)

    Google Scholar 

  13. Janson, S.: Gaussian–Hilbert Spaces. Cambridge University Press, Cambridge (1997)

    MATH  Google Scholar 

  14. Karhunen, K.: Über lineare Methoden in der Wahrscheinlichkeitsrechnung, Vol. 37. Annales AcademiæScientiarum Fennicæ, Ser. A. I (1947)

  15. Karniadikis, G.E., Su, C.-H., Xiu, D., Lucor, D., Schwab, C., Todor, R.A.: Generalized polynomial chaos solution for differential equations with random inputs. Report No. 2005-01, ETH Zürich (2005)

  16. Keese, A., Matthies, H.G.: Numerical methods and Smolyak quadrature for nonlinear stochastic partial differential equations. Informatikbericht Nr. 2003-5, TU Braunschweig (2003)

  17. Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations, 3rd edn. Springer, Berlin (1999)

    Google Scholar 

  18. Korn, R., Korn, E.: Option Pricing and Portfolio Optimization. Graduate Studies in Mathematics, vol. 31. Oxford University Press, Oxford (2001)

    Book  MATH  Google Scholar 

  19. Krawczyk, J.B.: A Markovian approximated solution to a portfolio management problem. ITEM 1(1) (2001)

  20. Kushner, H.J.: Numerical methods for stochastic control problems in continuous time. SIAM J. Control Optim. 28, 999–1048 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  21. Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, Berlin (2001)

    Book  MATH  Google Scholar 

  22. Leineweber, D.B.: Analyse und Restrukturierung eines Verfahrens zur direkten Lösung von Optimal-Steuerungsproblemen. Diploma thesis, Ruprecht-Karls-Universität, Heidelberg (1995)

    Google Scholar 

  23. Leineweber, D.B., Bauer, I., Bock, H.G., Schlöder, J.P.: An efficient multiple shooting based reduced SQP strategy for large-scale dynamic process optimization, part I: theoretical aspects. Comput. Chem. Eng. 27, 157–166 (2003)

    Article  Google Scholar 

  24. Loève, M.: Probability Theory II, 4th edn. Springer, Berlin (1978)

    Book  MATH  Google Scholar 

  25. Lototsky, S., Rozovskii, B.: Stochastic differential equations: a Wiener chaos approach. In: From Stochastic Calculus to Mathematical Finance, pp. 433–506. Springer, Berlin (2006)

    Chapter  Google Scholar 

  26. Luo, W.: Wiener Chaos Expansion and Numerical Solutions of Stochastic Partial Differential Equations. Caltech (2006)

  27. Luschgy, H., Pagès, G.: Functional quantization of Gaussian processes. J. Funct. Anal. 196, 486–531 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  28. Luschgy, H., Pagès, G.: Functional quantization of a class of Brownian diffusions: a constructive approach. Stoch. Process. Appl. 116, 310–336 (2006)

    Article  MATH  Google Scholar 

  29. Mathelin, L., Gallivan, K.A.: A compressed sensing approach for partial differential equations with random data input. Commun. Comput. Phys. 12, 919–954 (2012)

    MathSciNet  Google Scholar 

  30. Nourdin, I., Peccati, G.: Normal Approximations with Malliavin Calculus: From Stein’s Method to Universality. Cambridge University Press, Cambridge (2012)

    Book  Google Scholar 

  31. Nualart, D.: The Malliavin Calculus and Related Topics, 2nd edn. Springer, Berlin, New York (2006)

    MATH  Google Scholar 

  32. Ogura, H.: Orthogonal functionals of the Poisson process. IEEE Trans. Inf. Theory 18, 473–481 (1972)

    Article  MATH  MathSciNet  Google Scholar 

  33. Øksendal, B.: Stochastic Differential Equations, 6th edn. Springer, Berlin (2007)

    Google Scholar 

  34. Osborne, M.R.: On shooting methods for boundary value problems. J. Math. Anal. Appl. 27, 417–433 (1969)

    Article  MATH  MathSciNet  Google Scholar 

  35. Pagès, G., Pham, H.: A quantization algorithm for multidimensional stochastic control problems. Pre-print 697, Laboratoire de Probabilités & Modèles Aléatoires, Universités de Paris 6/7 (2001)

  36. Pagès, G., Printems, J.: Optimal quadratic quantization for numerics: the Gaussian case. Monte Carlo Methods Appl. 9, 135–165 (2003)

    Article  MATH  MathSciNet  Google Scholar 

  37. Potschka, A.: Handling path constraints in a direct multiple shooting method for optimal control problems. Diploma thesis, Universität Heidelberg (2006)

  38. Powell, M.J.D.: A fast algorithm for nonlinearly constrained optimization calculations. In: Watson, G.A. (ed.) Numerical Analysis, Dundee, 1977. Lecture Notes in Mathematics, vol. 630. Springer, Berlin (1978)

    Chapter  Google Scholar 

  39. Schillings, C., Schulz, V.: On the influence of robustness measures on shape optimization with stochastic uncertainties. Technical report, Universität Trier (2012)

  40. Segall, A., Kailath, T.: Orthogonal functionals of independent-increment processes. IEEE Trans. Inf. Theory 22, 287–298 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  41. Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. Sov. Math. Dokl. 4, 240–243 (1963)

    Google Scholar 

  42. Wiener, N.: The homogeneous chaos. Am. J. Math. 60, 897–936 (1938)

    Article  MathSciNet  Google Scholar 

  43. Wiener, N.: Nonlinear Problems in Random Theory. Technology Press of the Massachusetts Institute of Technology/Wiley, New York (1958)

    MATH  Google Scholar 

  44. Wilson, R.B.: A simplicial algorithm for concave programming. PhD thesis, Harvard University (1963)

Download references

Acknowledgements

The authors like to express their gratitude to Prof. M. Podolskij from the University of Heidelberg for his helpful comments and suggestions.

This research was supported by the Heidelberg Graduate School Mathematical and Computational Methods for the Sciences and by the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement n o FP7-ICT-2009-4 248940.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tony Huschto.

Appendix: Sparse and Adaptive Indices

Appendix: Sparse and Adaptive Indices

Table 4 List of sparse (sp) and adaptive (ad) indices used for the numerical examples in Sect. 6. The reference numbers coincide with those in Tables 1 and 3, the dagger symbol denotes the combinations used in the figures

Rights and permissions

Reprints and permissions

About this article

Cite this article

Huschto, T., Sager, S. Solving Stochastic Optimal Control Problems by a Wiener Chaos Approach. Viet J Math 42, 83–113 (2014). https://doi.org/10.1007/s10013-014-0060-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10013-014-0060-8

Keywords

Mathematics Subject Classification (2000)

Navigation