Advertisement

Vietnam Journal of Mathematics

, Volume 42, Issue 1, pp 83–113 | Cite as

Solving Stochastic Optimal Control Problems by a Wiener Chaos Approach

  • Tony HuschtoEmail author
  • Sebastian Sager
Article

Abstract

We introduce a novel generic methodology to solve continuous finite-horizon stochastic optimal control problems (SOCPs). We treat controlled stochastic differential equations (SDEs) within the Wiener chaos framework by utilizing Malliavin calculus and developing innovative ideas to preserve the feedback character of optimal Markov decision rules.

This allows a direct reformulation of SOCPs into deterministic ones. Hence, it facilitates using Bock’s direct multiple shooting method for solving SOCPs and pioneers the extension of sophisticated methods for deterministic control to the broad context of SDEs.

Numerical examples validate this new framework with huge computational advantages compared to standard ideas in SOC.

Keywords

Stochastic optimal control Markov control Wiener chaos Malliavin calculus Direct multiple shooting 

Mathematics Subject Classification (2000)

93E20 60H35 49J15 

Notes

Acknowledgements

The authors like to express their gratitude to Prof. M. Podolskij from the University of Heidelberg for his helpful comments and suggestions.

This research was supported by the Heidelberg Graduate School Mathematical and Computational Methods for the Sciences and by the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement n o  FP7-ICT-2009-4 248940.

References

  1. 1.
    Augustin, F., Gilg, A., Paffrath, M., Rentrop, P., Wever, U.: Polynomial chaos for the approximation of uncertainties: chances and limits. Eur. J. Appl. Math. 19, 149–190 (2008) CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Augustin, F., Rentrop, P.: Stochastic Galerkin techniques for random ordinary differential equations. Numer. Math. 122, 399–419 (2012) CrossRefzbMATHMathSciNetGoogle Scholar
  3. 3.
    Azzato, J.D., Krawczyk, J.B.: SOCSol4L: an improved MATLAB package for approximating the solution to a continuous-time stochastic optimal control problem. Working paper of the Victoria University of Wellington (2006) Google Scholar
  4. 4.
    Barczy, M., Iglói, E.: Karhunen–Loève expansions of α-Wiener bridges. Cent. Eur. J. Math. 9, 65–84 (2011) CrossRefzbMATHMathSciNetGoogle Scholar
  5. 5.
    Blatman, G., Sudret, B.: Sparse polynomial chaos expansions based on an adaptive least angle regression algorithm. In: 19ème Congrès Français de Mécanique, Marseille (2009) Google Scholar
  6. 6.
    Bock, H.G., Plitt, K.J.: A multiple shooting algorithm for direct solution of optimal control problems. In: Proceedings of the 9th IFACWorld Congress, Budapest, pp. 242–247. Pergamon, New York (1984) Google Scholar
  7. 7.
    Bulirsch, R.: Die Mehrzielmethode zur numerischen Lösung von nichtlinearen Randwertproblemen und Aufgaben der optimalen Steuerung. Technical report, Carl-Cranz-Gesellschaft, Oberpfaffenhofen (1971) Google Scholar
  8. 8.
    Cameron, R.H., Martin, W.T.: The orthogonal development of non-linear functionals in series of Fourier–Hermite functionals. Ann. Math. 48, 385–392 (1947) CrossRefzbMATHMathSciNetGoogle Scholar
  9. 9.
    Fleming, W.H., Rishel, R.W.: Deterministic and Stochastic Optimal Control. Springer, New York (1982) CrossRefGoogle Scholar
  10. 10.
    Frauenfelder, P., Schwab, C., Todor, R.A.: Finite elements for elliptic problems with stochastic coefficients. Comput. Methods Appl. Mech. Eng. 194, 205–228 (2005) CrossRefzbMATHMathSciNetGoogle Scholar
  11. 11.
    Ghanem, R.G., Spanos, P.D.: Stochastic Finite Elements. Springer, Heidelberg (1991) CrossRefzbMATHGoogle Scholar
  12. 12.
    Hanson, F.B.: Stochastic digital control system techniques. In: Control and Dynamic Systems. Advances in Theory and Applications, vol. 76, pp. 103–162. Academic Press, New York (1996) Google Scholar
  13. 13.
    Janson, S.: Gaussian–Hilbert Spaces. Cambridge University Press, Cambridge (1997) zbMATHGoogle Scholar
  14. 14.
    Karhunen, K.: Über lineare Methoden in der Wahrscheinlichkeitsrechnung, Vol. 37. Annales AcademiæScientiarum Fennicæ, Ser. A. I (1947) Google Scholar
  15. 15.
    Karniadikis, G.E., Su, C.-H., Xiu, D., Lucor, D., Schwab, C., Todor, R.A.: Generalized polynomial chaos solution for differential equations with random inputs. Report No. 2005-01, ETH Zürich (2005) Google Scholar
  16. 16.
    Keese, A., Matthies, H.G.: Numerical methods and Smolyak quadrature for nonlinear stochastic partial differential equations. Informatikbericht Nr. 2003-5, TU Braunschweig (2003) Google Scholar
  17. 17.
    Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations, 3rd edn. Springer, Berlin (1999) Google Scholar
  18. 18.
    Korn, R., Korn, E.: Option Pricing and Portfolio Optimization. Graduate Studies in Mathematics, vol. 31. Oxford University Press, Oxford (2001) CrossRefzbMATHGoogle Scholar
  19. 19.
    Krawczyk, J.B.: A Markovian approximated solution to a portfolio management problem. ITEM 1(1) (2001) Google Scholar
  20. 20.
    Kushner, H.J.: Numerical methods for stochastic control problems in continuous time. SIAM J. Control Optim. 28, 999–1048 (1990) CrossRefzbMATHMathSciNetGoogle Scholar
  21. 21.
    Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, Berlin (2001) CrossRefzbMATHGoogle Scholar
  22. 22.
    Leineweber, D.B.: Analyse und Restrukturierung eines Verfahrens zur direkten Lösung von Optimal-Steuerungsproblemen. Diploma thesis, Ruprecht-Karls-Universität, Heidelberg (1995) Google Scholar
  23. 23.
    Leineweber, D.B., Bauer, I., Bock, H.G., Schlöder, J.P.: An efficient multiple shooting based reduced SQP strategy for large-scale dynamic process optimization, part I: theoretical aspects. Comput. Chem. Eng. 27, 157–166 (2003) CrossRefGoogle Scholar
  24. 24.
    Loève, M.: Probability Theory II, 4th edn. Springer, Berlin (1978) CrossRefzbMATHGoogle Scholar
  25. 25.
    Lototsky, S., Rozovskii, B.: Stochastic differential equations: a Wiener chaos approach. In: From Stochastic Calculus to Mathematical Finance, pp. 433–506. Springer, Berlin (2006) CrossRefGoogle Scholar
  26. 26.
    Luo, W.: Wiener Chaos Expansion and Numerical Solutions of Stochastic Partial Differential Equations. Caltech (2006) Google Scholar
  27. 27.
    Luschgy, H., Pagès, G.: Functional quantization of Gaussian processes. J. Funct. Anal. 196, 486–531 (2002) CrossRefzbMATHMathSciNetGoogle Scholar
  28. 28.
    Luschgy, H., Pagès, G.: Functional quantization of a class of Brownian diffusions: a constructive approach. Stoch. Process. Appl. 116, 310–336 (2006) CrossRefzbMATHGoogle Scholar
  29. 29.
    Mathelin, L., Gallivan, K.A.: A compressed sensing approach for partial differential equations with random data input. Commun. Comput. Phys. 12, 919–954 (2012) MathSciNetGoogle Scholar
  30. 30.
    Nourdin, I., Peccati, G.: Normal Approximations with Malliavin Calculus: From Stein’s Method to Universality. Cambridge University Press, Cambridge (2012) CrossRefGoogle Scholar
  31. 31.
    Nualart, D.: The Malliavin Calculus and Related Topics, 2nd edn. Springer, Berlin, New York (2006) zbMATHGoogle Scholar
  32. 32.
    Ogura, H.: Orthogonal functionals of the Poisson process. IEEE Trans. Inf. Theory 18, 473–481 (1972) CrossRefzbMATHMathSciNetGoogle Scholar
  33. 33.
    Øksendal, B.: Stochastic Differential Equations, 6th edn. Springer, Berlin (2007) Google Scholar
  34. 34.
    Osborne, M.R.: On shooting methods for boundary value problems. J. Math. Anal. Appl. 27, 417–433 (1969) CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Pagès, G., Pham, H.: A quantization algorithm for multidimensional stochastic control problems. Pre-print 697, Laboratoire de Probabilités & Modèles Aléatoires, Universités de Paris 6/7 (2001) Google Scholar
  36. 36.
    Pagès, G., Printems, J.: Optimal quadratic quantization for numerics: the Gaussian case. Monte Carlo Methods Appl. 9, 135–165 (2003) CrossRefzbMATHMathSciNetGoogle Scholar
  37. 37.
    Potschka, A.: Handling path constraints in a direct multiple shooting method for optimal control problems. Diploma thesis, Universität Heidelberg (2006) Google Scholar
  38. 38.
    Powell, M.J.D.: A fast algorithm for nonlinearly constrained optimization calculations. In: Watson, G.A. (ed.) Numerical Analysis, Dundee, 1977. Lecture Notes in Mathematics, vol. 630. Springer, Berlin (1978) CrossRefGoogle Scholar
  39. 39.
    Schillings, C., Schulz, V.: On the influence of robustness measures on shape optimization with stochastic uncertainties. Technical report, Universität Trier (2012) Google Scholar
  40. 40.
    Segall, A., Kailath, T.: Orthogonal functionals of independent-increment processes. IEEE Trans. Inf. Theory 22, 287–298 (1976) CrossRefzbMATHMathSciNetGoogle Scholar
  41. 41.
    Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. Sov. Math. Dokl. 4, 240–243 (1963) Google Scholar
  42. 42.
    Wiener, N.: The homogeneous chaos. Am. J. Math. 60, 897–936 (1938) CrossRefMathSciNetGoogle Scholar
  43. 43.
    Wiener, N.: Nonlinear Problems in Random Theory. Technology Press of the Massachusetts Institute of Technology/Wiley, New York (1958) zbMATHGoogle Scholar
  44. 44.
    Wilson, R.B.: A simplicial algorithm for concave programming. PhD thesis, Harvard University (1963) Google Scholar

Copyright information

© Vietnam Academy of Science and Technology (VAST) and Springer Science+Business Media Singapore 2014

Authors and Affiliations

  1. 1.Interdisciplinary Center for Scientific ComputingRuprecht-Karls University of HeidelbergHeidelbergGermany
  2. 2.Institute for Mathematical OptimizationOtto-von-Guericke University MagdeburgMagdeburgGermany

Personalised recommendations