Abstract
We introduce a novel generic methodology to solve continuous finite-horizon stochastic optimal control problems (SOCPs). We treat controlled stochastic differential equations (SDEs) within the Wiener chaos framework by utilizing Malliavin calculus and developing innovative ideas to preserve the feedback character of optimal Markov decision rules.
This allows a direct reformulation of SOCPs into deterministic ones. Hence, it facilitates using Bock’s direct multiple shooting method for solving SOCPs and pioneers the extension of sophisticated methods for deterministic control to the broad context of SDEs.
Numerical examples validate this new framework with huge computational advantages compared to standard ideas in SOC.
Similar content being viewed by others
References
Augustin, F., Gilg, A., Paffrath, M., Rentrop, P., Wever, U.: Polynomial chaos for the approximation of uncertainties: chances and limits. Eur. J. Appl. Math. 19, 149–190 (2008)
Augustin, F., Rentrop, P.: Stochastic Galerkin techniques for random ordinary differential equations. Numer. Math. 122, 399–419 (2012)
Azzato, J.D., Krawczyk, J.B.: SOCSol4L: an improved MATLAB package for approximating the solution to a continuous-time stochastic optimal control problem. Working paper of the Victoria University of Wellington (2006)
Barczy, M., Iglói, E.: Karhunen–Loève expansions of α-Wiener bridges. Cent. Eur. J. Math. 9, 65–84 (2011)
Blatman, G., Sudret, B.: Sparse polynomial chaos expansions based on an adaptive least angle regression algorithm. In: 19ème Congrès Français de Mécanique, Marseille (2009)
Bock, H.G., Plitt, K.J.: A multiple shooting algorithm for direct solution of optimal control problems. In: Proceedings of the 9th IFACWorld Congress, Budapest, pp. 242–247. Pergamon, New York (1984)
Bulirsch, R.: Die Mehrzielmethode zur numerischen Lösung von nichtlinearen Randwertproblemen und Aufgaben der optimalen Steuerung. Technical report, Carl-Cranz-Gesellschaft, Oberpfaffenhofen (1971)
Cameron, R.H., Martin, W.T.: The orthogonal development of non-linear functionals in series of Fourier–Hermite functionals. Ann. Math. 48, 385–392 (1947)
Fleming, W.H., Rishel, R.W.: Deterministic and Stochastic Optimal Control. Springer, New York (1982)
Frauenfelder, P., Schwab, C., Todor, R.A.: Finite elements for elliptic problems with stochastic coefficients. Comput. Methods Appl. Mech. Eng. 194, 205–228 (2005)
Ghanem, R.G., Spanos, P.D.: Stochastic Finite Elements. Springer, Heidelberg (1991)
Hanson, F.B.: Stochastic digital control system techniques. In: Control and Dynamic Systems. Advances in Theory and Applications, vol. 76, pp. 103–162. Academic Press, New York (1996)
Janson, S.: Gaussian–Hilbert Spaces. Cambridge University Press, Cambridge (1997)
Karhunen, K.: Über lineare Methoden in der Wahrscheinlichkeitsrechnung, Vol. 37. Annales AcademiæScientiarum Fennicæ, Ser. A. I (1947)
Karniadikis, G.E., Su, C.-H., Xiu, D., Lucor, D., Schwab, C., Todor, R.A.: Generalized polynomial chaos solution for differential equations with random inputs. Report No. 2005-01, ETH Zürich (2005)
Keese, A., Matthies, H.G.: Numerical methods and Smolyak quadrature for nonlinear stochastic partial differential equations. Informatikbericht Nr. 2003-5, TU Braunschweig (2003)
Kloeden, P.E., Platen, E.: Numerical Solution of Stochastic Differential Equations, 3rd edn. Springer, Berlin (1999)
Korn, R., Korn, E.: Option Pricing and Portfolio Optimization. Graduate Studies in Mathematics, vol. 31. Oxford University Press, Oxford (2001)
Krawczyk, J.B.: A Markovian approximated solution to a portfolio management problem. ITEM 1(1) (2001)
Kushner, H.J.: Numerical methods for stochastic control problems in continuous time. SIAM J. Control Optim. 28, 999–1048 (1990)
Kushner, H.J., Dupuis, P.: Numerical Methods for Stochastic Control Problems in Continuous Time. Springer, Berlin (2001)
Leineweber, D.B.: Analyse und Restrukturierung eines Verfahrens zur direkten Lösung von Optimal-Steuerungsproblemen. Diploma thesis, Ruprecht-Karls-Universität, Heidelberg (1995)
Leineweber, D.B., Bauer, I., Bock, H.G., Schlöder, J.P.: An efficient multiple shooting based reduced SQP strategy for large-scale dynamic process optimization, part I: theoretical aspects. Comput. Chem. Eng. 27, 157–166 (2003)
Loève, M.: Probability Theory II, 4th edn. Springer, Berlin (1978)
Lototsky, S., Rozovskii, B.: Stochastic differential equations: a Wiener chaos approach. In: From Stochastic Calculus to Mathematical Finance, pp. 433–506. Springer, Berlin (2006)
Luo, W.: Wiener Chaos Expansion and Numerical Solutions of Stochastic Partial Differential Equations. Caltech (2006)
Luschgy, H., Pagès, G.: Functional quantization of Gaussian processes. J. Funct. Anal. 196, 486–531 (2002)
Luschgy, H., Pagès, G.: Functional quantization of a class of Brownian diffusions: a constructive approach. Stoch. Process. Appl. 116, 310–336 (2006)
Mathelin, L., Gallivan, K.A.: A compressed sensing approach for partial differential equations with random data input. Commun. Comput. Phys. 12, 919–954 (2012)
Nourdin, I., Peccati, G.: Normal Approximations with Malliavin Calculus: From Stein’s Method to Universality. Cambridge University Press, Cambridge (2012)
Nualart, D.: The Malliavin Calculus and Related Topics, 2nd edn. Springer, Berlin, New York (2006)
Ogura, H.: Orthogonal functionals of the Poisson process. IEEE Trans. Inf. Theory 18, 473–481 (1972)
Øksendal, B.: Stochastic Differential Equations, 6th edn. Springer, Berlin (2007)
Osborne, M.R.: On shooting methods for boundary value problems. J. Math. Anal. Appl. 27, 417–433 (1969)
Pagès, G., Pham, H.: A quantization algorithm for multidimensional stochastic control problems. Pre-print 697, Laboratoire de Probabilités & Modèles Aléatoires, Universités de Paris 6/7 (2001)
Pagès, G., Printems, J.: Optimal quadratic quantization for numerics: the Gaussian case. Monte Carlo Methods Appl. 9, 135–165 (2003)
Potschka, A.: Handling path constraints in a direct multiple shooting method for optimal control problems. Diploma thesis, Universität Heidelberg (2006)
Powell, M.J.D.: A fast algorithm for nonlinearly constrained optimization calculations. In: Watson, G.A. (ed.) Numerical Analysis, Dundee, 1977. Lecture Notes in Mathematics, vol. 630. Springer, Berlin (1978)
Schillings, C., Schulz, V.: On the influence of robustness measures on shape optimization with stochastic uncertainties. Technical report, Universität Trier (2012)
Segall, A., Kailath, T.: Orthogonal functionals of independent-increment processes. IEEE Trans. Inf. Theory 22, 287–298 (1976)
Smolyak, S.A.: Quadrature and interpolation formulas for tensor products of certain classes of functions. Sov. Math. Dokl. 4, 240–243 (1963)
Wiener, N.: The homogeneous chaos. Am. J. Math. 60, 897–936 (1938)
Wiener, N.: Nonlinear Problems in Random Theory. Technology Press of the Massachusetts Institute of Technology/Wiley, New York (1958)
Wilson, R.B.: A simplicial algorithm for concave programming. PhD thesis, Harvard University (1963)
Acknowledgements
The authors like to express their gratitude to Prof. M. Podolskij from the University of Heidelberg for his helpful comments and suggestions.
This research was supported by the Heidelberg Graduate School Mathematical and Computational Methods for the Sciences and by the European Union Seventh Framework Programme FP7/2007-2013 under grant agreement n o FP7-ICT-2009-4 248940.
Author information
Authors and Affiliations
Corresponding author
Appendix: Sparse and Adaptive Indices
Appendix: Sparse and Adaptive Indices
Rights and permissions
About this article
Cite this article
Huschto, T., Sager, S. Solving Stochastic Optimal Control Problems by a Wiener Chaos Approach. Viet J Math 42, 83–113 (2014). https://doi.org/10.1007/s10013-014-0060-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10013-014-0060-8