Abstract
Stochastic dual dynamic programming is a cutting plane type algorithm for multi-stage stochastic optimization originated about 30 years ago. In spite of its popularity in practice, there does not exist any analysis on the convergence rates of this method. In this paper, we first establish the number of iterations, i.e., iteration complexity, required by a basic dual dynamic programming method for solving single-scenario multi-stage optimization problems, by introducing novel mathematical tools including the saturation of search points. We then refine these basic tools and establish the iteration complexity for an explorative dual dynamic programing method proposed herein and the classic stochastic dual dynamic programming method for solving more general multi-stage stochastic optimization problems under the standard stage-wise independence assumption. Our results indicate that the complexity of these methods mildly increases with the number of stages T, in fact linearly dependent on T for discounted problems. Therefore, they are efficient for strategic decision making which involves a large number of stages, but with a relatively small number of decision variables in each stage. Without explicitly discretizing the state and action spaces, these methods might also be pertinent to the related reinforcement learning and stochastic control areas.
Similar content being viewed by others
Change history
11 March 2022
A Correction to this paper has been published: https://doi.org/10.1007/s10107-022-01798-4
References
Ahmed, S., Cabral, F.G., Costa, B.F.P.D.: Stochastic lipschitz dynamic programming (2019)
Bao, H., Zhou, Z., Kotsalis, G., Lan, G., Tong, Z.: Lignin valorization process control under feedstock uncertainty through a dynamic stochastic programming approach. React. Chem. Eng. 4, 1740–1747 (2019)
Baucke, R., Downward, A., Zakeri, G.: A deterministic algorithm for solving multistage stochastic programming problems. Technical report, The University of Auckland, 70 Symonds Street, Grafton, Auckland, July 2017 (2017)
Birge, J.R.: Decomposition and partitioning methods for multistage stochastic linear programs. Oper. Res. 33(5), 989–1007 (1985)
Birge, J.R., Louveaux, F.V.: Introduction to Stochastic Programming. Springer, New York (1997)
Donohue, C.J., Birge, J.R.: The abridged nested decomposition method for multistage stochastic linear programs with relatively complete recourse. Algorithm. Oper. Res. 1(1), 20 (2006)
Georghiou, A., Tsoukalas, A., Wiesemann, W.: Robust dual dynamic programming. Oper. Res. 67(3), 813–830 (2019)
Girardeau, P., Leclere, V., Philpott, A.B.: On the convergence of decomposition methods for multistage stochastic convex programs. Math. Oper. Res. 40, 130–145 (2015)
Guigues, V.: Sddp for some interstage dependent risk-averse problems and application to hydro-thermal planning. Comput. Optim. Appl. 57, 167–203 (2014)
Guigues, V.: Inexact cuts in deterministic and stochastic dual dynamic programming applied to linear optimization problems (2018)
Higle, J.L., Sen, S.: Stochastic decomposition: an algorithm for two-stage linear programs with recourse. Math. Oper. Res. 16, 650–669 (1991)
Hindsberger, M., Philpott, A.B.: Resa: a method for solving multistage stochastic linear programs. J. Appl. Oper. Res. 6(1), 2–15 (2014)
Kelley, J.E.: The cutting plane method for solving convex programs. J. SIAM 8, 703–712 (1960)
Kozmík, V., Morton, D.P.: Evaluating policies in risk-averse multi-stage stochastic programming. Math. Program. 152(1–2), 275–300 (2015)
Lan, G.: First-Order and Stochastic Optimization Methods for Machine Learning. Springer, Basel (2020)
Lan, G., Nemirovski, A.S., Shapiro, A.: Validation analysis of mirror descent stochastic approximation method. Math. Program. 134, 425–458 (2012)
Lan, G., Zhou, Z.: Dynamic stochastic approximation for multi-stage stochastic optimization. Manuscript, Georgia Institute of Technology, 2017. Mathematical Programming, under minor revision (2017)
Leclère, V., Carpentier, P., Chancelier, J.P., Lenoir, A., Pacaud, F.: Exact converging bounds for stochastic dual dynamic programming via fenchel duality. SIAM J. Optim. 30(2), 1223–1250 (2020)
Linowsky, K., Philpott, A.B.: On the convergence of sampling-based decomposition algorithms for multistage stochastic programs. J. Optim. Theory Appl. 125, 349–366 (2005)
Nesterov, Y.E.: Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, Norwell, MA (2004)
Pereira, M., Pinto, L.: Multi-stage stochastic optimization applied to energy planning. Math. Program. 52(1–3), 359–375 (1991)
Philpott, A., Matos, Vd, Finardi, E.: On solving multistage stochastic programs with coherent risk measures. Oper. Res. 61, 957–970 (2013)
Philpott, A., Wahid, F., Bonnans, F.: Midas: A mixed integer dynamic approximation scheme, 2016. PhD thesis, Inria Saclay Ile de France (2016)
Tyrrell Rockafellar, R., Wets, Roger J.-B.: Scenarios and policy aggregation in optimization under uncertainty. Math. Oper. Res. 16(1), 119–147 (1991)
Ruszczyński, A.: Decomposition methods. In: Ruszczyński, A., Shapiro, A. (eds.) Stochastic Programming, pp. 141–211. Elsevier, Amsterdam (2003)
Shapiro, A.: On complexity of multistage stochastic programs. Oper. Res. Lett. 34, 1–8 (2006)
Shapiro, A.: Analysis of stochastic dual dynamic programming method. Eur. J. Oper. Res. 209, 63–72 (2011)
Shapiro, A., Dentcheva, D., Ruszczyński, A.: Lectures on Stochastic Programming: Modeling and Theory. SIAM, Philadelphia (2009)
Shapiro, A., Nemirovski, A.: On complexity of stochastic programming problems. E-print available at: http://www.optimization-online.org (2004)
Zou, J., Ahmed, S., Sun, X.A.: Stochastic dual dynamic integer programming. Math. Program. 175(1–2), 461–502 (2019)
Author information
Authors and Affiliations
Corresponding author
Additional information
Dedicated to Professor Alexander Shapiro on the occasion of his 70th birthday for his profound contributions to stochastic optimization.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research was partially supported by the NSF grant 1953199 and NIFA Grant 2020-67021-31526.
Rights and permissions
About this article
Cite this article
Lan, G. Complexity of stochastic dual dynamic programming. Math. Program. 191, 717–754 (2022). https://doi.org/10.1007/s10107-020-01567-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-020-01567-1