Abstract
We study a risk-constrained version of the stochastic shortest path (SSP) problem, where the risk measure considered is Conditional Value-at-Risk (CVaR). We propose two algorithms that obtain a locally risk-optimal policy by employing four tools: stochastic approximation, mini batches, policy gradients and importance sampling. Both the algorithms incorporate a CVaR estimation procedure, along the lines of [3], which in turn is based on Rockafellar-Uryasev’s representation for CVaR and utilize the likelihood ratio principle for estimating the gradient of the sum of one cost function (objective of the SSP) and the gradient of the CVaR of the sum of another cost function (constraint of the SSP). The algorithms differ in the manner in which they approximate the CVaR estimates/necessary gradients - the first algorithm uses stochastic approximation, while the second employs mini-batches in the spirit of Monte Carlo methods. We establish asymptotic convergence of both the algorithms. Further, since estimating CVaR is related to rare-event simulation, we incorporate an importance sampling based variance reduction scheme into our proposed algorithms.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Artzner, P., Delbaen, F., Eber, J.M., Heath, D.: Coherent measures of risk. Mathematical Finance 9(3), 203–228 (1999)
Atchade, Y.F., Fort, G., Moulines, E.: On stochastic proximal gradient algorithms. arXiv preprint arXiv:1402.2365 (2014)
Bardou, O., Frikha, N., Pages, G.: Computing VaR and CVaR using stochastic approximation and adaptive unconstrained importance sampling. Monte Carlo Methods and Applications 15(3), 173–210 (2009)
Bartlett, P.L., Baxter, J.: Infinite-horizon policy-gradient estimation. arXiv preprint arXiv:1106.0665 (2011)
Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. II. Athena Scientific (2007)
Borkar, V.: An actor-critic algorithm for constrained Markov decision processes. Systems & Control Letters 54(3), 207–213 (2005)
Borkar, V.: Stochastic approximation: a dynamical systems viewpoint. Cambridge University Press (2008)
Borkar, V., Jain, R.: Risk-constrained Markov decision processes. In: IEEE Conference on Decision and Control (CDC), pp. 2664–2669 (2010)
Glynn, P.W.: Likelilood ratio gradient estimation: an overview. In: Proceedings of the 19th Conference on Winter Simulation, pp. 366–375. ACM (1987)
Kushner, H., Clark, D.: Stochastic approximation methods for constrained and unconstrained systems. Springer (1978)
Lemaire, V., Pages, G.: Unconstrained recursive importance sampling. The Annals of Applied Probability 20(3), 1029–1067 (2010)
Mannor, S., Tsitsiklis, J.: Mean-variance optimization in Markov decision processes. arXiv preprint arXiv:1104.5601 (2011)
Mas-Colell, A., Whinston, M., Green, J.: Microeconomic theory. Oxford University Press (1995)
Prashanth, L.A., Ghavamzadeh, M.: Actor-critic algorithms for risk-sensitive MDPs. In: Neural Information Processing Systems 26, pp. 252–260 (2013)
Robbins, H., Monro, S.: A stochastic approximation method. The Annals of Mathematical Statistics, 400–407 (1951)
Rockafellar, R.T., Uryasev, S.: Optimization of conditional value-at-risk. Journal of Risk 2, 21–42 (2000)
Sutton, R., Barto, A.: Reinforcement learning: An introduction. MIT Press (1998)
Tamar, A., Di Castro, D., Mannor, S.: Policy gradients with variance related risk criteria. In: International Conference on Machine Learning, pp. 387–396 (2012)
Tamar, A., Mannor, S.: Variance Adjusted Actor-Critic Algorithms. arXiv preprint arXiv:1310.3697 (2013)
Tamar, A., Glassner, Y., Mannor, S.: Policy Gradients Beyond Expectations: Conditional Value-at-Risk. arXiv preprint arXiv:1404.3862 (2014)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Prashanth, L.A. (2014). Policy Gradients for CVaR-Constrained MDPs. In: Auer, P., Clark, A., Zeugmann, T., Zilles, S. (eds) Algorithmic Learning Theory. ALT 2014. Lecture Notes in Computer Science(), vol 8776. Springer, Cham. https://doi.org/10.1007/978-3-319-11662-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-11662-4_12
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-11661-7
Online ISBN: 978-3-319-11662-4
eBook Packages: Computer ScienceComputer Science (R0)