Policy Gradients for CVaR-Constrained MDPs

  • L. A. Prashanth
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8776)


We study a risk-constrained version of the stochastic shortest path (SSP) problem, where the risk measure considered is Conditional Value-at-Risk (CVaR). We propose two algorithms that obtain a locally risk-optimal policy by employing four tools: stochastic approximation, mini batches, policy gradients and importance sampling. Both the algorithms incorporate a CVaR estimation procedure, along the lines of [3], which in turn is based on Rockafellar-Uryasev’s representation for CVaR and utilize the likelihood ratio principle for estimating the gradient of the sum of one cost function (objective of the SSP) and the gradient of the CVaR of the sum of another cost function (constraint of the SSP). The algorithms differ in the manner in which they approximate the CVaR estimates/necessary gradients - the first algorithm uses stochastic approximation, while the second employs mini-batches in the spirit of Monte Carlo methods. We establish asymptotic convergence of both the algorithms. Further, since estimating CVaR is related to rare-event simulation, we incorporate an importance sampling based variance reduction scheme into our proposed algorithms.


Risk Measure Importance Sampling Stochastic Approximation Coherent Risk Measure Policy Parameter 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Artzner, P., Delbaen, F., Eber, J.M., Heath, D.: Coherent measures of risk. Mathematical Finance 9(3), 203–228 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Atchade, Y.F., Fort, G., Moulines, E.: On stochastic proximal gradient algorithms. arXiv preprint arXiv:1402.2365 (2014)Google Scholar
  3. 3.
    Bardou, O., Frikha, N., Pages, G.: Computing VaR and CVaR using stochastic approximation and adaptive unconstrained importance sampling. Monte Carlo Methods and Applications 15(3), 173–210 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Bartlett, P.L., Baxter, J.: Infinite-horizon policy-gradient estimation. arXiv preprint arXiv:1106.0665 (2011)Google Scholar
  5. 5.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control, 3rd edn., vol. II. Athena Scientific (2007)Google Scholar
  6. 6.
    Borkar, V.: An actor-critic algorithm for constrained Markov decision processes. Systems & Control Letters 54(3), 207–213 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Borkar, V.: Stochastic approximation: a dynamical systems viewpoint. Cambridge University Press (2008)Google Scholar
  8. 8.
    Borkar, V., Jain, R.: Risk-constrained Markov decision processes. In: IEEE Conference on Decision and Control (CDC), pp. 2664–2669 (2010)Google Scholar
  9. 9.
    Glynn, P.W.: Likelilood ratio gradient estimation: an overview. In: Proceedings of the 19th Conference on Winter Simulation, pp. 366–375. ACM (1987)Google Scholar
  10. 10.
    Kushner, H., Clark, D.: Stochastic approximation methods for constrained and unconstrained systems. Springer (1978)Google Scholar
  11. 11.
    Lemaire, V., Pages, G.: Unconstrained recursive importance sampling. The Annals of Applied Probability 20(3), 1029–1067 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Mannor, S., Tsitsiklis, J.: Mean-variance optimization in Markov decision processes. arXiv preprint arXiv:1104.5601 (2011)Google Scholar
  13. 13.
    Mas-Colell, A., Whinston, M., Green, J.: Microeconomic theory. Oxford University Press (1995)Google Scholar
  14. 14.
    Prashanth, L.A., Ghavamzadeh, M.: Actor-critic algorithms for risk-sensitive MDPs. In: Neural Information Processing Systems 26, pp. 252–260 (2013)Google Scholar
  15. 15.
    Robbins, H., Monro, S.: A stochastic approximation method. The Annals of Mathematical Statistics, 400–407 (1951)Google Scholar
  16. 16.
    Rockafellar, R.T., Uryasev, S.: Optimization of conditional value-at-risk. Journal of Risk 2, 21–42 (2000)Google Scholar
  17. 17.
    Sutton, R., Barto, A.: Reinforcement learning: An introduction. MIT Press (1998)Google Scholar
  18. 18.
    Tamar, A., Di Castro, D., Mannor, S.: Policy gradients with variance related risk criteria. In: International Conference on Machine Learning, pp. 387–396 (2012)Google Scholar
  19. 19.
    Tamar, A., Mannor, S.: Variance Adjusted Actor-Critic Algorithms. arXiv preprint arXiv:1310.3697 (2013)Google Scholar
  20. 20.
    Tamar, A., Glassner, Y., Mannor, S.: Policy Gradients Beyond Expectations: Conditional Value-at-Risk. arXiv preprint arXiv:1404.3862 (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • L. A. Prashanth
    • 1
  1. 1.INRIA Lille - Nord Europe, Team SequeLFrance

Personalised recommendations