Robust Risk-Averse Stochastic Multi-armed Bandits

  • Odalric-Ambrym Maillard
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8139)


We study a variant of the standard stochastic multi-armed bandit problem when one is not interested in the arm with the best mean, but instead in the arm maximizing some coherent risk measure criterion. Further, we are studying the deviations of the regret instead of the less informative expected regret. We provide an algorithm, called RA-UCB to solve this problem, together with a high probability bound on its regret.


Multi-armed bandits coherent risk measure cumulant generative function concentration of measure 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahmadi-Javid, A.: Entropic value-at-risk: A new coherent risk measure. Journal of Optimization Theory and Applications 155(3), 1105–1123 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Artzner, P., Delbaen, F., Eber, J.-M., Heath, D., Ku, H.: Coherent multiperiod risk adjusted values and bellman’s principle. Annals of Operations Research 152(1), 5–22 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Audibert, J.-Y., Munos, R., Szepesvári, C.: Exploration-exploitation trade-off using variance estimates in multi-armed bandits. Theoretical Computer Science 410(19), 1876–1902 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3), 235–256 (2002)CrossRefzbMATHGoogle Scholar
  5. 5.
    Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The nonstochastic multiarmed bandit problem. SIAM Journal on Computing 32, 48–77 (2003)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Borwein, J., Lewis, A.: Duality relationships for entropy-like minimization problem. SIAM Journal on Computation and Optimization 29(2), 325–338 (1991)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Burnetas, A., Katehakis, M.: Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics 17(2), 122–142 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cappé, O., Garivier, A., Maillard, O.-A., Munos, R., Stoltz, G.: Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics (2013)Google Scholar
  9. 9.
    Cover, T., Thomas, J.: Elements of Information Theory. John Wiley (1991)Google Scholar
  10. 10.
    Defourny, B., Ernst, D., Wehenkel, L.: Risk-aware decision making and dynamic programming. In: NIPS Workshop on Model Uncertainty and Risk in RL (2008)Google Scholar
  11. 11.
    Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications, 2nd edn. Springer (1998)Google Scholar
  12. 12.
    Denardo, E., Rothblum, U.: Optimal stopping, exponential utility and linear programming. Mathematical Programming 16, 228–244 (1979)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Even-Dar, E., Kearns, M., Wortman, J.: Risk-sensitive online learning. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds.) ALT 2006. LNCS (LNAI), vol. 4264, pp. 199–213. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Garivier, A., Cappé, O.: The KL-UCB algorithm for bounded stochastic bandits and beyond. In: Proceedings of the 24th Annual Conference on Learning Theory (2011)Google Scholar
  15. 15.
    Harari-Kermadec, H.: Vraisemblance empirique généralisée et estimation semi-paramétrique. PhD thesis, Université Paris–Ouest (December 2006)Google Scholar
  16. 16.
    Honda, J., Takemura, A.: An asymptotically optimal bandit algorithm for bounded support models. In: Proceedings of the 23rd Annual Conference on Learning Theory, Haifa, Israel (2010)Google Scholar
  17. 17.
    Honda, J., Takemura, A.: An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning 85, 361–391 (2011)CrossRefMathSciNetzbMATHGoogle Scholar
  18. 18.
    Honda, J., Takemura, A.: Finite-time regret bound of a bandit algorithm for the semi-bounded support model. arXiv:1202.2277 (2012)Google Scholar
  19. 19.
    Howard, R.A., Matheson, J.E.: Risk-sensitive markov decision processes. Management Science 18, 356–369 (1972)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Kaufmann, E., Korda, N., Munos, R.: Thompson sampling: An asymptotically optimal finite-time analysis. In: Bshouty, N.H., Stoltz, G., Vayatis, N., Zeugmann, T. (eds.) ALT 2012. LNCS (LNAI), vol. 7568, pp. 199–213. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  21. 21.
    Lai, T.L., Robbins, H.: Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6, 4–22 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Liu, Y., Koenig, S.: An exact algorithm for solving mdps under risk-sensitive planning objectives with one-switch utility functions. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2008, Richland, SC, vol. 1, pp. 453–460. International Foundation for Autonomous Agents and Multiagent Systems (2008)Google Scholar
  23. 23.
    Maillard, O.-A.: Robust risk-averse stochastic multi-armed bandits. Technical Report HAL-INRIA open archive (2013),
  24. 24.
    Maillard, O.-A., Munos, R., Stoltz, G.: A finite-time analysis of multi-armed bandits problems with Kullback-Leibler divergences. In: Proceedings of the 23rd Annual Conference on Learning Theory, Budapest, Hungary (2011)Google Scholar
  25. 25.
    Markowitz, H.: Portfolio selection. The Journal of Finance 7(1), 77–91 (1952)Google Scholar
  26. 26.
    Patek, S.D.: On terminating markov decision processes with a risk-averse objective function. Automatica 37(9), 1379–1386 (2001)CrossRefzbMATHGoogle Scholar
  27. 27.
    Robbins, H.: Some aspects of the sequential design of experiments. Bulletin of the American Mathematics Society 58, 527–535 (1952)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Rockafellar, R.T.: Coherent approaches to risk in optimization under uncertainty. Tutorials in Operation Research, 38–61 (2007)Google Scholar
  29. 29.
    Salomon, A., Audibert, J.-Y.: Robustness of Anytime Bandit Policies (2011),
  30. 30.
    Sani, A., Lazaric, A., Munos, R.: Risk-aversion in multi-armed bandits. In: Proceedings of Advancezs in Neural Information Processing System (2012)Google Scholar
  31. 31.
    Thompson, W.: On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294 (1933)zbMATHGoogle Scholar
  32. 32.
    Thompson, W.: On the theory of apportionment. American Journal of Mathematics 57, 450–456 (1935)MathSciNetCrossRefGoogle Scholar
  33. 33.
    von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior, Princeton Classic Editions. Princeton University Press (1947)Google Scholar
  34. 34.
    Warmuth, M.K., Kuzmin, D.: Online variance minimization. In: Lugosi, G., Simon, H.U. (eds.) COLT 2006. LNCS (LNAI), vol. 4005, pp. 514–528. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Odalric-Ambrym Maillard
    • 1
  1. 1.Faculty of Electrical EngineeringTechnionHaifaIsrael

Personalised recommendations