Skip to main content
SpringerLink
Log in

Robust Risk-Averse Stochastic Multi-armed Bandits

  • Conference paper
Book cover Algorithmic Learning Theory (ALT 2013)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 8139))

Included in the following conference series:

Abstract

We study a variant of the standard stochastic multi-armed bandit problem when one is not interested in the arm with the best mean, but instead in the arm maximizing some coherent risk measure criterion. Further, we are studying the deviations of the regret instead of the less informative expected regret. We provide an algorithm, called RA-UCB to solve this problem, together with a high probability bound on its regret.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahmadi-Javid, A.: Entropic value-at-risk: A new coherent risk measure. Journal of Optimization Theory and Applications 155(3), 1105–1123 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  2. Artzner, P., Delbaen, F., Eber, J.-M., Heath, D., Ku, H.: Coherent multiperiod risk adjusted values and bellman’s principle. Annals of Operations Research 152(1), 5–22 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  3. Audibert, J.-Y., Munos, R., Szepesvári, C.: Exploration-exploitation trade-off using variance estimates in multi-armed bandits. Theoretical Computer Science 410(19), 1876–1902 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  4. Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3), 235–256 (2002)

    Article  MATH  Google Scholar 

  5. Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The nonstochastic multiarmed bandit problem. SIAM Journal on Computing 32, 48–77 (2003)

    Article  MathSciNet  Google Scholar 

  6. Borwein, J., Lewis, A.: Duality relationships for entropy-like minimization problem. SIAM Journal on Computation and Optimization 29(2), 325–338 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  7. Burnetas, A., Katehakis, M.: Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics 17(2), 122–142 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cappé, O., Garivier, A., Maillard, O.-A., Munos, R., Stoltz, G.: Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics (2013)

    Google Scholar 

  9. Cover, T., Thomas, J.: Elements of Information Theory. John Wiley (1991)

    Google Scholar 

  10. Defourny, B., Ernst, D., Wehenkel, L.: Risk-aware decision making and dynamic programming. In: NIPS Workshop on Model Uncertainty and Risk in RL (2008)

    Google Scholar 

  11. Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications, 2nd edn. Springer (1998)

    Google Scholar 

  12. Denardo, E., Rothblum, U.: Optimal stopping, exponential utility and linear programming. Mathematical Programming 16, 228–244 (1979)

    Article  MathSciNet  Google Scholar 

  13. Even-Dar, E., Kearns, M., Wortman, J.: Risk-sensitive online learning. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds.) ALT 2006. LNCS (LNAI), vol. 4264, pp. 199–213. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  14. Garivier, A., Cappé, O.: The KL-UCB algorithm for bounded stochastic bandits and beyond. In: Proceedings of the 24th Annual Conference on Learning Theory (2011)

    Google Scholar 

  15. Harari-Kermadec, H.: Vraisemblance empirique généralisée et estimation semi-paramétrique. PhD thesis, Université Paris–Ouest (December 2006)

    Google Scholar 

  16. Honda, J., Takemura, A.: An asymptotically optimal bandit algorithm for bounded support models. In: Proceedings of the 23rd Annual Conference on Learning Theory, Haifa, Israel (2010)

    Google Scholar 

  17. Honda, J., Takemura, A.: An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning 85, 361–391 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  18. Honda, J., Takemura, A.: Finite-time regret bound of a bandit algorithm for the semi-bounded support model. arXiv:1202.2277 (2012)

    Google Scholar 

  19. Howard, R.A., Matheson, J.E.: Risk-sensitive markov decision processes. Management Science 18, 356–369 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  20. Kaufmann, E., Korda, N., Munos, R.: Thompson sampling: An asymptotically optimal finite-time analysis. In: Bshouty, N.H., Stoltz, G., Vayatis, N., Zeugmann, T. (eds.) ALT 2012. LNCS (LNAI), vol. 7568, pp. 199–213. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  21. Lai, T.L., Robbins, H.: Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6, 4–22 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  22. Liu, Y., Koenig, S.: An exact algorithm for solving mdps under risk-sensitive planning objectives with one-switch utility functions. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2008, Richland, SC, vol. 1, pp. 453–460. International Foundation for Autonomous Agents and Multiagent Systems (2008)

    Google Scholar 

  23. Maillard, O.-A.: Robust risk-averse stochastic multi-armed bandits. Technical Report HAL-INRIA open archive (2013), http://hal.inria.fr/hal-00821670

  24. Maillard, O.-A., Munos, R., Stoltz, G.: A finite-time analysis of multi-armed bandits problems with Kullback-Leibler divergences. In: Proceedings of the 23rd Annual Conference on Learning Theory, Budapest, Hungary (2011)

    Google Scholar 

  25. Markowitz, H.: Portfolio selection. The Journal of Finance 7(1), 77–91 (1952)

    Google Scholar 

  26. Patek, S.D.: On terminating markov decision processes with a risk-averse objective function. Automatica 37(9), 1379–1386 (2001)

    Article  MATH  Google Scholar 

  27. Robbins, H.: Some aspects of the sequential design of experiments. Bulletin of the American Mathematics Society 58, 527–535 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  28. Rockafellar, R.T.: Coherent approaches to risk in optimization under uncertainty. Tutorials in Operation Research, 38–61 (2007)

    Google Scholar 

  29. Salomon, A., Audibert, J.-Y.: Robustness of Anytime Bandit Policies (2011), http://hal.archives-ouvertes.fr/hal-00579607

  30. Sani, A., Lazaric, A., Munos, R.: Risk-aversion in multi-armed bandits. In: Proceedings of Advancezs in Neural Information Processing System (2012)

    Google Scholar 

  31. Thompson, W.: On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294 (1933)

    MATH  Google Scholar 

  32. Thompson, W.: On the theory of apportionment. American Journal of Mathematics 57, 450–456 (1935)

    Article  MathSciNet  Google Scholar 

  33. von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior, Princeton Classic Editions. Princeton University Press (1947)

    Google Scholar 

  34. Warmuth, M.K., Kuzmin, D.: Online variance minimization. In: Lugosi, G., Simon, H.U. (eds.) COLT 2006. LNCS (LNAI), vol. 4005, pp. 514–528. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, Technion, Haifa, Israel

    Odalric-Ambrym Maillard

Authors
  1. Odalric-Ambrym Maillard
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. National University of Singapore, Republic of Singapore

    Sanjay Jain  & Frank Stephan  & 

  2. Inria Lille - Nord Europe, Villeneuve d’Ascq, France

    Rémi Munos

  3. Hokkaido University, Sapporo, Japan

    Thomas Zeugmann

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Maillard, OA. (2013). Robust Risk-Averse Stochastic Multi-armed Bandits. In: Jain, S., Munos, R., Stephan, F., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2013. Lecture Notes in Computer Science(), vol 8139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40935-6_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-40935-6_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-40934-9

  • Online ISBN: 978-3-642-40935-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation