Skip to main content
Book cover

International Conference on Algorithmic Learning Theory

ALT 2013: Algorithmic Learning Theory pp 218–233Cite as

Robust Risk-Averse Stochastic Multi-armed Bandits

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 8139)

Abstract

We study a variant of the standard stochastic multi-armed bandit problem when one is not interested in the arm with the best mean, but instead in the arm maximizing some coherent risk measure criterion. Further, we are studying the deviations of the regret instead of the less informative expected regret. We provide an algorithm, called RA-UCB to solve this problem, together with a high probability bound on its regret.

Keywords

  • Multi-armed bandits
  • coherent risk measure
  • cumulant generative function
  • concentration of measure

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-40935-6_16
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   39.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-40935-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   54.99
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahmadi-Javid, A.: Entropic value-at-risk: A new coherent risk measure. Journal of Optimization Theory and Applications 155(3), 1105–1123 (2012)

    CrossRef  MathSciNet  MATH  Google Scholar 

  2. Artzner, P., Delbaen, F., Eber, J.-M., Heath, D., Ku, H.: Coherent multiperiod risk adjusted values and bellman’s principle. Annals of Operations Research 152(1), 5–22 (2007)

    CrossRef  MathSciNet  MATH  Google Scholar 

  3. Audibert, J.-Y., Munos, R., Szepesvári, C.: Exploration-exploitation trade-off using variance estimates in multi-armed bandits. Theoretical Computer Science 410(19), 1876–1902 (2009)

    CrossRef  MathSciNet  MATH  Google Scholar 

  4. Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Machine Learning 47(2-3), 235–256 (2002)

    CrossRef  MATH  Google Scholar 

  5. Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The nonstochastic multiarmed bandit problem. SIAM Journal on Computing 32, 48–77 (2003)

    CrossRef  MathSciNet  Google Scholar 

  6. Borwein, J., Lewis, A.: Duality relationships for entropy-like minimization problem. SIAM Journal on Computation and Optimization 29(2), 325–338 (1991)

    CrossRef  MathSciNet  MATH  Google Scholar 

  7. Burnetas, A., Katehakis, M.: Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics 17(2), 122–142 (1996)

    CrossRef  MathSciNet  MATH  Google Scholar 

  8. Cappé, O., Garivier, A., Maillard, O.-A., Munos, R., Stoltz, G.: Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics (2013)

    Google Scholar 

  9. Cover, T., Thomas, J.: Elements of Information Theory. John Wiley (1991)

    Google Scholar 

  10. Defourny, B., Ernst, D., Wehenkel, L.: Risk-aware decision making and dynamic programming. In: NIPS Workshop on Model Uncertainty and Risk in RL (2008)

    Google Scholar 

  11. Dembo, A., Zeitouni, O.: Large Deviations Techniques and Applications, 2nd edn. Springer (1998)

    Google Scholar 

  12. Denardo, E., Rothblum, U.: Optimal stopping, exponential utility and linear programming. Mathematical Programming 16, 228–244 (1979)

    CrossRef  MathSciNet  Google Scholar 

  13. Even-Dar, E., Kearns, M., Wortman, J.: Risk-sensitive online learning. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds.) ALT 2006. LNCS (LNAI), vol. 4264, pp. 199–213. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  14. Garivier, A., Cappé, O.: The KL-UCB algorithm for bounded stochastic bandits and beyond. In: Proceedings of the 24th Annual Conference on Learning Theory (2011)

    Google Scholar 

  15. Harari-Kermadec, H.: Vraisemblance empirique généralisée et estimation semi-paramétrique. PhD thesis, Université Paris–Ouest (December 2006)

    Google Scholar 

  16. Honda, J., Takemura, A.: An asymptotically optimal bandit algorithm for bounded support models. In: Proceedings of the 23rd Annual Conference on Learning Theory, Haifa, Israel (2010)

    Google Scholar 

  17. Honda, J., Takemura, A.: An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning 85, 361–391 (2011)

    CrossRef  MathSciNet  MATH  Google Scholar 

  18. Honda, J., Takemura, A.: Finite-time regret bound of a bandit algorithm for the semi-bounded support model. arXiv:1202.2277 (2012)

    Google Scholar 

  19. Howard, R.A., Matheson, J.E.: Risk-sensitive markov decision processes. Management Science 18, 356–369 (1972)

    CrossRef  MathSciNet  MATH  Google Scholar 

  20. Kaufmann, E., Korda, N., Munos, R.: Thompson sampling: An asymptotically optimal finite-time analysis. In: Bshouty, N.H., Stoltz, G., Vayatis, N., Zeugmann, T. (eds.) ALT 2012. LNCS (LNAI), vol. 7568, pp. 199–213. Springer, Heidelberg (2012)

    CrossRef  Google Scholar 

  21. Lai, T.L., Robbins, H.: Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics 6, 4–22 (1985)

    CrossRef  MathSciNet  MATH  Google Scholar 

  22. Liu, Y., Koenig, S.: An exact algorithm for solving mdps under risk-sensitive planning objectives with one-switch utility functions. In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS 2008, Richland, SC, vol. 1, pp. 453–460. International Foundation for Autonomous Agents and Multiagent Systems (2008)

    Google Scholar 

  23. Maillard, O.-A.: Robust risk-averse stochastic multi-armed bandits. Technical Report HAL-INRIA open archive (2013), http://hal.inria.fr/hal-00821670

  24. Maillard, O.-A., Munos, R., Stoltz, G.: A finite-time analysis of multi-armed bandits problems with Kullback-Leibler divergences. In: Proceedings of the 23rd Annual Conference on Learning Theory, Budapest, Hungary (2011)

    Google Scholar 

  25. Markowitz, H.: Portfolio selection. The Journal of Finance 7(1), 77–91 (1952)

    Google Scholar 

  26. Patek, S.D.: On terminating markov decision processes with a risk-averse objective function. Automatica 37(9), 1379–1386 (2001)

    CrossRef  MATH  Google Scholar 

  27. Robbins, H.: Some aspects of the sequential design of experiments. Bulletin of the American Mathematics Society 58, 527–535 (1952)

    CrossRef  MathSciNet  MATH  Google Scholar 

  28. Rockafellar, R.T.: Coherent approaches to risk in optimization under uncertainty. Tutorials in Operation Research, 38–61 (2007)

    Google Scholar 

  29. Salomon, A., Audibert, J.-Y.: Robustness of Anytime Bandit Policies (2011), http://hal.archives-ouvertes.fr/hal-00579607

  30. Sani, A., Lazaric, A., Munos, R.: Risk-aversion in multi-armed bandits. In: Proceedings of Advancezs in Neural Information Processing System (2012)

    Google Scholar 

  31. Thompson, W.: On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294 (1933)

    MATH  Google Scholar 

  32. Thompson, W.: On the theory of apportionment. American Journal of Mathematics 57, 450–456 (1935)

    CrossRef  MathSciNet  Google Scholar 

  33. von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior, Princeton Classic Editions. Princeton University Press (1947)

    Google Scholar 

  34. Warmuth, M.K., Kuzmin, D.: Online variance minimization. In: Lugosi, G., Simon, H.U. (eds.) COLT 2006. LNCS (LNAI), vol. 4005, pp. 514–528. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, Technion, Haifa, Israel

    Odalric-Ambrym Maillard

Authors
  1. Odalric-Ambrym Maillard
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. National University of Singapore, Republic of Singapore

    Sanjay Jain & Frank Stephan & 

  2. Inria Lille - Nord Europe, Villeneuve d’Ascq, France

    Rémi Munos

  3. Hokkaido University, Sapporo, Japan

    Thomas Zeugmann

Rights and permissions

Reprints and Permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Maillard, OA. (2013). Robust Risk-Averse Stochastic Multi-armed Bandits. In: Jain, S., Munos, R., Stephan, F., Zeugmann, T. (eds) Algorithmic Learning Theory. ALT 2013. Lecture Notes in Computer Science(), vol 8139. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40935-6_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-40935-6_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-40934-9

  • Online ISBN: 978-3-642-40935-6

  • eBook Packages: Computer ScienceComputer Science (R0)