Machine Learning

, Volume 85, Issue 3, pp 361–391

An asymptotically optimal policy for finite support models in the multiarmed bandit problem


DOI: 10.1007/s10994-011-5257-4

Cite this article as:
Honda, J. & Takemura, A. Mach Learn (2011) 85: 361. doi:10.1007/s10994-011-5257-4


In the multiarmed bandit problem the dilemma between exploration and exploitation in reinforcement learning is expressed as a model of a gambler playing a slot machine with multiple arms. A policy chooses an arm in each round so as to minimize the number of times that arms with suboptimal expected rewards are pulled. We propose the minimum empirical divergence (MED) policy and derive an upper bound on the finite-time regret which meets the asymptotic bound for the case of finite support models. In a setting similar to ours, Burnetas and Katehakis have already proposed an asymptotically optimal policy. However, we do not assume any knowledge of the support except for its upper and lower bounds. Furthermore, the criterion for choosing an arm, minimum empirical divergence, can be computed easily by a convex optimization technique. We confirm by simulations that the MED policy demonstrates good performance in finite time in comparison to other currently popular policies.


Bandit problems Finite-time regret MED policy Convex optimization 
Download to read the full article text

Copyright information

© The Author(s) 2011

Authors and Affiliations

  1. 1.Graduate School of Frontier SciencesThe University of TokyoKashiwa-shi ChibaJapan
  2. 2.Graduate School of Information Science and TechnologyThe University of TokyoBunkyo-ku TokyoJapan

Personalised recommendations