Machine Learning

, Volume 85, Issue 3, pp 361–391

An asymptotically optimal policy for finite support models in the multiarmed bandit problem

Authors

    • Graduate School of Frontier SciencesThe University of Tokyo
  • Akimichi Takemura
    • Graduate School of Information Science and TechnologyThe University of Tokyo
Article

DOI: 10.1007/s10994-011-5257-4

Cite this article as:
Honda, J. & Takemura, A. Mach Learn (2011) 85: 361. doi:10.1007/s10994-011-5257-4

Abstract

In the multiarmed bandit problem the dilemma between exploration and exploitation in reinforcement learning is expressed as a model of a gambler playing a slot machine with multiple arms. A policy chooses an arm in each round so as to minimize the number of times that arms with suboptimal expected rewards are pulled. We propose the minimum empirical divergence (MED) policy and derive an upper bound on the finite-time regret which meets the asymptotic bound for the case of finite support models. In a setting similar to ours, Burnetas and Katehakis have already proposed an asymptotically optimal policy. However, we do not assume any knowledge of the support except for its upper and lower bounds. Furthermore, the criterion for choosing an arm, minimum empirical divergence, can be computed easily by a convex optimization technique. We confirm by simulations that the MED policy demonstrates good performance in finite time in comparison to other currently popular policies.

Keywords

Bandit problems Finite-time regret MED policy Convex optimization

Copyright information

© The Author(s) 2011