Graduate School of Frontier SciencesThe University of Tokyo

Akimichi Takemura

Graduate School of Information Science and TechnologyThe University of Tokyo

Article

First Online:

Received:

Accepted:

DOI:
10.1007/s10994-011-5257-4

Cite this article as:

Honda, J. & Takemura, A. Mach Learn (2011) 85: 361. doi:10.1007/s10994-011-5257-4

Abstract

In the multiarmed bandit problem the dilemma between exploration and exploitation in reinforcement learning is expressed as a model of a gambler playing a slot machine with multiple arms. A policy chooses an arm in each round so as to minimize the number of times that arms with suboptimal expected rewards are pulled. We propose the minimum empirical divergence (MED) policy and derive an upper bound on the finite-time regret which meets the asymptotic bound for the case of finite support models. In a setting similar to ours, Burnetas and Katehakis have already proposed an asymptotically optimal policy. However, we do not assume any knowledge of the support except for its upper and lower bounds. Furthermore, the criterion for choosing an arm, minimum empirical divergence, can be computed easily by a convex optimization technique. We confirm by simulations that the MED policy demonstrates good performance in finite time in comparison to other currently popular policies.