Navigating and Stopping Multi-Parameter Bandit Processes

Abstract

In recent years, considerable effort has been devoted to the development of a theory for multi-parameter processes. These are stochastic processes that evolve in “time” which is only partially ordered. The multi-parameter theory provides a natural way to formulate problems in dynamic allocation of resources, including discrete and continuous time multi-armed bandits as special cases. Multi-parameter processes that describe a game played by a gambler against a multi-armed bandit are called bandit processes. My talk will focus on two control problems for bandit processes. The first problem, the optimal stopping problem, is that of a gambler who can stop playing at any time. The reward from the game depends only on the state of affairs at the time of stopping, and the gambler’s problem is to choose an optimal stopping time. In the second problem, the optimal navigation problem, the gambler plays forever and seeks to maximize total discounted reward over an infinite horizon.

Research partially supported by NSF grant ECS 8603857.

A substantial part of this manuscript is based on [8,9] which have been published, and on [10] which, hopefully, will be published.

AMS 1980 subject classification. Primary 62L99, 60G40, 93E20; Secondary 60J60, 60K10, 60G17, 60J55.