The Role of Dynamic Allocation Indices in the Evaluation of Suboptimal Strategies for Families of Bandit Processes

Abstract

A family of N alternative bandit processes {(Ω_j, P_j, C_j, α); j = l, 2,...N} is a cost-discounted Markov decision process with the following special features:

• (a) Its state at time t∈ℕ is x(t)={x₁(t), x₂(t),..., x_N(t)} where x_j(t)∈Ω_j, 1 ≤ j ≤ N. State space Ω_j may be finite, countable or continuous.

• (b) The action space A is {a₁, a₂,...,a_N}. Action a_j denotes the choice of bandit process j. An action is taken at each time t∈ℕ.

• (c) If action a_j is taken at time t∈ℕ only the j^th component of x(t) changes. Hence x_i(t+1)=x_i(t), i ≠ j, and x_j(t+1) is determined according to a probabilistic law of motion P_j{x_j(t)}.

• (d) The transition of the process under action a_j described in (c) incurs a cost α^t C_j{x_j(t), x_j(t+1)} where 0≤α≤1. The costs are assumed to be bounded.

• (e) An optimal strategy is a rule for choosing actions which minimises the total expected cost.