Control of a Markov Chain with Unknown Dynamics and Cost Structure

Abstract

This chapter deals with the application of the “absolutely expedient” learning algorithms (developed in chapter 3) for the problem of control of a finite state Markov chain whose transition probabilities as a function of a finite number of control actions are unknown. At any instant of time depending on the state of the Markov chain and the control action chosen a reward is incurred. It is assumed that this reward is a two valued (binary) random variable whose distribution as a function of the state and the control action is unknown, but the sequence of states actually visited by the Markov chain is available. In other words we consider a Markov chain whose dynamics and reward structure are unknown but the state is observable exactly.