IFIP International Conference on Artificial Intelligence Applications and Innovations

AIAI 2005: Artificial Intelligence Applications and Innovations pp 765-775

Solution of Mdps Using Simulation-Based Value Iteration

  • Mohammed Shahid Abdulla
  • Shalabh Bhatnagar
Conference paper

DOI: 10.1007/0-387-29295-0_83

Volume 187 of the book series IFIP — The International Federation for Information Processing (IFIPAICT)

Abstract

This article proposes a three-timescale simulation based algorithm for solution of infinite horizon Markov Decision Processes (MDPs). We assume a finite state space and discounted cost criterion and adopt the value iteration approach. An approximation of the Dynamic Programming operator T is applied to the value function iterates. This ‘approximate’ operator is implemented using three timescales, the slowest of which updates the value function iterates. On the middle timescale we perform a gradient search over the feasible action set of each state using Simultaneous Perturbation Stochastic Approximation (SPSA) gradient estimates, thus finding the minimizing action in T. On the fastest timescale, the ‘critic’ estimates, over which the gradient search is performed, are obtained. A sketch of convergence explaining the dynamics of the algorithm using associated ODEs is also presented. Numerical experiments on rate based flow control on a bottleneck node using a continuous-time queueing model are performed using the proposed algorithm. The results obtained are verified against classical value iteration where the feasible set is suitably discretized. Over such a discretized setting, a variant of the algorithm of missing data is compared and the proposed algorithm is found to converge faster.

Download to read the full conference paper text

Copyright information

© International Federation for Information Processing 2005

Authors and Affiliations

  • Mohammed Shahid Abdulla
    • 1
  • Shalabh Bhatnagar
    • 1
  1. 1.Department of Computer Science and AutomationIndian Institute of ScienceBangaloreIndia