Machine Learning

, Volume 22, Issue 1, pp 59–94

Feature-based methods for large scale dynamic programming

Authors

  • John N. Tsitsiklis
    • Laboratory for Information and Decision SystemsMassachusetts Institute of Technology
  • Benjamin van Roy
    • Laboratory for Information and Decision SystemsMassachusetts Institute of Technology
Article

DOI: 10.1007/BF00114724

Cite this article as:
Tsitsiklis, J.N. & van Roy, B. Mach Learn (1996) 22: 59. doi:10.1007/BF00114724

Abstract

We develop a methodological framework and present a few different ways in which dynamic programming and compact representations can be combined to solve large scale stochastic control problems. In particular, we develop algorithms that employ two types of feature-based compact representations; that is, representations that involve feature extraction and a relatively simple approximation architecture. We prove the convergence of these algorithms and provide bounds on the approximation error. As an example, one of these algorithms is used to generate a strategy for the game of Tetris. Furthermore, we provide a counter-example illustrating the difficulties of integrating compact representations with dynamic programming, which exemplifies the shortcomings of certain simple approaches.

Keywords

Compact representation curse of dimensionality dynamic programming features function approximation neuro-dynamic programming reinforcement learning

Copyright information

© Kluwer Academic Publishers 1996