Discrete Event Dynamic Systems

, Volume 14, Issue 3, pp 309–341

Parallel Rollout for Online Solution of Partially Observable Markov Decision Processes

  • Hyeong Soo Chang
  • Robert Givan
  • Edwin K. P. Chong

DOI: 10.1023/B:DISC.0000028199.78776.c4

Cite this article as:
Chang, H.S., Givan, R. & Chong, E.K.P. Discrete Event Dynamic Systems (2004) 14: 309. doi:10.1023/B:DISC.0000028199.78776.c4


We propose a novel approach, called parallel rollout, to solving (partially observable) Markov decision processes. Our approach generalizes the rollout algorithm of Bertsekas and Castanon (1999) by rolling out a set of multiple heuristic policies rather than a single policy. In particular, the parallel rollout approach aims at the class of problems where we have multiple heuristic policies available such that each policy performs near-optimal for a different set of system paths. Parallel rollout automatically combines the given multiple policies to create a new policy that adapts to the different system paths and improves the performance of each policy in the set. We formally prove this claim for two criteria: total expected reward and infinite horizon discounted reward. The parallel rollout approach also resolves the key issue of selecting which policy to roll out among multiple heuristic policies whose performances cannot be predicted in advance. We present two example problems to illustrate the effectiveness of the parallel rollout approach: a buffer management problem and a multiclass scheduling problem.

partially observable Markov decision processrolloutsimulationmulticlass schedulingbuffer management

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Hyeong Soo Chang
    • 1
  • Robert Givan
    • 2
  • Edwin K. P. Chong
    • 3
  1. 1.Department of Computer Science and EngineeringSogang UniversitySeoulKorea
  2. 2.School of Electrical and Computer EngineeringPurdue UniversityWest LafayetteUSA
  3. 3.Department of Electrical and Computer EngineeringColorado State UniversityFort CollinsUSA