Point-Based Policy Transformation: Adapting Policy to Changing POMDP Models

  • Hanna Kurniawati
  • Nicholas M. Patrikalakis
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 86)


Motion planning under uncertainty that can efficiently take into account changes in the environment is critical for robots to operate reliably in our living spaces. Partially Observable Markov Decision Process (POMDP) provides a systematic and general framework for motion planning under uncertainty. Point-based POMDP has advanced POMDP planning tremendously over the past few years, enabling POMDP planning to be practical for many simple to moderately difficult robotics problems. However, when environmental changes alter the POMDP model, most existing POMDP planners recompute the solution from scratch, often wasting significant computational resources that have been spent for solving the original problem. In this paper, we propose a novel algorithm, called Point-Based Policy Transformation (PBPT), that solves the altered POMDP problem by transforming the solution of the original problem to accommodate changes in the problem. PBPT uses the point-based POMDP approach. It transforms the original solution by modifying the set of sampled beliefs that represents the belief space B, and then uses this new set of sampled beliefs to revise the original solution. Preliminary results indicate that PBPT generates a good policy for the altered POMDP model in a matter of minutes, while recomputing the policy using the fastest offline POMDP planner today fails to find a policy with similar quality after two hours of planning time, even when the policy for the original problem is reused as an initial policy.


Optimal Policy Autonomous Underwater Vehicle Reward Function Good Policy State Trace 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The non-stochastic multi-armed bandit problem. SIAM Journal on Computing 32(1), 48–77 (2003)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Bai, H., Hsu, D., Lee, W.S., Ngo, V.A.: Monte Carlo Value Iteration for Continuous-State POMDPs. In: Hsu, D., Isler, V., Latombe, J.-C., Lin, M.C. (eds.) Algorithmic Foundations of Robotics IX. STAR, vol. 68, pp. 175–191. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  3. 3.
    van den Berg, J., Abbeel, P., Goldberg, K.: LQG-MP: Optimized Path Planning for Robots with Motion Uncertainty and Imperfect State Information. In: RSS (2010)Google Scholar
  4. 4.
    van den Berg, J., Overmars, M.: Roadmap-based motion planning in dynamic environments. IEEE TRO 21(5), 885–897 (2005)Google Scholar
  5. 5.
    de Berg, M., Cheong, O., van Kreveld, M., Overmars, M.: Computational Geometry: Algorithms and Applications. Springer (2000)Google Scholar
  6. 6.
    Dudley, R.M.: Real Analysis and Probability. Cambridge University Press (2002)Google Scholar
  7. 7.
    Hauser, K.: Randomized Belief-Space Replanning in Partially-Observable Continuous Spaces. In: Hsu, D., Isler, V., Latombe, J.-C., Lin, M.C. (eds.) Algorithmic Foundations of Robotics IX. STAR, vol. 68, pp. 193–209. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    He, R., Brunskill, E., Roy, N.: PUMA: planning under uncertainty with macro-actions. In: AAAI (2010)Google Scholar
  9. 9.
    Jaillet, L., Siméon, T.: A PRM-based motion planner for dynamically changing environments. In: IROS (2004)Google Scholar
  10. 10.
    Kurniawati, H., Bandyopadhyay, T., Patrikalakis, N.M.: Global motion planning under uncertain motion, sensing, and environment map. In: RSS (2011)Google Scholar
  11. 11.
    Kurniawati, H., Hsu, D., Lee, W.S.: SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: RSS (2008)Google Scholar
  12. 12.
    Lamiraux, F., Bonnafous, D., Lefebvre, O.: Reactive path deformation for nonholonomic mobile robots. IEEE TRO 20(6), 967–977 (2004)Google Scholar
  13. 13.
    LaValle, S.M., Sharma, R.: On motion planning in changing, partially-predictable environments. IJRR 16(6), 775–805 (1997)Google Scholar
  14. 14.
    Leven, P., Hutchinson, S.: Real-time path planning in changing environments. IJRR 21(12), 999–1030 (2001)Google Scholar
  15. 15.
    Papadimitriou, C.H., Tsitsiklis, J.N.: The Complexity of Markov Decision Processes. Math. of Operation Research 12(3), 441–450 (1987)CrossRefzbMATHMathSciNetGoogle Scholar
  16. 16.
    Pineau, J., Gordon, G., Thrun, S.: Point-based value iteration: An anytime algorithm for POMDPs. In: IJCAI, pp. 1025–1032 (2003)Google Scholar
  17. 17.
    Platt, R., Tedrake, R., Lozano-Perez, T., Kaelbling, L.P.: Belief space planning assuming maximum likelihood observations. In: RSS (2010)Google Scholar
  18. 18.
    Porta, J.M., Vlassis, N., Spaan, M.T.J., Poupart, P.: Point-Based Value Iteration for Continuous POMDPs. JMLR 7, 2329–2367 (2006)zbMATHMathSciNetGoogle Scholar
  19. 19.
    Prentice, S., Roy, N.: The Belief Roadmap: Efficient Planning in Linear POMDPs by Factoring the Covariance. In: ISRR (2007)Google Scholar
  20. 20.
    Ross, S., Chaib-draa, B., Pineau, J.: Bayes-adaptive POMDPs. In: NIPS (2007)Google Scholar
  21. 21.
    Ross, S., Pineau, J., Paquet, S., Chaib-draa, B.: Online planning algorithms for POMDPs. JAIR 32, 663–704 (2008)zbMATHMathSciNetGoogle Scholar
  22. 22.
    Smith, T., Simmons, R.: Point-based POMDP algorithms: Improved analysis and implementation. In: UAI (July 2005)Google Scholar
  23. 23.
    Stentz, A.: The Focussed D* Algorithm for Real-Time Replanning. In: IJCAI (1995)Google Scholar
  24. 24.
    Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: A survey. JMLR 10(1), 1633–1685 (2009)zbMATHMathSciNetGoogle Scholar
  25. 25.
    Thrun, S.: Monte carlo POMDPs. In: NIPS, pp. 1064–1070 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.School of Information Technology & Electrical EngineeringUniversity of QueenslandQueenslandAustralia
  2. 2.Department of Mechanical Engineering, Center for Ocean EngineeringMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations