Abstract
Partially observable Markov decision processes (POMDPs) have been successfully applied to various robot motion planning tasks under uncertainty. However, most existing POMDP algorithms assume a discrete state space, while the natural state space of a robot is often continuous. This paper presents Monte Carlo Value Iteration (MCVI) for continuous-state POMDPs. MCVI samples both a robot’s state space and the corresponding belief space, and avoids inefficient a priori discretization of the state space as a grid. Both theoretical results and preliminary experimental results indicate that MCVI is a promising new approach for robot motion planning under uncertainty.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bagnell, J.A., Kakade, S., Ng, A., Schneider, J.: Policy search by dynamic programming. In: Advances in Neural Information Processing Systems (NIPS), vol. 16 (2003)
Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)
Brooks, A., Makarendo, A., Williams, S., Durrant-Whyte, H.: Parametric POMDPs for planning in continuous state spaces. Robotics & Autonomous Systems 54(11), 887–897 (2006)
Brunskill, E., Kaelbling, L., Lozano-Perez, T., Roy, N.: Continuous-state POMDPs with hybrid dynamics. In: Int. Symp. on Artificial Intelligence & Mathematics (2008)
Choset, H., Lynch, K.M., Hutchinson, S., Kantor, G., Burgard, W., Kavraki, L.E., Thrun, S.: Principles of Robot Motion: Theory, Algorithms, and Implementations, vol. ch. 7. The MIT Press, Cambridge (2005)
Hansen, E.A.: Solving POMDPs by searching in policy space. In: Proc. AAAI Conf. on Artificial Intelligence, pp. 211–219 (1998)
He, R., Brunskill, E., Roy, N.: PUMA: Planning under uncertainty with macro-actions. In: Proc. AAAI Conf. on Artificial Intelligence (2010)
Hsiao, K., Kaelbling, L.P., Lozano-Pérez, T.: Grasping POMDPs. In: Proc. IEEE Int. Conf. on Robotics & Automation, pp. 4485–4692 (2007)
Hsu, D., Lee, W.S., Rong, N.: What makes some POMDP problems easy to approximate? In: Advances in Neural Information Processing Systems (NIPS) (2007)
Hsu, D., Lee, W.S., Rong, N.: A point-based POMDP planner for target tracking. In: Proc. IEEE Int. Conf. on Robotics & Automation, pp. 2644–2650 (2008)
Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101(1-2), 99–134 (1998)
Kurniawati, H., Hsu, D., Lee, W.S.: SARSOP: Efficient point-based POMDP planning by approximating optimally reachable belief spaces. In: Proc. Robotics: Science and Systems (2008)
Papadimitriou, C., Tsisiklis, J.N.: The complexity of Markov decision processes. Mathematics of Operations Research 12(3), 441–450 (1987)
Pineau, J., Gordon, G., Thrun, S.: Point-based value iteration: An anytime algorithm for POMDPs. In: Proc. Int. Jnt. Conf. on Artificial Intelligence, pp. 477–484 (2003)
Porta, J.M., Vlassis, N., Spaan, M.T.J., Poupart, P.: Point-based value iteration for continuous POMDPs. J. Machine Learning Research 7, 2329–2367 (2006)
Prentice, S., Roy, N.: The belief roadmap: Efficient planning in linear pomdps by factoring the covariance. In: Proc. Int. Symp. on Robotics Research (2007)
Ross, S., Pineau, J., Paquet, S., Chaib-Draa, B.: Online planning algorithms for POMDPs. J. Artificial Intelligence Research 32(1), 663–704 (2008)
Roy, N., Thrun, S.: Coastal navigation with mobile robots. In: Advances in Neural Information Processing Systems (NIPS), vol. 12, pp. 1043–1049 (1999)
Smith, T., Simmons, R.: Point-based POMDP algorithms: Improved analysis and implementation. In: Proc. Uncertainty in Artificial Intelligence (2005)
Spaan, M.T.J., Vlassis, N.: A point-based POMDP algorithm for robot planning. In: Proc. IEEE Int. Conf. on Robotics & Automation (2004)
Thrun, S.: Monte carlo POMDPs. In: Advances in Neural Information Processing Systems (NIPS). The MIT Press, Cambridge (2000)
Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics. The MIT Press, Cambridge (2005)
Traub, J.F., Werschulz, A.G.: Complexity and Information. Cambridge University Press. Cambridge (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Bai, H., Hsu, D., Lee, W.S., Ngo, V.A. (2010). Monte Carlo Value Iteration for Continuous-State POMDPs. In: Hsu, D., Isler, V., Latombe, JC., Lin, M.C. (eds) Algorithmic Foundations of Robotics IX. Springer Tracts in Advanced Robotics, vol 68. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-17452-0_11
Download citation
DOI: https://doi.org/10.1007/978-3-642-17452-0_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-17451-3
Online ISBN: 978-3-642-17452-0
eBook Packages: EngineeringEngineering (R0)