Skip to main content
SpringerLink
Log in

Knowledge-Based Hierarchical POMDPs for Task Planning

  • Regular Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The main goal in task planning is to build a sequence of actions that takes an agent from an initial state to a goal state. In robotics, this is particularly difficult because actions usually have several possible results, and sensors are prone to produce measurements with error. Partially observable Markov decision processes (POMDPs) are commonly employed, thanks to their ability to model the uncertainty of actions that modify and monitor the state of a system. However, since solving a POMDP is computationally expensive, their usage becomes prohibitive for most robotic applications. In this article, we propose a task planning architecture for service robotics. In the context of service robot design, we present a scheme to encode knowledge about the robot and its environment, that promotes the modularity and reuse of information. Also, we introduce a new recursive definition of a POMDP that enables our architecture to autonomously build a hierarchy of POMDPs, so that it can be used to generate and execute plans that solve the task at hand. Experimental results show that, in comparison to baseline methods, by following a recursive hierarchical approach the architecture is able to significantly reduce the planning time, while maintaining (or even improving) the robustness under several scenarios that vary in uncertainty and size.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code Availability

https://github.com/saSerrano/hpomdp.git

References

  1. Balai, E., Gelfond, M., Zhang, Y.: Towards answer set programming with sorts. In: International Conference on Logic Programming and Nonmonotonic Reasoning, pp 135–147. Springer (2013)

  2. Bellman, R.E.: Dynamic programming (1957)

  3. Chin, R.T., Harlow, C.A.: Automated visual inspection: a survey. IEEE Trans. Patt. Anal. Mach. Intell 6, 557–573 (1982)

    Article  Google Scholar 

  4. Eversheim, W., Herrmann, P.: Recent trends in flexible automated manufacturing. J. Manuf. Syst. 1(2), 139–148 (1982)

    Article  Google Scholar 

  5. Fikes, R.E., Nilsson, N.J.: Strips: a new approach to the application of theorem proving to problem solving. Artif. Intell. 2(3-4), 189–208 (1971)

    Article  Google Scholar 

  6. Fine, S., Singer, Y., Tishby, N.: The hierarchical hidden markov model: analysis and applications. Mach. Learn. 32(1), 41–62 (1998)

    Article  Google Scholar 

  7. Fox, M., Long, D.: Pddl2. 1: an extension to pddl for expressing temporal planning domains. J. Artif. Intell. Res. 20, 61–124 (2003)

    Article  Google Scholar 

  8. Gelfond, M., Kahl, Y.: Knowledge Representation, Reasoning, and the Design of Intelligent Agents: The Answer-Set Programming Approach. Cambridge University Press, Cambridge (2014)

    Book  Google Scholar 

  9. Hanheide, M., Göbelbecker, M., Horn, G.S., Pronobis, A., Sjöö, K., Aydemir, A., Jensfelt, P., Gretton, C., Dearden, R., Janicek, M., et al.: Robot task planning and explanation in open and uncertain worlds. Artif. Intell. 247, 119–150 (2017)

    Article  MathSciNet  Google Scholar 

  10. Ingrand, F., Ghallab, M.: Deliberation for autonomous robots: a survey. Artif. Intell. 247, 10–44 (2017)

    Article  MathSciNet  Google Scholar 

  11. Kaelbling, L.P., Littman, M.L., Cassandra, A. R.: Planning and acting in partially observable stochastic domains. Artif. Intell. 101(1-2), 99–134 (1998)

    Article  MathSciNet  Google Scholar 

  12. Kress-Gazit, H., Fainekos, G.E., Pappas, G.J.: Temporal-logic-based reactive mission and motion planning. IEEE Trans. Robot. 25(6), 1370–1381 (2009)

    Article  Google Scholar 

  13. Kurniawati, H., Hsu, D., Lee, W. S.: Sarsop: efficient point-based pomdp planning by approximating optimally reachable belief spaces. In: Robotics: Science and Systems, vol. 2008. Zurich, Switzerland. (2008)

  14. McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., Wilkins, D.: Pddl-the planning domain definition language (1998)

  15. Miller, S.A., Harris, Z.A., Chong, E.K.: A pomdp framework for coordinated guidance of autonomous uavs for multitarget tracking. EURASIP J. Adv. Sig. Process. 2009(1), 724597 (2009)

    Article  Google Scholar 

  16. Ng, A.Y., Russell, S.J.: Algorithms for inverse reinforcement learning. In: Proceedings of the Seventeenth International Conference on Machine Learning, pp 663–670 (2000)

  17. Pajarinen, J., Kyrki, V.: Robotic manipulation of multiple objects as a pomdp. Artif. Intell. 247, 213–228 (2017)

    Article  MathSciNet  Google Scholar 

  18. Pandey, A.K., Gelin, R.: A mass-produced sociable humanoid robot: pepper: The first machine of its kind. IEEE Robot. Autom. Mag. 25(3), 40–48 (2018)

    Article  Google Scholar 

  19. Papadimitriou, C.H., Tsitsiklis, J.N.: The complexity of markov decision processes. Math. Oper. Res. 12(3), 441–450 (1987)

    Article  MathSciNet  Google Scholar 

  20. Pineau, J., Gordon, G., Thrun, S., et al.: Point-based value iteration: an anytime algorithm for pomdps. In: IJCAI, vol. 3, pp 1025–1032 (2003)

  21. Pineau, J., Roy, N., Thrun, S.: A hierarchical approach to pomdp planning and execution. In: Workshop on Hierarchy and Memory in Reinforcement Learning (ICML), vol. 65, p 51 (2001)

  22. Pineau, J., Thrun, S.: An integrated approach to hierarchy and abstraction for pomdps (2002)

  23. Png, S., Pineau, J.: Bayesian reinforcement learning for pomdp-based dialogue systems. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 2156–2159. IEEE (2011)

  24. Poudel, D.B.: Coordinating hundreds of cooperative, autonomous robots in a warehouse. Jan 27, 1–13 (2013)

    Google Scholar 

  25. Puterman, M.L.: Markov Decision Processes.: Discrete Stochastic Dynamic Programming. Wiley, Hoboken (2014)

    MATH  Google Scholar 

  26. Shani, G., Pineau, J., Kaplow, R.: A survey of point-based pomdp solvers. Auton. Agent. Multi-Agent Syst. 27(1), 1–51 (2013)

    Article  Google Scholar 

  27. Smith, T., Simmons, R.: Heuristic search value iteration for pomdps. In: Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, pp 520–527 (2004)

  28. Spaan, M.T., Vlassis, N.: Perseus: Randomized point-based value iteration for pomdps. J. Artif. Intell. Res. 24, 195–220 (2005)

    Article  Google Scholar 

  29. Sridharan, M., Gelfond, M., Zhang, S., Wyatt, J.: Reba: a refinement-based architecture for knowledge representation and reasoning in robotics. J. Artif. Intell. Res. 65, 87–180 (2019)

    Article  MathSciNet  Google Scholar 

  30. Sridharan, M., Wyatt, J., Dearden, R.: Planning to see: a hierarchical approach to planning visual actions on a robot using pomdps. Artif. Intell. 174(11), 704–725 (2010)

    Article  Google Scholar 

  31. Theocharous, G., Mahadevan, S.: Hierarchical Learning and Planning in Partially Observable Markov Decision Processes. PhD thesis, Michigan State University. Department of Computer Science & Engineering (2002)

  32. Theocharous, G., Rohanimanesh, K., Maharlevan, S.: Learning hierarchical observable markov decision process models for robot navigation. In: Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164), vol. 1, pp 511–516. IEEE (2001)

  33. Younes, H.L., Littman, M.L.: Ppddl1. 0:, An extension to pddl for expressing planning domains with probabilistic effects, vol. 2 (2004)

  34. Zhang, S., Khandelwal, P., Stone, P.: Dynamically constructed (Po) Mdps for adaptive robot planning. In: AAAI, pp 3855–3863 (2017)

  35. Zhang, S., Sridharan, M., Bao, F.S.: Asp+ Pomdp: integrating non-monotonic logic programming and probabilistic planning on robots. In: 2012 IEEE International Conference On Development and Learning and Epigenetic Robotics (ICDL), pp 1–7. IEEE (2012)

  36. Zhang, S., Stone, P.: Integrated commonsense reasoning and probabilistic planning. In: Proceedings of the 5th Workshop on Planning and Robotics (PlanRob), pp 111–114 (2017)

Download references

Acknowledgements

Elizabeth Santiago deeply thanks to the postdoctoral scholarship and Sergio A. Serrano thanks the scholarship No. 489044, both granted by CONACYT, and also to the Robotics Laboratory of the National Institute of Astrophysics, Optical and Electronic which were important for the realization of this work.

Funding

This work was supported with postdoctoral and master scholarships provided by CONACYT.

Author information

Authors and Affiliations

  1. Instituto Nacional de Astrofísica, Óptica y Electrónica, Computer Science Department, Sta. Ma. Tonantzintla, Puebla, México

    Sergio A. Serrano, Jose Martinez-Carranza, Eduardo F. Morales & L. Enrique Sucar

  2. Instituto Nacional de Enfermedades Respiratorias, Laboratory of Computational Biology, Ciudad de México, México

    Elizabeth Santiago

  3. Computer Science Department, University of Bristol, Bristol, UK

    Jose Martinez-Carranza

  4. Centro de Investigaciones en Matemáticas, Guanajuato, México

    Eduardo F. Morales

Authors
  1. Sergio A. Serrano
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elizabeth Santiago
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jose Martinez-Carranza
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eduardo F. Morales
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. L. Enrique Sucar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Sergio A. Serrano, Elizabeth Santiago, Jose Martinez-Carranza, Eduardo F. Morales and L. Enrique Sucar contributed to the study conception, design and implementation. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sergio A. Serrano.

Ethics declarations

Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Conflict of interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Serrano, S.A., Santiago, E., Martinez-Carranza, J. et al. Knowledge-Based Hierarchical POMDPs for Task Planning. J Intell Robot Syst 101, 82 (2021). https://doi.org/10.1007/s10846-021-01348-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-021-01348-8

Keywords

Search

Navigation