Learning High-Level Navigation Strategies via Inverse Reinforcement Learning: A Comparative Analysis

  • Michael HermanEmail author
  • Tobias Gindele
  • Jörg Wagner
  • Felix Schmitt
  • Christophe Quignon
  • Wolfram Burgard
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9992)


With an increasing number of robots acting in populated environments, there is an emerging necessity for programming techniques that allow for efficient adjustment of the robot’s behavior to new environments or tasks. A promising approach for teaching robots a certain behavior is Inverse Reinforcement Learning (IRL), which estimates the underlying reward function of a Markov Decision Process (MDP) from observed behavior of an expert. Recently, an approach called Simultaneous Estimation of Rewards and Dynamics (SERD) has been proposed, which extends IRL by simultaneously estimating the dynamics. The objective of this work is to compare classical IRL algorithms with SERD for learning high level navigation strategies in a realistic hallway navigation scenario solely from human expert demonstrations. We show that the theoretical advantages of SERD also pay off in practice by estimating better models of the dynamics and explaining the expert’s demonstrations more accurately.


Inverse Reinforcement Learning Simultaneous Estimation of Rewards and Dynamics Reinforcement Learning Learning from demonstration 


  1. 1.
    Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the Twenty-first International Conference on Machine Learning (2004)Google Scholar
  2. 2.
    Bloem, M., Bamos, N.: Infinite time horizon maximum causal entropy inverse reinforcement learning. In: IEEE 53rd Annual Conference on Decision and Control (2014)Google Scholar
  3. 3.
    Boularias, A., Kober, J., Peters, J.: Relative entropy inverse reinforcement learning. In: Proceedings of 14th International Conference on Artificial Intelligence and Statistics (2011)Google Scholar
  4. 4.
    Helbing, D., Molnar, P.: Social force model for pedestrian dynamics. Phys. Rev. E 51(5), 4282 (1995)CrossRefGoogle Scholar
  5. 5.
    Henry, P., Vollmer, C., Ferris, B., Fox, D.: Learning to navigate through crowded environments. In: IEEE International Conference on Robotics and Automation (2010)Google Scholar
  6. 6.
    Herman, M., Fischer, V., Gindele, T., Burgard, W.: Inverse reinforcement learning of behavioral models for online-adapting navigation strategies. In: IEEE ICRA (2015)Google Scholar
  7. 7.
    Herman, M., Gindele, T., Wagner, J., Schmitt, F., Burgard, W.: Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In: AISTATS (2016)Google Scholar
  8. 8.
    Klein, E., Geist, M., Piot, B., Pietquin, O.: Inverse reinforcement learning through structured classification. In: Proceedings of Advances in Neural Information Processing Systems (2012)Google Scholar
  9. 9.
    Neu, G., Szepesvári, C.: Apprenticeship learning using inverse reinforcement learning and gradient methods. In: UAI (2007)Google Scholar
  10. 10.
    Ng, A.Y., Russell, S.J.: Algorithms for inverse reinforcement learning. In: ICML (2000)Google Scholar
  11. 11.
    Ziebart, B.D., Bagnell, J.A., Dey, A.K.: Modeling interaction via the principle of maximum causal entropy. In: Proceedings of the International Conference on Machine Learning (2010)Google Scholar
  12. 12.
    Ziebart, B.D., Maas, A., Bagnell, J.A.D., Dey, A.: Maximum entropy inverse reinforcement learning. In: Proceeding of AAAI 2008 (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Michael Herman
    • 1
    • 2
    Email author
  • Tobias Gindele
    • 1
  • Jörg Wagner
    • 1
  • Felix Schmitt
    • 1
  • Christophe Quignon
    • 1
  • Wolfram Burgard
    • 2
  1. 1.Robert Bosch GmbHStuttgartGermany
  2. 2.University of FreiburgFreiburgGermany

Personalised recommendations