An Experimental Protocol for Benchmarking Robotic Indoor Navigation

  • Christoph Sprunk
  • Jörg Röwekämper
  • Gershon Parent
  • Luciano Spinello
  • Gian Diego Tipaldi
  • Wolfram Burgard
  • Mihai Jalobeanu
Part of the Springer Tracts in Advanced Robotics book series (STAR, volume 109)


Robot navigation is one of the most studied problems in robotics and the key capability for robot autonomy. Navigation techniques have become more and more reliable, but evaluation mainly focused on individual navigation components (i.e., mapping, localization, and planning) using datasets or simulations. The goal of this paper is to define an experimental protocol to evaluate the whole navigation system, deployed in a real environment. To ensure repeatability and reproducibility of experiments, our benchmark protocol provides detailed definitions and controls the environment dynamics. We define standardized environments and introduce the concept of a reference robot to allow comparison between different navigation systems at different experimentation sites. We present applications of our protocol in experiments in two different research groups, showing the usefulness of the benchmark.


Benchmark Autonomous navigation Indoor robots Dynamic environments 



This work has partly been supported by the EC under FP7-260026-TAPAS, FP7-610917-STAMINA, FP7-610603-EUROPA2, and FP7-267686-LIFENAV. The authors thank all members of the AIS Lab, the Microsoft Robotics Team, Studio99 and the Building 99 Hardware Lab for their patient help with the experiments.


  1. 1.
    Bache, K., Lichman, M.: UCI machine learning repository. University of California, Irvine (2013).
  2. 2.
    Bennett, J., Lanning, S.: The netflix prize. In: KDD Cup and Workshop at the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Minig (2007)Google Scholar
  3. 3.
    Borenstein, J., Feng, L.: Umbmark: a benchmark test for measuring odometry errors in mobile robots. Proc. SPIE 2591, 113-124 (1995)CrossRefGoogle Scholar
  4. 4.
    Burgard, W., Stachniss, C., Grisetti, G., Steder, B., Kümmerle, R., Dornhege, C., Ruhnke, M., Kleiner, A., Tardós, J.D.: A comparison of SLAM algorithms based on a graph of relations. In: International Conference on Intelligent Robots and Systems (2009)Google Scholar
  5. 5.
    Calisi, D., Iocchi, L., Nardi, D.: A unified benchmark framework for autonomous mobile robots and vehicles motion algorithms (MoVeMA benchmarks). In: RSS-Wksp. on Experimental Methodology and Benchmarking in Robotics Research (2008)Google Scholar
  6. 6.
  7. 7.
  8. 8.
    Del Pobil, A.P., Madhavan, R., Messina, E.: Benchmarks in robotics research. In: IROS Workshop on Benchmarks in Robotics Research (2007)Google Scholar
  9. 9.
    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  10. 10.
    Dillmann, R.: Ka 1.10 benchmarks for robotics research (2004).
  11. 11.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2) (2010)Google Scholar
  12. 12.
    Gutmann, J.-S., Burgard, W., Fox, D., Konolige, K.: An experimental comparison of localization methods. In: International Conference on Robotics and Automation (1998)Google Scholar
  13. 13.
    Kikkeri, H., Parent, G., Jalobeanu, M., Birchfield, S.: An inexpensive methodology for evaluating the performance of a mobile robot navigation system. In: International Conference on Robotics and Automation (2014)Google Scholar
  14. 14.
    Knotts, R., Nourbakhsh, I., Morris, R.: Navigates: a benchmark for indoor navigation. In: International Conference and Experiments on Robotics for Challenging, Environments (1998)Google Scholar
  15. 15.
    Nowak, W., Zakharov, A., Blumenthal, S., Prassler, E.: Benchmarks for mobile manipulation and robust obstacle avoidance and navigation. BRICS Deliverable D3.1 (2010)Google Scholar
  16. 16.
    Pang, B., Lee, L.: Opinion mining and sentiment analysis. Found. Trends Inf. Retr. 2(1-2), 1-135 (2008)CrossRefGoogle Scholar
  17. 17.
    PETS 2009 data set.
  18. 18.
    Röwekämper, J., Sprunk, C., Tipaldi, G., Stachniss, C., Pfaff, P., Burgard, W.: On the position accuracy of mobile robot localization based on particle filters combined with scan matching. In: International Conference on Intelligent Robots and Systems (2012)Google Scholar
  19. 19.
    Sprunk, C., Lau, B., Pfaff, P., Burgard, W.: Online generation of kinodynamic trajectories for non-circular omnidirectional robots. In: International Conference on Robotics and Automation (2011)Google Scholar
  20. 20.
    Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: International Conference on Intelligent Robots and Systems (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Christoph Sprunk
    • 1
  • Jörg Röwekämper
    • 1
  • Gershon Parent
    • 2
  • Luciano Spinello
    • 1
  • Gian Diego Tipaldi
    • 1
  • Wolfram Burgard
    • 1
  • Mihai Jalobeanu
    • 2
  1. 1.Department of Computer ScienceUniversity of FreiburgFreiburgGermany
  2. 2.Microsoft RoboticsMicrosoft CorporationWashingtonUSA

Personalised recommendations