Advertisement

Autonomous Robots

, 27:387 | Cite as

On measuring the accuracy of SLAM algorithms

  • Rainer Kümmerle
  • Bastian Steder
  • Christian Dornhege
  • Michael Ruhnke
  • Giorgio Grisetti
  • Cyrill Stachniss
  • Alexander Kleiner
Article

Abstract

In this paper, we address the problem of creating an objective benchmark for evaluating SLAM approaches. We propose a framework for analyzing the results of a SLAM approach based on a metric for measuring the error of the corrected trajectory. This metric uses only relative relations between poses and does not rely on a global reference frame. This overcomes serious shortcomings of approaches using a global reference frame to compute the error. Our method furthermore allows us to compare SLAM approaches that use different estimation techniques or different sensor modalities since all computations are made based on the corrected trajectory of the robot.

We provide sets of relative relations needed to compute our metric for an extensive set of datasets frequently used in the robotics community. The relations have been obtained by manually matching laser-range observations to avoid the errors caused by matching algorithms. Our benchmark framework allows the user to easily analyze and objectively compare different SLAM approaches.

Keywords

SLAM Mapping accuracy Benchmarking 

References

  1. Amigoni, F., Gasparini, S., &  Gini, M. (2007). Good experimental methodologies for robotic mapping: A proposal. In Proc. of the IEEE int. conf. on robotics & automation (ICRA). Google Scholar
  2. Balaguer, B., Carpin, S., & Balakirsky, S. (2007). Towards quantitative comparisons of robot algorithms: Experiences with SLAM in simulation and real world systems. In IROS 2007 workshop. Google Scholar
  3. Bar-Shalom, Y., Li, X. R., & Kirubarajan, T. (2001). Estimation with application to tracking and navigation. New York: Wiley. CrossRefGoogle Scholar
  4. Bonarini, A., Burgard, W., Fontana, G., Matteucci, M., Sorrenti, D. G., & Tardos, J. D. (2006). Rawseeds a project on SLAM benchmarking. In Proceedings of the IROS’06 workshop on benchmarks in robotics research. Available online at http://www.robot.uji.es/EURON/pdfs/LectureNotesIROS06.pdf.
  5. Bosse, M., Newman, P. M., Leonard, J. J., & Teller, S. (2003). An ALTAS framework for scalable mapping. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 1899–1906). Taipei, Taiwan. Google Scholar
  6. Burgard, W., Stachniss, C., Grisetti, G., Steder, B., Kümmerle, R., Dornhege, C., Ruhnke, M., Kleiner, A., & Tardós, J. D. (2009, to appear). A comparison of slam algorithms based on a graph of relations. In Proc. of the int. conf. on intelligent robots and systems (IROS). Google Scholar
  7. Canny, J. (1986). A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6), 679–698. CrossRefGoogle Scholar
  8. Censi, A. (2006). Scan matching in a probabilistic framework. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2291–2296). Google Scholar
  9. Darpa (2007). Darpa Urban Challenge. http://www.darpa.mil/grandchallenge/.
  10. Dellaert, F. (2005). Square Root SAM. In Proc. of robotics: science and systems (RSS) (pp. 177–184). Cambridge, MA, USA. Google Scholar
  11. Dellaert, F., Fox, D., Burgard, W., & Thrun, S. (1998). Monte Carlo localization for mobile robots. In Proc. of the IEEE int. conf. on robotics & automation (ICRA). Leuven, Belgium. Google Scholar
  12. Dissanayake, G., Durrant-Whyte, H., & Bailey, T. (2000). A computationally efficient solution to the simultaneous localisation and map building (SLAM) problem. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 1009–1014). Google Scholar
  13. Doucet, A., de Freitas, N., & Gordan, N. (Eds.) (2001). Sequential Monte-Carlo methods in practice. Berlin: Springer. zbMATHGoogle Scholar
  14. Duckett, T., Marsland, S., & Shapiro, J. (2002). Fast, on-line learning of globally consistent maps. Autonomous Robots, 12(3), 287–300. zbMATHCrossRefGoogle Scholar
  15. EPFL and IROS (2002). Cleaning Robot Contest. http://robotika.cz/competitions/cleaning2002/en.
  16. ESA (2008). Lunar robotics challenge. http://www.esa.int/esaCP/SEM4GKRTKMF_index_0.html.
  17. Estrada, C., Neira, J., & Tardós, J. D. (2005). Hierarchical SLAM: Real-time accurate mapping of large environments. IEEE Transactions on Robotics, 21(4), 588–596. CrossRefGoogle Scholar
  18. Eustice, R., Singh, H., & Leonard, J. J. (2005a). Exactly sparse delayed-state filters. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2428–2435). Google Scholar
  19. Eustice, R., Walter, M., & Leonard, J. J. (2005b). Sparse extended information filters: Insights into sparsification. In Proc. of the int. conf. on intelligent robots and systems (IROS) (pp. 641–648). Edmonton, Canada. Google Scholar
  20. Frese, U. (2006). Treemap: An o(log n) algorithm for indoor simultaneous localization and mapping. Autonomous Robots, 21(2), 103–122. CrossRefGoogle Scholar
  21. Frese, U. (2008). Dlr spatial cognition data set. http://www.informatik.uni-bremen.de/agebv/en/DlrSpatialCognitionDataSet.
  22. Frese, U., Larsson, P., & Duckett, T. (2005). A multilevel relaxation algorithm for simultaneous localisation and mapping. IEEE Transactions on Robotics, 21(2), 1–12. CrossRefGoogle Scholar
  23. Früh, C., & Zakhor, A. (2004). An automated method for large-scale, ground-based city model acquisition. International Journal of Computer Vision, 60, 5–24. CrossRefGoogle Scholar
  24. Grisetti, G., Grzonka, S., Stachniss, C., Pfaff, P., & Burgard, W. (2007a). Efficient estimation of accurate maximum likelihood maps in 3D. In Proc. of the int. conf. on intelligent robots and systems (IROS). San Diego, CA, USA. Google Scholar
  25. Grisetti, G., Stachniss, C., & Burgard, W. (2007b). Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Transactions on Robotics, 23, 34–46. CrossRefGoogle Scholar
  26. Grisetti, G., Stachniss, C., Grzonka, S., & Burgard, W. (2007c). A tree parameterization for efficiently computing maximum likelihood maps using gradient descent. In Proc. of robotics: science and systems (RSS). Google Scholar
  27. Gutmann, J.-S., & Konolige, K. (1999). Incremental mapping of large cyclic environments. In Proc. of the IEEE int. symposium on computational intelligence in robotics and automation (CIRA). Google Scholar
  28. Hähnel, D., Burgard, W., Fox, D., & Thrun, S. (2003). An efficient FastSLAM algorithm for generating maps of large-scale cyclic environments from raw laser range measurements. In Proc. of the int. conf. on intelligent robots and systems (IROS) (pp. 206–211). Google Scholar
  29. Hermosillo, J., Pradalier, C., Sekhavat, S., Laugier, C., & Baille, G. (2003). Towards motion autonomy of a bi-steerable car: Experimental issues from map-building to trajectory execution. In Proc. of the IEEE int. conf. on robotics & automation (ICRA). Google Scholar
  30. Hoover, A., Jean-Baptiste, G., Jiang, X., Flynn, P. J., Bunke, H., Goldgof, D. B., Bowyer, K. K., Eggert, D. W., Fitzgibbon, A. W., & Fisher, R. B. (1996). An experimental comparison of range image segmentation algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(7), 673–689. CrossRefGoogle Scholar
  31. Howard, A., & Roy, N. (2003). Radish: The robotics data set repository, standard data sets for the robotics community. http://radish.sourceforge.net/.
  32. Julier, S., Uhlmann, J., & Durrant-Whyte, H. (1995). A new approach for filtering nonlinear systems. In Proc. of the American control conference (pp. 1628–1632). Google Scholar
  33. Kaess, M., Ranganathan, A., & Dellaert, F. (2007). iSAM: Fast incremental smoothing and mapping with efficient data association. In Proc. of the IEEE int. conf. on robotics & automation (ICRA). Google Scholar
  34. Kümmerle, R., Steder, B., Dornhege, C., Kleiner, A., Grisetti, G., & Burgard, W. (2009). Large scale graph-based SLAM using aerial images as prior information. In Proc. of robotics: science and systems (RSS). Google Scholar
  35. Leonard, J. J., & Durrant-Whyte, H. F. (1991). Mobile robot localization by tracking geometric beacons. IEEE Transactions on Robotics and Automation, 7(4), 376–382. CrossRefGoogle Scholar
  36. Lu, F., & Milios, E. (1994). Robot pose estimation in unknown environments by matching 2d range scans. In IEEE computer vision and pattern recognition conference (CVPR) (pp. 935–938). Google Scholar
  37. Lu, F., & Milios, E. (1997). Globally consistent range scan alignment for environment mapping. Autonomous Robots, 4, 333–349. CrossRefGoogle Scholar
  38. Montemerlo, M., Thrun, S., Koller, D., & Wegbreit, B. (2003). FastSLAM 2.0: An improved particle filtering algorithm for simultaneous localization and mapping that provably converges. In Proc. of the int. conf. on artificial intelligence (IJCAI) (pp. 1151–1156). Google Scholar
  39. Nüchter, A., Lingemann, K., Hertzberg, J., & Surmann, H. (2005). 6d SLAM with approximate data association. In Proc. of the 12th int. conference on advanced robotics (ICAR) (pp. 242–249). Google Scholar
  40. Olson, E. (2008). Robust and efficient robotic mapping. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA, USA. Google Scholar
  41. Olson, E., Leonard, J., & Teller, S. (2006). Fast iterative optimization of pose graphs with poor initial estimates. In Proc. of the IEEE int. conf. on robotics & automation (ICRA) (pp. 2262–2269). Google Scholar
  42. Pfaff, P., Burgard, W., & Fox, D. (2006). Robust Monte-Carlo localization using adaptive likelihood models. In H.I. Christiensen (Ed.), STAR Springer tracts in advanced robotics : Vol. 22. European robotics symposium 2006 (pp. 181–194). Berlin: Springer. CrossRefGoogle Scholar
  43. Ranganathan, A., Kaess, M., & Dellaert, F. (2007). Loopy sam. In Proc. of the int. conf. on artificial intelligence (IJCAI). Google Scholar
  44. RoboCup Federation (2009). RoboCup Competitions. http://www.robocup.org.
  45. Scharstein, D., & Szeliski, R. (2002). Middlebury stereo vision page. http://www.middlebury.edu/stereo.
  46. Smith, R. C., & Cheeseman, P. (1986). On the representation and estimation of spatial uncertainty. International Journal of Robotics Research, 5(4), 56–68. CrossRefGoogle Scholar
  47. Smith, R., Self, M., & Cheeseman, P. (1990). Estimating uncertain spatial relationships in robotics. In I. Cox & G. Wilfong (Eds.), Autonomous robot vehicles (pp. 167–193). Berlin: Springer. Google Scholar
  48. Stachniss, C., Frese, U., & Grisetti, G. (2007a). OpenSLAM. org—give your algorithm to the community. http://www.openslam.org.
  49. Stachniss, C., Grisetti, G., Roy, N., & Burgard, W. (2007b). Evaluation of Gaussian proposal distributions for mapping with Rao-Blackwellized particle filters. In Proc. of the int. conf. on intelligent robots and systems (IROS). Google Scholar
  50. Symeo GmbH (2008). http://www.symeo.de.
  51. Thrun, S. (2001). An online mapping algorithm for teams of mobile robots. International Journal of Robotics Research, 20(5), 335–363. CrossRefGoogle Scholar
  52. Thrun, S., Liu, Y., Koller, D., Ng, A. Y., Ghahramani, Z., & Durrant-Whyte, H. (2004). Simultaneous localization and mapping with sparse extended information filters. International Journal of Robotics Research, 23(7/8), 693–716. Google Scholar
  53. Thrun, S., Burgard, W., & Fox, D. (2005). Probabilistic robotics. Cambridge: MIT Press. zbMATHGoogle Scholar
  54. Thrun, S. et al. (2006). Winning the Darpa grand challenge. Journal on Field Robotics. Google Scholar
  55. Torralba, A., Murphy, K. P., & Freeman, W. T. (2007). Labelme: the open annotation tool. http://labelme.csail.mit.edu/.
  56. Uhlmann, J. (1995). Dynamic map building and localization: New theoretical foundations. PhD thesis, University of Oxford. Google Scholar
  57. Wulf, O., Nüchter, A., Hertzberg, J., & Wagner, B. (2008). Benchmarking urban six-degree-of-freedom simultaneous localization and mapping. Journal of Field Robotics, 25(3), 148–163. CrossRefGoogle Scholar
  58. Yguel, M., Keat, C. T. M., Braillon, C., Laugier, C., & Aycard, O. (2007). Dense mapping for range sensors: Efficient algorithms and sparse representations. In Proc. of robotics: science and systems (RSS). Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Rainer Kümmerle
    • 1
  • Bastian Steder
    • 1
  • Christian Dornhege
    • 2
  • Michael Ruhnke
    • 1
  • Giorgio Grisetti
    • 1
  • Cyrill Stachniss
    • 1
  • Alexander Kleiner
    • 2
  1. 1.Dept. of Computer ScienceUniversity of FreiburgFreiburgGermany
  2. 2.Dept. of Computer ScienceUniversity of FreiburgFreiburgGermany

Personalised recommendations