Autonomous Robots

, Volume 40, Issue 5, pp 761–787 | Cite as

Map evaluation using matched topology graphs

  • Sören SchwertfegerEmail author
  • Andreas Birk


Mapping is an important task for mobile robots. The assessment of the quality of maps in a simple, efficient and automated way is not trivial and an ongoing research topic. Here, a new approach for the evaluation of 2D grid maps is presented. This structure-based method makes use of a topology graph, i.e., a topological representation that includes abstracted local metric information. It is shown how the topology graph is constructed from a Voronoi diagram that is pruned and simplified such that only high level topological information remains to concentrate on larger, topologically distinctive places. Several methods for computing the similarity of vertices in two topology graphs, i.e., for performing a place-recognition, are presented. Based on the similarities, it is shown how subgraph-isomorphisms can be efficiently computed and two topology graphs can be matched. The match between the graphs is then used to calculate a number of standard map evaluation attributes like coverage, global accuracy, relative accuracy, consistency, and brokenness. Experiments with robot generated maps are used to highlight the capabilities of the proposed approach and to evaluate the performance of the underlying algorithms.


Mobile robot Performance metric Map quality Ground truth comparison Topology Place recognition Simultaneous localization and mapping (SLAM ) 



The authors would like to thank all RoboCup Rescue teams that agreed to have their maps published in the context of this research: CASualty: University of New South Wales, Australia and Resko: University of Koblenz, Germany.


  1. Aurenhammer, F. (1991). Voronoi diagramsã survey of a fundamental geometric data structure. ACM Computing Surveys, 23, 345–405. doi: 10.1145/116873.116880.CrossRefGoogle Scholar
  2. Bai, X., & Latecki, L. (2008). Path similarity skeleton graph matching. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30(7), 1282–1292. doi: 10.1109/TPAMI.2007.70769.CrossRefGoogle Scholar
  3. Bai, X., Latecki, L., & yu Liu, W. (2007). Skeleton pruning by contour partitioning with discrete curve evolution. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 29(3), 449–462. doi: 10.1109/TPAMI.2007.59.CrossRefGoogle Scholar
  4. Balaguer, B., Balakirsky, S., Carpin, S., & Visser, A. (2009). Evaluating maps produced by urban search and rescue robots: Lessons learned from robocup. Autonomous Robots, 27, 449–464.CrossRefGoogle Scholar
  5. Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2), 239–256.CrossRefGoogle Scholar
  6. Birk, A. (2010). A quantitative assessment of structural errors in grid maps. Autonomous Robots, 28, 187–196.CrossRefGoogle Scholar
  7. Birk, A., Wiggerich, B., Bülow, H., Pfingsthorn, M., & Schwertfeger, S. (2011). Safety, security, and rescue missions with an unmanned aerial vehicle (UAV): Aerial mosaicking and autonomous flight at the 2009 european land robots trials (elrob) and the 2010 response robot evaluation exercises (rree). Journal of Intelligent and Robotic Systems, 64(1), 57–76.CrossRefGoogle Scholar
  8. Chen, Q. S., Defrise, M., & Deconinck, F. (1994). Symmetric phase-only matched filtering of fourier-mellin transforms for image registration and recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 16(12), 1156–1168.CrossRefGoogle Scholar
  9. Edelsbrunner, H., Kirkpatrick, D., & Seidel, R. (1983). On the shape of a set of points in the plane. Information Theory, IEEE Transactions on, 29(4), 551–559. doi: 10.1109/TIT.1983.1056714.MathSciNetCrossRefzbMATHGoogle Scholar
  10. Fortin, S. (1996). The graph isomorphism problem, Technical report.Google Scholar
  11. Handa, A., Whelan, T., McDonald, J. & Davison, A. (2014) A benchmark for rgb-d visual odometry, 3D reconstruction and slam. In: Robotics and automation (ICRA), 2014 IEEE international conference on, pp. 1524–1531 . doi: 10.1109/ICRA.2014.6907054
  12. Horn, B. K. P. (1987). Closed-form solution of absolute orientation using unit quaternions. Journal of the Optical Society of America, 4(4), 629–642.MathSciNetCrossRefGoogle Scholar
  13. Karavelas, M. (2011). 2D Voronoi diagram adaptor. In: CGAL user and reference manual, 3.8 edn. CGAL Editorial Board. Accessed 12 April 2012.
  14. Klein, R. (1988). Abstract voronoi diagrams and their applications. In H. Noltemeier (Ed.), Computational geometry and its applications., Lecture notes in computer science Berlin: Springer.Google Scholar
  15. Kümmerle, R., Steder, B., Dornhege, C., Ruhnke, M., Grisetti, G., Stachniss, C., et al. (2009). On measuring the accuracy of slam algorithms. Autonomous Robots, 27(4), 387–407.CrossRefGoogle Scholar
  16. Lakaemper, R., & Adluru, N. (2009). Using virtual scans for improved mapping and evaluation. Autonomous Robots, 27(4), 431–448. doi: 10.1007/s10514-009-9149-4.CrossRefGoogle Scholar
  17. Lee, D.C. (1996). The map-building and exploration strategies of a simple sonar-equipped mobile robot: An experimental, quantitative evaluation, distinguished dissertations in computer science, Cambridge University Press.Google Scholar
  18. Leonard, J. J., & Durrant-Whyte, H. F. (1992). Directed sonar sensing for mobile robot navigation. New York: Springer.CrossRefzbMATHGoogle Scholar
  19. Pellenz, J. & Paulus, D. (2008). Mapping and map scoring at the roboCupRescue competition. Quantitative performance evaluation of navigation solutions for mobile robots (RSS 2008, Workshop CD)Google Scholar
  20. Pomerleau, F., Liu, M., Colas, F., & Siegwart, R. (2012). Challenging data sets for point cloud registration algorithms. The International Journal of Robotics Research, 31(14), 1705–1711.CrossRefGoogle Scholar
  21. Schwertfeger, S.: Robocuprescue interleague mapping challenge (2010). Accessed 16 Mar 2014.
  22. Schwertfeger, S. (2012). Robotic mapping in the real world: Performance evaluation and system integration. Ph.D. thesis, Department of Electrical Engeneering and Computer Science, Jacobs University Bremen. Accessed 16 Mar 2014.
  23. Schwertfeger, S. & Birk, A. (2013) Evaluation of map quality by matching and scoring high-level, topological map structures. In: Robotics and Automation (ICRA), 2013 IEEE International Conference onGoogle Scholar
  24. Schwertfeger, S., Bülow, H. & Birk, A.(2010) On the effects of sampling resolution in improved Fourier mellin based registration for underwater mapping. In 7th international symposium on intelligent autonomous vehicles (IAV 2010), IFAC.Google Scholar
  25. Schwertfeger, S., Jacoff, A., Pellenz, J. & Birk, A. (2011) Using a fiducial map metric for assessing map quality in the context of robocup rescue. In: International workshop on safety, security, and rescue robotics (SSRR), IEEE Press.Google Scholar
  26. Schwertfeger, S., Jacoff, A., Scrapper, C., Pellenz, J. & Kleiner, A. (2010). Evaluation of maps using fixed shapes: The fiducial map metric. In Proceedings of PerMISGoogle Scholar
  27. Scrapper, C., Madhavan, R. & Balakirsky, S. (2007). Stable navigation solutions for robots in complex environments. In: IEEE international workshop on safety, security and rescue robotics (SSRR), pp. 1–6.Google Scholar
  28. Siddiqi, K., Shokoufandeh, A., Dickenson, S. & Zucker, S. (1998) Shock graphs and shape matching. In: Computer vision, 1998. Sixth international conference on, pp. 222–229. doi: 10.1109/ICCV.1998.710722
  29. Sturm, J., Engelhard, N., Endres, F., Burgard, W. & Cremers, D. (2012). A benchmark for the evaluation of rgb-d slam systems. In Intelligent robots and systems (IROS), 2012 IEEE/RSJ international conference on, pp. 573–580. doi: 10.1109/IROS.2012.6385773
  30. Texas A&M University College Station, T. (2010) National Institute of Standards and Technology: NIST response robot evaluation exercise #6. Accessed 30 Aug 2015.
  31. Thrun, S. (2002). Robotic mapping: A survey. In G. Lakemeyer & B. Nebel (Eds.), Exploring artificial intelligence in the new millenium. San Diego: Morgan Kaufmann.Google Scholar
  32. Torsello, A., & Hancock, E. (2001). A skeletal measure of 2D shape similarity. In C. Arcelli, L. Cordella, & G. di Baja (Eds.), Visual form 2001 (Vol. 2059, pp. 260–271)., Lecture notes in computer science Berlin: Springer.CrossRefGoogle Scholar
  33. Varsadan, I., Birk, A., & Pfingsthorn, M. (2009). Determining map quality through an image similarity metric. In L. Locchi, H. Matsubara, A. Weitzenfeld, & C. Zhou (Eds.), RoboCup 2008: Robot World Cup XII (pp. 355–365)., Lecture notes in artificial intelligence New York: Springer.CrossRefGoogle Scholar
  34. Voronoi, G. (1908). Nouvelles applications des paramètres continus à la théorie des formes quadratiques. Premier mémoire. Sur quelques propriétés des formes quadratiques positives parfaites. Journal Für Die Reine und Angewandte Mathematik, 133, 97–102. doi: 10.1515/crll.1908.133.97.
  35. Wagan, A.I., Godil, A. & Li, X. (2008). Map quality assessment. In: PerMIS ’08: Proceedings of the 8th workshop on performance metrics for intelligent systems, pp. 278–282. ACM, New York, doi: 10.1145/1774674.1774718
  36. Wulf, O., Nüchter, A., Hertzberg, J. & Wagner, B. (2007). Ground truth evaluation of large urban 6D slam. pp. 650–657. doi: 10.1109/IROS.2007.4399026
  37. Zemlyachenko, V. N., Korneenko, N. M., & Tyshkevich, R. I. (1985). Graph isomorphism problem. Journal of Mathematical Sciences, 29(4), 1426–1481. doi: 10.1007/BF02104746.zbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.School of Information Science and TechnologyShanghaiTech UniversityShanghaiChina
  2. 2.Department of Electrical Engineering and Computer ScienceJacobs University BremenBremenGermany

Personalised recommendations