Determining Map Quality through an Image Similarity Metric

  • Ioana Varsadan
  • Andreas Birk
  • Max Pfingsthorn
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5399)


A quantitative assessment of the quality of a robot generated map is of high interest for many reasons. First of all, it allows individual researchers to quantify the quality of their mapping approach and to study the effects of system specific choices like different parameter values in an objective way. Second, it allows peer groups to rank the quality of their different approaches to determine scientific progress; similarly, it allows rankings within competition environments like RoboCup. A quantitative assessment of map quality based on an image similarity metric Ψ is introduced here. It is shown through synthetic as well as through real world data that the metric captures intuitive notions of map quality. Furthermore, the metric is compared to a seemingly more straightforward metric based on Least Mean Squared Euclidean distances (LMS-ED) between map points and ground truth. It is shown that both capture intuitive notions of map quality in a similar way, but that Ψ can be computed much more efficiently than the LMS-ED.


Ground Truth Mobile Robot Image Similarity Ground Truth Data Test Element 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    The robocup federation,
  2. 2.
  3. 3.
    Association for the advancement of artificial intelligence,
  4. 4.
  5. 5.
    Jacoff, A., Weiss, B., Messina, E.: Evolution of a performance metric for urban search and rescue. In: Performance Metrics for Intelligent Systems (PERMIS), Gaithersburg, MD (2003)Google Scholar
  6. 6.
    Jacoff, A., Messina, E., Evans, J.: Performance evaluation of autonomous mobile robots. Industrial Robot: An International Journal 29(3), 259–267 (2002)CrossRefGoogle Scholar
  7. 7.
    Jacoff, A., Messina, E., Weiss, B., Tadokoro, S., Nakagawa, Y.: Test arenas and performance metrics for urban search and rescue robots. In: Proceedings of the Intelligent and Robotic Systems (IROS) Conference (2003)Google Scholar
  8. 8.
    Jacoff, A., Messina, E., Evans, J.: Experiences in deploying test arenas for autonomous mobile robots. In: Performance Metrics for Intelligent Systems (PERMIS), Mexico City (2001)Google Scholar
  9. 9.
    Thrun, S.: Robotic mapping: A survey. In: Lakemeyer, G., Nebel, B. (eds.) Exploring Artificial Intelligence in the New Millenium. Morgan Kaufmann, San Francisco (2002)Google Scholar
  10. 10.
    Elfes, A.: Using occupancy grids for mobile robot perception and navigation. Computer 22(6), 46–57 (1989)CrossRefGoogle Scholar
  11. 11.
    Moravec, H., Elfes, A.: High resolution maps from wide angle sonar. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 116–121 (1985)Google Scholar
  12. 12.
    Birk, A.: Learning geometric concepts with an evolutionary algorithm. In: Proc. of The Fifth Annual Conference on Evolutionary Programming. MIT Press, Cambridge (1996)Google Scholar
  13. 13.
    Moravec, M.B.H.: Learning sensor models for evidence grids. In: CMU Robotics Institute 1991 Annual Research Review, pp. 8–15 (1993)Google Scholar
  14. 14.
    Kullback, R.A.L.S.: On information and sufficiency. Ann. Math. Stat., 79–86 (1951)Google Scholar
  15. 15.
    Yairi, T.: Covisibility based map learning method for mobile robots. In: Zhang, C., Guesgen, H.W., Yeap, W.-K. (eds.) PRICAI 2004. LNCS, vol. 3157, pp. 703–712. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  16. 16.
    Wnuk, K.: Dense 3d mapping with monocular vision (March 2005)Google Scholar
  17. 17.
    Blanco, J.-L., González, J., Fernandez-Madrigal, J.-A.: A new method for robust and efficient occupancy grid-map matching. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds.) IbPRIA 2007. LNCS, vol. 4478, pp. 194–201. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  18. 18.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91 (2004)CrossRefGoogle Scholar
  19. 19.
    Nuchter, A., Wulf, O., Hertzberg, J., Wagner, B.: Benchmarking urban 6d slam. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (2007)Google Scholar
  20. 20.
    Nuechter, A., Lingemann, K., Hertzberg, J., Surmann, H.: 6d slam – mapping outdoor environments. In: IEEE International Workshop on Safety, Security, and Rescue Robotics (SSRR). IEEE Press, Los Alamitos (2006)Google Scholar
  21. 21.
    Dellaert, F., Fox, D., Burgard, W., Thrun, S.: Monte carlo localization for mobile robots. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 2, pp. 1322–1328 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Ioana Varsadan
    • 1
  • Andreas Birk
    • 1
  • Max Pfingsthorn
    • 1
  1. 1.Jacobs University BremenBremenGermany

Personalised recommendations