Quantitative Assessment of Robot-Generated Maps

  • C. Scrapper
  • R. Madhavan
  • R. Lakaemper
  • A. Censi
  • A. Godil
  • A. Wagan
  • A. Jacoff


Mobile robotic mapping is now considered to be a sufficiently mature field with demonstrated successes in various domains. While much progress has been made in the development of computationally efficient and consistent mapping schemes, it is still murky, at best, on how these maps can be evaluated. We are motivated by the absence of an accepted standard for quantitatively measuring the performance of robotic mapping systems against user-defined requirements. It is our belief that the development of standardized methods for quantitatively evaluating existing robotic technologies will improve the utility of mobile robots in already established application areas, such as vacuum cleaning, robot surveillance, and bomb disposal. This approach will also enable the proliferation and acceptance of such technologies in emerging markets. This chapter summarizes our preliminary efforts by bringing together the research community towards addressing this important problem which has ramifications not only from researchers’ perspective but also from consumers’, robot manufacturers’, and developers’ viewpoints.


Ground Truth Scale Invariant Feature Transform Iterative Close Point Fisher Information Matrix Ground Truth Data 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Abdallah, S., Asmar, D,. Zelek, J.: Towards benchmarks for vision SLAM algorithms. In: Proceedings of the IEEE International Conference on Robotics and Automation, pp. 3834–3839 (2006)Google Scholar
  2. 2.
    ASTM Committee E54.08.01 on Homeland Security Operations; Operational Equippment; Robots.
  3. 3.
    ASTM International: In: Form and style for ASTM Standards. (2007)Google Scholar
  4. 4.
    Ballard, D.H.: Generalizing the Hough Transform to Detect Arbitrary Shapes, Los Altos, pp. 714–725 (1987)Google Scholar
  5. 5.
    Bertel, S., Barkowsky, T., Engel, D., Freska, C.: Computational modeling of reasoning with mental images: basic requirements. In: D. Fum, F. del Missier, A. Stocco (Eds.), Proceedings of the 7th International Conference on Cognitive Modeling ICCM06 (2006)Google Scholar
  6. 6.
    Besl, P., McKay, N.: A method for registration of 3.d shapes. IEEE PAMI, 14(2) (1992)Google Scholar
  7. 7.
    Censi, A.: On achievable accuracy for range-finder localization. In: Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 4170–4175. Rome, Italy (2007). DOI 10.1109/ROBOT.2007.364120. URL See also expanded version at
  8. 8.
    Censi, A.: On achievable accuracy for pose-tracking. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA) (2009).URL
  9. 9.
    Chandran-Ramesh, M., Newman, P.: Assessing Map Quality using Conditional Random Fields.k In: Proceedings of the Field and Service Robotics (2007)Google Scholar
  10. 10.
    Crisan, D., Doucet, A.: A survey of convergence results on particle filtering for practitioners. IEEE Transactions on Signal Processing 50(3), 736–746 (2002).URL Scholar
  11. 11.
    Fox, D.: Adapting the sample size in particle filters through KLD-sampling. International Journal of Robotics Research 22(12) (2003)Google Scholar
  12. 12.
    Frese, U.: A discussion of simultaneous localization and mapping. Autonomous Robots 20(1), 25–42 (2006)CrossRefGoogle Scholar
  13. 13.
    Gonzalez, R., Woods, R., Eddins, S.:  Chapter 11, pp. 714–725 (2003)
  14. 14.
    Grisetti, G., Stachniss, C., Burgard, W.: Improving grid-based slam with rao-blackwellized particle filters by adaptive proposals and selective resampling. ICRA (2005)Google Scholar
  15. 15.
    Harris, C., Stephens, M.: A combined corner and edge detection. In: Proceedings of The Fourth Alvey Vision Conference, pp. 147–151 (1988).Google Scholar
  16. 16.
    Hough, P.V.C.: Methods and means for recognizing complex patterns. US patent 3,069,654 (1962)Google Scholar
  17. 17.
    Huang, G., Mourikis, A., Roumeliotis, S.: Analysis and improvement of the consistency of Extended Kalman Filter based SLAM. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA), pp. 473–479 (2008). DOI 10.1109/ROBOT.2008.4543252Google Scholar
  18. 18.
    Huang, S., Dissanayake, G.: Convergence analysis for extended kalman filter based slam. IEEE International Conference on Robotics and Automation (2006)Google Scholar
  19. 19.
    Huang, S., Dissanayake, G.: Convergence and consistency analysis for Extended Kalman Filter based SLAM. IEEE Transactions on Robotics 23(5), 1036–1049 (2007). DOI 10.1109/TRO.2007.903811CrossRefGoogle Scholar
  20. 20.
    Illingworth, J., Kittler, J.: A Survey of the Hough Transform. Computer Vision, Graphics, and Image Processing 44(1), 87–116 (1988)CrossRefGoogle Scholar
  21. 21.
    Jacobs Map Analysis Toolkit.
  22. 22.
    Joshi, S., Kaziska, D., Srivastava, A., Mio, W.: Riemannian structures on shape spaces: A framework for statistical inferences. Statistics and Analysis of Shapes pp. 313–333 (2006)Google Scholar
  23. 23.
    Lagunovsky, D.S.A.: Fast line and rectangle detection by clustering and grouping. Proc. of CAIP’97, Kiel, Germany (1997)Google Scholar
  24. 24.
    Lakaemper, R., Adluru, N., Latecki, L., Madhavan, R.: Multi robot mapping using force field simulation. Journal of Field Robotics, Special Issue on Quantitative Performance Evaluation of Robotic and Intelligent Systems (2007)Google Scholar
  25. 25.
    Le, H., Kendall, D.G.: The Riemannian structure of Euclidean shape spaces: A novel environment for statistics. Annals of Statistics 21(3), 1225–1271 (1993)MATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Lowe, D.G.: Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  27. 27.
    Madhavan, R., Scrapper, C., Kleiner, A.e.: Quantitative Performance Evaluation of Navigation Solutions for Mobile Robots. Workshop Proceedings, Robotics: Science and Systems (RSS) Conference, Zurich, Switzerland;∼rss/(2008)
  28. 28.
    Michor, P.W., Mumford, D.: Riemannian geometries on spaces of plane curves. Journal of the European Mathematical Society 8, 1–48 (2003)MathSciNetGoogle Scholar
  29. 29.
    Mourikis, A.: Personal communication (2008)Google Scholar
  30. 30.
    Mourikis, A., Roumeliotis, S.: On the treatment of relative-pose measurements for mobile robot localization. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA). Orlando, FL (2006)Google Scholar
  31. 31.
    Nuechter, A., Lingemann, K., Hertzberg, J., Surmann, H., Pervoelz, K., Hennig, M., Tiruchinapalli, K., Worst, R., Christaller, T.: Mapping of rescue environments with kurt3d. Proceedings of the International Workshop on Safety, Security and Rescue Robotics (SSRR ’05),Kobe, Japan (2005)Google Scholar
  32. 32.
  33. 33.
    Radish: The Robotics Data Set Repository.
  34. 34.
  35. 35.
    Scrapper, C., Madhavan, R., Balakirsky, S.: Stable Navigation Solutions for Robots in Complex Environments. In: IEEE International Workshop on Safety, Security, and Rescue Robotics (SSRR2007) (2007)Google Scholar
  36. 36.
    Scrapper, C., Madhavan, R., Balakirsky, S.: Performance analysis for stable mobile robot navigation solutions. pp. 696206-1-696206-12. SPIE (2008)Google Scholar
  37. 37.
    Scrapper, C., Madhavan, R., Balakirsky, S.O.: Special Session on ’Quantitative Assessment of Robot-generated Maps’. Proceedings of the Performance Metrics for Intelligent Systems (PerMIS) Workshop, R. Madhavan and E. Messina (eds.); (2008)
  38. 38.
    Thrun, S.: Learning Metric-Topological Maps for Indoor Mobile Robot Navigation. Artificial Intelligence pp. 21–71 (1998)Google Scholar
  39. 39.
    Trajkovic, M., Hedley, M.: Fast corner detection”. Image and Vision Computing 16, 75–87 (1998)CrossRefGoogle Scholar
  40. 40.
    Trees, H.L.V., Bell, K.L.: Bayesian Bounds for Parameter Estimation and Nonlinear Filtering/ Tracking. Wiley-IEEE Press (2007)Google Scholar
  41. 41.
    Varsadan, I., Birk, A. and Pfingsthorn, M.: Determining Map Quality through an Image Similarity Metric. In: Proceedings of the RoboCup Symposium (2008)Google Scholar
  42. 42.
    Wang, Z., Bovik, A.: A Universal Image Quality Index. In: IEEE Signal Processing Letters, vol. 9, pp. 81–84 (2002)Google Scholar
  43. 43.
    Wang, Z., Bovik, A.C.: Why is Image Quality Assessment so Difficult. In: in Proc. IEEE Int. Conf. Acoust., Speech, and Signal Processing, pp. 3313–3316 (2002)Google Scholar
  44. 44.
    Weingarten, J., Siegwart, R.: EKF-based 3D SLAM for Structured Environment Reconstruction. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3834–3839 (2005)Google Scholar
  45. 45.
    Zhou, W., Miro, J., Dissanayake, G.: Information Efficient 3D Visual SLAM in Unstructured Domains. In: Proceedings of the International Conference on Intelligent Sensors, Sensor Networks and Information, pp. 323–328 (2007)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • C. Scrapper
    • 1
  • R. Madhavan
    • 2
    • 3
  • R. Lakaemper
    • 4
  • A. Censi
    • 5
  • A. Godil
    • 6
  • A. Wagan
    • 6
  • A. Jacoff
    • 3
  1. 1.The MITRE CorporationMcLeanUSA
  2. 2.Computational Sciences and Engineering DivisionOak Ridge National LaboratoryOak RidgeUSA
  3. 3.Intelligent Systems DivisionNational Institute of Standards and Technology (NIST)GaithersburgUSA
  4. 4.Department of Computer and Information SciencesTemple UniversityPhiladelphiaUSA
  5. 5.Control&Dynamical Systems, California Institute of TechnologyPasadenaUSA
  6. 6.Information Technology LaboratoryNational Institute of Standards and Technology (NIST)GaithersburgUSA

Personalised recommendations