Skip to main content

Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup

Abstract

This paper presents the map evaluation methodology developed for the Virtual Robots Rescue competition held as part of RoboCup. The procedure aims to evaluate the quality of maps produced by multi-robot systems with respect to a number of factors, including usability, exploration, annotation and other aspects relevant to robots and first responders. In addition to the design choices, we illustrate practical examples of maps and scores coming from the latest RoboCup contest, outlining strengths and weaknesses of our modus operandi. We also show how a benchmarking methodology developed for a simulation testbed effortlessly and faithfully transfers to maps built by a real robot. A number of conclusions may be derived from the experience reported in this paper and a thorough discussion is offered.

References

  • Abdallah, S. M., Smar, D. C., & Zelek, J. S. (2007). A benchmark for outdoor vision SLAM systems. Journal of Field Robotics, 24(1/2), 145–165.

    Article  Google Scholar 

  • Amigoni, F. (2008). Experimental evaluation of some exploration strategies for mobile robots. In IEEE international conference on robotics and automation (pp. 2818–2823).

  • Amigoni, F., & Gallo, A. (2005). A multi-objective exploration strategy for mobile robots. In IEEE international conference on robotics and automation (pp. 3850–3855).

  • Balaguer, B., & Carpin, S. (2008a). Uc mercenary team description paper: Robocup 2008 virtual robot rescue simulation league. In Team description paper for RoboCup 2008.

  • Balaguer, B., Carpin, S. (2008b). Where Am I? A Simulated GPS Sensor for Outdoor Robotic Applications. In S. Carpin et al. (Eds.), Proceedings of the first international conference on simulation, modeling and programming for autonomous robots (pp. 222–233). Berlin: Springer.

    Chapter  Google Scholar 

  • Balaguer, B., Carpin, S., & Balakirsky, S. (2007). Towards quantitative comparisons of robot algorithms: Experiences with slam in simulation and real world systems. In Workshop on performance evaluation and benchmarking for intelligent robots and systems at IEEE/RSJ IROS.

  • Balakirsky, S., Carpin, S., Kleiner, A., Lewis, M., Visser, A., Wang, J., & Ziparo, V. A. (2007). Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from robocup rescue. Journal of Field Robotics, 24(11–12), 943–967.

    Article  Google Scholar 

  • Baltes, J. (2000). A benchmark suite for mobile robots. In Proceedings of the IEEE/RSJ conference on intelligent robots and systems (pp. 1101–1106).

  • Baron, R. J. (1981). Mechanisms of human facial recognition. International Journal of Man Machine Studies, 15, 137–178.

    Article  Google Scholar 

  • Basilico, N., & Amigoni, F. (2008). On evaluating performance of exploration strategies for autonomous mobile robots. In Proceedings of the performance evaluation and benchmarking for intelligent robots and systems workshop at IEEE/RSJ international conference on intelligent robots and systems.

  • Birk, A. (1996). Learning geometric concepts with an evolutionary algorithm. In The fifth annual conference on evolutionary programming.

  • Birk, A., Poppinga, J., Stoyanov, T., & Nevatia, Y. (2009). Planetary exploration in USARsim: a case study including real world data from Mars. In L. Iocchi, H. Matsubara, A. Weitzenfeld, & C. Zhou (Eds.), Lecture notes in artificial intelligence (LNAI). RoboCup 2008: Robot WorldCup XII. Berlin: Springer.

    Google Scholar 

  • Calisi, D., Iocchi, L., & Nardi, D. (2008). A unified benchmark framework for autonomous mobile robots and vehicles motion algorithms (movema benchmarks). In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008).

  • Carlson, J., Murphy, R., Christopher, S., & Casper, J. (2005). Conflict metric as a measure of sensing quality. In IEEE international conference on robotics and automation.

  • Carpin, S., Stoyanov, T., Nevatia, Y., Lewis, M., & Wang, J. (2006a). Quantitative assessments of usarsim accuracy. In Proceedings of the performance metrics for intelligent systems workshop.

  • Carpin, S., Wang, J., Lewis, M., Birk, A., & Jacoff, A. (2006b). High fidelity tools for rescue robotics: results and perspectives. In LNCS. Robocup 2005: Robot Soccer World Cup IX (pp. 301–311). Berlin: Springer.

    Chapter  Google Scholar 

  • Carpin, S., Lewis, M., Wang, J., Balakirsky, S., & Scrapper, C. (2007a). Bridging the gap between simulation and reality in urban search and rescue. In LNCS : Vol. 4434. Robocup 2006: Robot Soccer World Cup X (pp. 1–12). Berlin: Springer.

    Chapter  Google Scholar 

  • Carpin, S., Lewis, M., Wang, J., Balakirsky, S., & Scrapper, C. (2007b). USARSim: a robot simulator for research and education. In Proceedings of the IEEE international conference on robotics and automation (pp. 1400–1405).

  • Collins, J. J., & O’Sullivan, S. (2004). Developing a benchmarking framework for map building paradigms. In Ninth international symposium on artificial life and robots.

  • Collins, T., Collins, J. J., Mansfield, M., & O’Sullivan, S. (2005). Evaluating techniques for resolving redundant information and specularity in occupancy grids. In S. Zhang, & R. Jarvis (Eds.), Proceedings of AI 2006: advances in artificial intelligence (pp. 235–244). Berlin: Springer.

    Chapter  Google Scholar 

  • Collins, T., Collins, J. J., & Ryan, C. (2007). Occupancy grid mapping: and empirical evaluation. In Proceedings of the Mediterranean conference on control and automation (pp. 1–6).

  • Egerton, S. J., & Callaghan, V. (2004). A benchmark for measuring mobile robot environment modelling performacne. In Proceedings of the IEEE conference on robotics, automation and mechatronics (pp. 407–412).

  • Fontana, G., Matteucci, M., & Sorrenti, D. G. (2008). The RAWSEED proposal for representation-independent benchmarking of SLAM. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008).

  • Frontoni, E., Ascani, A., Mancini, A., & Zingaretti, P. (2008). Performance metric for vision based robot localization. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008).

  • The GeoTIFF website (2009). http://trac.osgeo.org/geotiff.

  • Grisetti, G., Stachniss, C., & Burgard, W. (2007). Improved techniques for grid mapping iwht Rao-Blackwellized particle filters. IEEE Transactions on Robotics, 23(1), 36–46.

    Article  Google Scholar 

  • Guyon, I., Gunn, S., Nikravesh, M., & Zadeh, L. (2006). Feature extraction, foundations and applications. Series studies in fuzziness and soft computing. Berlin: Springer.

    MATH  Book  Google Scholar 

  • Jacoff, A., Messina, E., Weiss, B., Tadokoro, S., & Nakagawa, Y. (2003). Test arenas and performance metrics for urban search and rescue robots. In International conference on intelligent robots and systems (pp. 3396–3403).

  • Kuipers, B. (1978). Modeling spatial knowledge. Cognitive Science, 2, 129–153.

    Article  Google Scholar 

  • Martin, M. C., & Moravec, H. P. (1996). Robot evidence grids. Technical Report CMU-RI-TR-96-06, Robotics Institute–Carnegie Mellon University, March 1996.

  • Michel, O., Bourquin, Y., & Baillie, J. C. (2008). Robotstadium: Online humanoid robot soccer simulation competition. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008).

  • MOAST project (2009). http://sourceforge.net/projects/moast.

  • Moravec, H. P. (1988). Sensor fusion in certainty grids for mobile robots. AI Magazine, 9(3), 61–74.

    Google Scholar 

  • OpenSlam (2009). http://www.openslam.org.

  • O’Sullivan, S. (2003). An empirical evaluation of map building methodologies in mobile robotics using the feature prediction sonar noise filter and metric grip map benchmarking suite. Master’s thesis, University of Limerick.

  • O’Sullivan, S., Collins, J. J., Mansfield, M., Haskett, D., & Eaton, M. (2004a). Linear feature prediction for confidence estimation of sonar readings in map building. In Ninth international symposium on artificial life and robots.

  • O’Sullivan, S., Collins, J. J., Mansfield, M., Haskett, D., & Eaton, M. (2004b). A quantitative evaluation of sonar models and mathematical update methods for map building with mobile robots. In Ninth international symposium on artificial life and robots.

  • Radish—the robotics data set repository (2009). http://radish.sourceforge.net.

  • Sato, K., Yotsukura, S., & Takahashi, T. (2008). To a rescue simulation applicable to anywhere. Team Description Hinomiyagura, July 2008. Proceedings CD.

  • Schmits, T., & Visser, A. (2008). An omnidirectional camera simulation for the USARSim World. In Proceedings of the 12th RoboCup international symposium, July 2008. Proceedings CD. To be published in the Lecture Notes on Artificial Intelligence series.

  • Sim, R., Dudek, G., & Roy, N. (2004). Online control policy optimization for minimizing map uncertainty during exploration. In IEEE international conference on robotics and automation.

  • Stachniss, C., & Burgard, W. (2003). Exploring unknown environments with mobile robots using coverage maps. In 18th joint conference on artificial intelligence.

  • Varsadan, I., Birk, A., Pingsthorn, M., Schwertfeger, S., & Pathak, K. (2008). The Jacobs map analysis toolkit. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008).

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Stefano Carpin.

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Balaguer, B., Balakirsky, S., Carpin, S. et al. Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup. Auton Robot 27, 449 (2009). https://doi.org/10.1007/s10514-009-9141-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10514-009-9141-z

Keywords

  • Urban search and rescue
  • Multi-robot systems
  • Simultaneous localization and mapping
  • RoboCup
  • Robot benchmarking