Autonomous Robots

, 27:449 | Cite as

Evaluating maps produced by urban search and rescue robots: lessons learned from RoboCup

  • Benjamin Balaguer
  • Stephen Balakirsky
  • Stefano Carpin
  • Arnoud Visser
Open Access
Article

Abstract

This paper presents the map evaluation methodology developed for the Virtual Robots Rescue competition held as part of RoboCup. The procedure aims to evaluate the quality of maps produced by multi-robot systems with respect to a number of factors, including usability, exploration, annotation and other aspects relevant to robots and first responders. In addition to the design choices, we illustrate practical examples of maps and scores coming from the latest RoboCup contest, outlining strengths and weaknesses of our modus operandi. We also show how a benchmarking methodology developed for a simulation testbed effortlessly and faithfully transfers to maps built by a real robot. A number of conclusions may be derived from the experience reported in this paper and a thorough discussion is offered.

Keywords

Urban search and rescue Multi-robot systems Simultaneous localization and mapping RoboCup Robot benchmarking 

References

  1. Abdallah, S. M., Smar, D. C., & Zelek, J. S. (2007). A benchmark for outdoor vision SLAM systems. Journal of Field Robotics, 24(1/2), 145–165. CrossRefGoogle Scholar
  2. Amigoni, F. (2008). Experimental evaluation of some exploration strategies for mobile robots. In IEEE international conference on robotics and automation (pp. 2818–2823). Google Scholar
  3. Amigoni, F., & Gallo, A. (2005). A multi-objective exploration strategy for mobile robots. In IEEE international conference on robotics and automation (pp. 3850–3855). Google Scholar
  4. Balaguer, B., & Carpin, S. (2008a). Uc mercenary team description paper: Robocup 2008 virtual robot rescue simulation league. In Team description paper for RoboCup 2008. Google Scholar
  5. Balaguer, B., Carpin, S. (2008b). Where Am I? A Simulated GPS Sensor for Outdoor Robotic Applications. In S. Carpin et al. (Eds.), Proceedings of the first international conference on simulation, modeling and programming for autonomous robots (pp. 222–233). Berlin: Springer. CrossRefGoogle Scholar
  6. Balaguer, B., Carpin, S., & Balakirsky, S. (2007). Towards quantitative comparisons of robot algorithms: Experiences with slam in simulation and real world systems. In Workshop on performance evaluation and benchmarking for intelligent robots and systems at IEEE/RSJ IROS. Google Scholar
  7. Balakirsky, S., Carpin, S., Kleiner, A., Lewis, M., Visser, A., Wang, J., & Ziparo, V. A. (2007). Towards heterogeneous robot teams for disaster mitigation: Results and performance metrics from robocup rescue. Journal of Field Robotics, 24(11–12), 943–967. CrossRefGoogle Scholar
  8. Baltes, J. (2000). A benchmark suite for mobile robots. In Proceedings of the IEEE/RSJ conference on intelligent robots and systems (pp. 1101–1106). Google Scholar
  9. Baron, R. J. (1981). Mechanisms of human facial recognition. International Journal of Man Machine Studies, 15, 137–178. CrossRefGoogle Scholar
  10. Basilico, N., & Amigoni, F. (2008). On evaluating performance of exploration strategies for autonomous mobile robots. In Proceedings of the performance evaluation and benchmarking for intelligent robots and systems workshop at IEEE/RSJ international conference on intelligent robots and systems. Google Scholar
  11. Birk, A. (1996). Learning geometric concepts with an evolutionary algorithm. In The fifth annual conference on evolutionary programming. Google Scholar
  12. Birk, A., Poppinga, J., Stoyanov, T., & Nevatia, Y. (2009). Planetary exploration in USARsim: a case study including real world data from Mars. In L. Iocchi, H. Matsubara, A. Weitzenfeld, & C. Zhou (Eds.), Lecture notes in artificial intelligence (LNAI). RoboCup 2008: Robot WorldCup XII. Berlin: Springer. Google Scholar
  13. Calisi, D., Iocchi, L., & Nardi, D. (2008). A unified benchmark framework for autonomous mobile robots and vehicles motion algorithms (movema benchmarks). In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008). Google Scholar
  14. Carlson, J., Murphy, R., Christopher, S., & Casper, J. (2005). Conflict metric as a measure of sensing quality. In IEEE international conference on robotics and automation. Google Scholar
  15. Carpin, S., Stoyanov, T., Nevatia, Y., Lewis, M., & Wang, J. (2006a). Quantitative assessments of usarsim accuracy. In Proceedings of the performance metrics for intelligent systems workshop. Google Scholar
  16. Carpin, S., Wang, J., Lewis, M., Birk, A., & Jacoff, A. (2006b). High fidelity tools for rescue robotics: results and perspectives. In LNCS. Robocup 2005: Robot Soccer World Cup IX (pp. 301–311). Berlin: Springer. CrossRefGoogle Scholar
  17. Carpin, S., Lewis, M., Wang, J., Balakirsky, S., & Scrapper, C. (2007a). Bridging the gap between simulation and reality in urban search and rescue. In LNCS : Vol. 4434. Robocup 2006: Robot Soccer World Cup X (pp. 1–12). Berlin: Springer. CrossRefGoogle Scholar
  18. Carpin, S., Lewis, M., Wang, J., Balakirsky, S., & Scrapper, C. (2007b). USARSim: a robot simulator for research and education. In Proceedings of the IEEE international conference on robotics and automation (pp. 1400–1405). Google Scholar
  19. Collins, J. J., & O’Sullivan, S. (2004). Developing a benchmarking framework for map building paradigms. In Ninth international symposium on artificial life and robots. Google Scholar
  20. Collins, T., Collins, J. J., Mansfield, M., & O’Sullivan, S. (2005). Evaluating techniques for resolving redundant information and specularity in occupancy grids. In S. Zhang, & R. Jarvis (Eds.), Proceedings of AI 2006: advances in artificial intelligence (pp. 235–244). Berlin: Springer. CrossRefGoogle Scholar
  21. Collins, T., Collins, J. J., & Ryan, C. (2007). Occupancy grid mapping: and empirical evaluation. In Proceedings of the Mediterranean conference on control and automation (pp. 1–6). Google Scholar
  22. Egerton, S. J., & Callaghan, V. (2004). A benchmark for measuring mobile robot environment modelling performacne. In Proceedings of the IEEE conference on robotics, automation and mechatronics (pp. 407–412). Google Scholar
  23. Fontana, G., Matteucci, M., & Sorrenti, D. G. (2008). The RAWSEED proposal for representation-independent benchmarking of SLAM. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008). Google Scholar
  24. Frontoni, E., Ascani, A., Mancini, A., & Zingaretti, P. (2008). Performance metric for vision based robot localization. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008). Google Scholar
  25. The GeoTIFF website (2009). http://trac.osgeo.org/geotiff.
  26. Grisetti, G., Stachniss, C., & Burgard, W. (2007). Improved techniques for grid mapping iwht Rao-Blackwellized particle filters. IEEE Transactions on Robotics, 23(1), 36–46. CrossRefGoogle Scholar
  27. Guyon, I., Gunn, S., Nikravesh, M., & Zadeh, L. (2006). Feature extraction, foundations and applications. Series studies in fuzziness and soft computing. Berlin: Springer. MATHCrossRefGoogle Scholar
  28. Jacoff, A., Messina, E., Weiss, B., Tadokoro, S., & Nakagawa, Y. (2003). Test arenas and performance metrics for urban search and rescue robots. In International conference on intelligent robots and systems (pp. 3396–3403). Google Scholar
  29. Kuipers, B. (1978). Modeling spatial knowledge. Cognitive Science, 2, 129–153. CrossRefGoogle Scholar
  30. Martin, M. C., & Moravec, H. P. (1996). Robot evidence grids. Technical Report CMU-RI-TR-96-06, Robotics Institute–Carnegie Mellon University, March 1996. Google Scholar
  31. Michel, O., Bourquin, Y., & Baillie, J. C. (2008). Robotstadium: Online humanoid robot soccer simulation competition. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008). Google Scholar
  32. MOAST project (2009). http://sourceforge.net/projects/moast.
  33. Moravec, H. P. (1988). Sensor fusion in certainty grids for mobile robots. AI Magazine, 9(3), 61–74. Google Scholar
  34. OpenSlam (2009). http://www.openslam.org.
  35. O’Sullivan, S. (2003). An empirical evaluation of map building methodologies in mobile robotics using the feature prediction sonar noise filter and metric grip map benchmarking suite. Master’s thesis, University of Limerick. Google Scholar
  36. O’Sullivan, S., Collins, J. J., Mansfield, M., Haskett, D., & Eaton, M. (2004a). Linear feature prediction for confidence estimation of sonar readings in map building. In Ninth international symposium on artificial life and robots. Google Scholar
  37. O’Sullivan, S., Collins, J. J., Mansfield, M., Haskett, D., & Eaton, M. (2004b). A quantitative evaluation of sonar models and mathematical update methods for map building with mobile robots. In Ninth international symposium on artificial life and robots. Google Scholar
  38. Radish—the robotics data set repository (2009). http://radish.sourceforge.net.
  39. Sato, K., Yotsukura, S., & Takahashi, T. (2008). To a rescue simulation applicable to anywhere. Team Description Hinomiyagura, July 2008. Proceedings CD. Google Scholar
  40. Schmits, T., & Visser, A. (2008). An omnidirectional camera simulation for the USARSim World. In Proceedings of the 12th RoboCup international symposium, July 2008. Proceedings CD. To be published in the Lecture Notes on Artificial Intelligence series. Google Scholar
  41. Sim, R., Dudek, G., & Roy, N. (2004). Online control policy optimization for minimizing map uncertainty during exploration. In IEEE international conference on robotics and automation. Google Scholar
  42. Stachniss, C., & Burgard, W. (2003). Exploring unknown environments with mobile robots using coverage maps. In 18th joint conference on artificial intelligence. Google Scholar
  43. Varsadan, I., Birk, A., Pingsthorn, M., Schwertfeger, S., & Pathak, K. (2008). The Jacobs map analysis toolkit. In Workshop on experimental methodology and benchmarking in robotics research (RSS 2008). Google Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  • Benjamin Balaguer
    • 1
  • Stephen Balakirsky
    • 2
  • Stefano Carpin
    • 1
  • Arnoud Visser
    • 3
  1. 1.School of EngineeringUniversity of California, MercedMercedUSA
  2. 2.Intelligent Systems DivisionNational Institute of Standards and TechnologyGaithersburgUSA
  3. 3.Intelligent System Laboratory AmsterdamUniversiteit van AmsterdamAmsterdamThe Netherlands

Personalised recommendations