Rawseeds: Building a Benchmarking Toolkit for Autonomous Robotics

  • Giulio Fontana
  • Matteo Matteucci
  • Domenico G. Sorrenti
Part of the SpringerBriefs in Applied Sciences and Technology book series (BRIEFSAPPLSCIENCES)


Within computer science, autonomous robotics takes the uneasy role of a discipline where the features of both systems (i.e., robots) and their operating environment (i.e., the physical world) conspire to make the application of the experimental scientific method most difficult. This is the reason why much experimental work in robotics is, from the methodological point of view, built on shaky grounds. In particular, scientifically sound benchmarking tools are still largely missing. This chapter starts from Rawseeds, a project focused precisely on benchmarking in robotics, to highlight the reasons for these difficulties and to propose strategies for overcoming some of them. The main result of Rawseeds is a Benchmarking Toolkit: a readily usable instrument to assess and compare algorithms for SLAM, localization, and mapping. Its most innovative aspects include a set of high-quality, validated, multi-sensor datasets, collected both in indoor and in outdoor locations and complemented by ground truth data, and the explicit definition of a set of quantitative performance metrics for the evaluation of algorithms.


Robotic datasets Benchmarking SLAM Ground truth 


  1. 1.
    Bonsignorio F, Hallam J, del Pobil AP (2007) Good experimental methodology - GEM guidelines. http://www.heronrobots.com/EuronGEMSig/downloads/GemSigGuidelinesBeta.pdf. Accessed July 2013
  2. 2.
    PerMIS (2010) Performance metrics for intelligent systems. http://www.nist.gov/el/isd/ks/permis.cfm, Accessed July 2013
  3. 3.
    Howard A, Roy N (2003) The robotics data set repository (radish). http://www.rawseeds.org/. Accessed July 2013
  4. 4.
    Rawseeds (2006) http://radish.sourceforge.net/. Accessed July 2013
  5. 5.
    Zinas N (2011) GPS network real time kinematic tutorial. Tech. Rep. TEKMON-001, Tekmon Geomatics LLP. http://tekmon.gr/tekmon-research/gps-network-rtk-tutorial/. Accessed July 2013
  6. 6.
    Ceriani S, Fontana G, Giusti A, Marzorati D, Matteucci M, Migliore D, Rizzi D, Sorrenti D, Taddei P (2009) Rawseeds ground truth collection systems for indoor self-localization and mapping. Auton Robot 27(4):353–371CrossRefGoogle Scholar
  7. 7.
    Jacoff A, Messina E, Evans J (2002) Experiences in deploying test arenas for autonomous mobile robots. In: Proc 2011 PerMIS, Workshop, pp 87–94Google Scholar
  8. 8.
    Jacoff A, Messina E, Weiss BA, Tadokoro S, Nakagawa Y (2003) Test arenas and performance metrics for urban search and rescue robots. In: Proc 2003 IEEE/RSJ IROS, vol 3, pp 3396–3403Google Scholar

Copyright information

© The Author(s) 2014

Authors and Affiliations

  • Giulio Fontana
    • 1
  • Matteo Matteucci
    • 1
  • Domenico G. Sorrenti
    • 2
  1. 1.Dipartimento di Elettronica, Informazione e BioingegneriaPolitecnico di MilanoMilanItaly
  2. 2.Dipartimento di Informatica, Sistemistica e ComunicazioneUniversità degli Studi di Milano-BicoccaMilanItaly

Personalised recommendations