Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions

  • Meysam BasiriEmail author
  • Enrico Piazza
  • Matteo Matteucci
  • Pedro Lima
Technical Contribution


Benchmarking via carefully designed competitions makes it possible to provide a common framework for the rigorous comparison of intelligent and autonomous systems; competitions may play the role of scientific experiments while being appealing both to researchers and to the general public thus promoting critical analysis of systems outside the labs. This paper describes our approach to benchmarking domestic service robots through organizing recurrent competitions under the European Robotics League. It details the tools and benchmarks designed to evaluate the performance of robots at task and functionality levels. In particular, the functionality benchmarks for object perception and navigation are described and an overview of the new benchmarks to appear in the league is presented.


Verification of autonomous systems System-level AI for robots Benchmarking Competitions Domestic service robots Object perception ERL Navigation Task and functionality benchmarks 



This research was supported by EU Horizon 2020 Programme for research, technological development and demonstration under Grants agreements n688441 and n780086.


  1. 1.
    Amigoni F, Bastianelli E, Berghofer J, Bonarini A, Fontana G, Hochgeschwender N, Iocchi L, Kraetzschmar GK, Lima PU, Matteucci M, Miraldo M, Nardi D, Schiaffonati V (2015) Competitions for benchmarking: task and functionality scoring complete performance assessment. IEEE Robot Autom Mag 22(3):53–61CrossRefGoogle Scholar
  2. 2.
    Lima PU (2017) RoCKIn: benchmarking through robot competitions. InTechOpen, RijekaGoogle Scholar
  3. 3.
    Dias J, Althoefer K, Lima PU (2016) Robot competitions: what did we learn? IEEE Robot Autom Mag 23(1):16–18CrossRefGoogle Scholar
  4. 4.
    Anderson M, Jenkins OC, Osentoski S (2011) Recasting robotics challenges as experiments. IEEE Robot Autom Mag 18(2):10–11CrossRefGoogle Scholar
  5. 5.
    Prassler E, Munich ME, Pirjanian P, Kosuge K (2016) Domestic robotics. Springer handbook of robotics. Springer, Cham, pp 1729–1758CrossRefGoogle Scholar
  6. 6.
    Young JE, Hawkins R, Sharlin E, Igarashi T (2009) Toward acceptable domestic robots: applying insights from social psychology. Int J Soc Robot 1(1):95CrossRefGoogle Scholar
  7. 7.
    Müller AC, Behnke S (2014) Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. In: IEEE international conference on robotics and automation (ICRA), pp 6232-6237Google Scholar
  8. 8.
    Messias J, Ventura R, Lima P, Sequeira J, Alvito P, Marques C, Carriço P (2014) A robotic platform for edutainment activities in a pediatric hospital. In: IEEE international conference on autonomous robot systems and competitions (ICARSC), pp 193-198Google Scholar
  9. 9.
    Muszynski S, Stückler J, Behnke S (2012) Adjustable autonomy for mobile teleoperation of personal service robots. In: IEEE international symposium on robot and human interactive communication (Ro-MAN), pp 933–940Google Scholar
  10. 10.
    Pineda LA, Salinas L, Meza IV, Rascon C, Fuentes G (2013) Sitlog: a programming language for service robot tasks. Int J Adv Rob Syst 10(10):358CrossRefGoogle Scholar
  11. 11.
    RoCKIn: Robot competitions kick innovation in cognitive systems. Accessed Dec 2018
  12. 12.
    euRathlon: An outdoor robotics challenge for land, sea and air. Accessed Dec 2018
  13. 13.
    European Robotics League. Accessed Dec 2018
  14. 14.
    Basiri M et al (2018) Rulebook of the European robotic league for consumer service robots.

Copyright information

© Gesellschaft für Informatik e.V. and Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute For Systems and Robotics, Tecnico LisboaLisbonPortugal
  2. 2.Politecnico di MilanoMilanItaly

Personalised recommendations