Benchmarking Functionalities of Domestic Service Robots Through Scientific Competitions
- 11 Downloads
Benchmarking via carefully designed competitions makes it possible to provide a common framework for the rigorous comparison of intelligent and autonomous systems; competitions may play the role of scientific experiments while being appealing both to researchers and to the general public thus promoting critical analysis of systems outside the labs. This paper describes our approach to benchmarking domestic service robots through organizing recurrent competitions under the European Robotics League. It details the tools and benchmarks designed to evaluate the performance of robots at task and functionality levels. In particular, the functionality benchmarks for object perception and navigation are described and an overview of the new benchmarks to appear in the league is presented.
KeywordsVerification of autonomous systems System-level AI for robots Benchmarking Competitions Domestic service robots Object perception ERL Navigation Task and functionality benchmarks
This research was supported by EU Horizon 2020 Programme for research, technological development and demonstration under Grants agreements n688441 and n780086.
- 1.Amigoni F, Bastianelli E, Berghofer J, Bonarini A, Fontana G, Hochgeschwender N, Iocchi L, Kraetzschmar GK, Lima PU, Matteucci M, Miraldo M, Nardi D, Schiaffonati V (2015) Competitions for benchmarking: task and functionality scoring complete performance assessment. IEEE Robot Autom Mag 22(3):53–61CrossRefGoogle Scholar
- 2.Lima PU (2017) RoCKIn: benchmarking through robot competitions. InTechOpen, RijekaGoogle Scholar
- 7.Müller AC, Behnke S (2014) Learning depth-sensitive conditional random fields for semantic segmentation of rgb-d images. In: IEEE international conference on robotics and automation (ICRA), pp 6232-6237Google Scholar
- 8.Messias J, Ventura R, Lima P, Sequeira J, Alvito P, Marques C, Carriço P (2014) A robotic platform for edutainment activities in a pediatric hospital. In: IEEE international conference on autonomous robot systems and competitions (ICARSC), pp 193-198Google Scholar
- 9.Muszynski S, Stückler J, Behnke S (2012) Adjustable autonomy for mobile teleoperation of personal service robots. In: IEEE international symposium on robot and human interactive communication (Ro-MAN), pp 933–940Google Scholar
- 11.RoCKIn: Robot competitions kick innovation in cognitive systems. http://rockinrobotchallenge.eu. Accessed Dec 2018
- 12.euRathlon: An outdoor robotics challenge for land, sea and air. https://www.eurathlon.eu. Accessed Dec 2018
- 13.European Robotics League. https://www.eu-robotics.net/robotics_league/. Accessed Dec 2018
- 14.Basiri M et al (2018) Rulebook of the European robotic league for consumer service robots. https://eu-robotics.net/robotics_league/erl-consumer