RoboCup@Home: Results in Benchmarking Domestic Service Robots

  • Thomas Wisspeintner
  • Tijn van der Zan
  • Luca Iocchi
  • Stefan Schiffer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5949)


Benchmarking robotic technologies is of utmost importance for actual deployment of robotic applications in industrial and every-day environments, therefore many efforts have recently focused on this problem. Among the many different ways of benchmarking robotic systems, scientific competitions are recognized as one of the most effective ways of rapid development of scientific progress in a field. The RoboCup@Home league targets the development and deployment of autonomous service and assistive robot technology, being essential for future personal domestic applications, and offers an important approach to benchmarking domestic and service robots.

In this paper we present the new methodology for benchmarking DSR adopted in RoboCup@Home, that includes the definition of multiple benchmarks (tests) and of performance metrics based on the relationships between key abilities required to the robots and the tests. We also discuss the results of our benchmarking approach over the past years and provide an outlook on short- and mid-term goals of Home and of DSR in general.


Speech Recognition Functional Ability System Property Service Robot Robotic Application 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    van der Zant, T., Wisspeintner, T.: RoboCup@Home: Creating and Benchmarking Tomorrows Service Robot Applications. In: Robotic Soccer, pp. 521–528. I-Tech Education and Publishing (2007)Google Scholar
  2. 2.
    del Pobil, A.: Why do We Need Benchmarks in Robotics Research? In: Proc. of the Workshop on Benchmarks in Robotics Research, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006)Google Scholar
  3. 3.
    Sabanovic, S., Michalowski, M., Simmons, R.: Robots in the wild: observing human-robot social interaction outside the lab. In: 9th IEEE International Workshop on Advanced Motion Control, pp. 596–601 (2006)Google Scholar
  4. 4.
    Balch, T., Yanco, H.A.: Ten years of the aaai mobile robot competition and exhibition: looking back and to the future. AI Magazine 23(1), 13–22 (2002)Google Scholar
  5. 5.
    Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Matsubara, H.: RoboCup: A Challenge Problem for AI. AI Magazine 18(1), 73–85 (1997)Google Scholar
  6. 6.
    The RoboCup Federation: RoboCup (2008), (Retrieved 09/2008)
  7. 7.
    Meeden, L., Schultz, A.C., Balch, T.R., Bhargava, R., Haigh, K.Z., Bohlen, M., Stein, C., Miller, D.P.: The aaai 1999 mobile robot competitions and exhibitions. AI Magazine 21(3), 69–78 (2000)Google Scholar
  8. 8.
    Kitano, H., Tadokoro, S.: RoboCup Rescue: A Grand Challenge for Multiagent and Intelligent Systems. AI Magazine 22(1), 39–52 (2001)Google Scholar
  9. 9.
    Steinfeld, A., Fong, T., Kaber, D.B., Lewis, M., Scholtz, J., Schultz, A.C., Goodrich, M.A.: Common metrics for human-robot interaction. In: Proc. of HRI, pp. 33–40 (2006)Google Scholar
  10. 10.
    Yanco, H.A., Drury, J.L., Scholtz, J.: Beyond Usability Evaluation: Analysis of Human-Robot Interaction at a Major Robotics Competition. Human-Computer Interaction 19, 117–149 (2004)CrossRefGoogle Scholar
  11. 11.
    Drury, J.L., Yanco, H.A., Scholtz, J.: Using competitions to study human-robot interaction in urban search and rescue. Interactions 12(2), 39–41 (2005)CrossRefGoogle Scholar
  12. 12.
    Prassler, E., Hägele, M., Siegwart, R.: International Contest for Cleaning Robots: Fun Event or a First Step towards Benchmarking Service Robots (2006)Google Scholar
  13. 13.
    Savage, J., Ayala, F., Cuellar, S., Weitzenfeld, A.: The use of scripts based on conceptual dependency primitives for the operation of service mobile robots. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)Google Scholar
  14. 14.
    Ruiz-del-Solar, J.: Personal robots as ubiquitous-multimedial-mobile web interfaces. In: Proc. of 5th Latin American Web Congress (LA-WEB), pp. 120–127 (2007)Google Scholar
  15. 15.
    Schiffer, S., Ferrein, A., Lakemeyer, G.: Football is coming Home. In: Proc. of International Symposium on Practical Cognitive Agents and Robots (PCAR 2006). University of Western Australia Press (2006)Google Scholar
  16. 16.
    Doostdar, M., Schiffer, S., Lakemeyer, G.: A Robust Speech Recognition System for Service-Robotics Applications. In: Iocchi, L., Matsubara, H., Weitzenfeld, A., Zhou, C. (eds.) RoboCup 2008. LNCS (LNAI), vol. 5399, pp. 1–12. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Correa, M., Ruiz-del-Solar, J., Bernuy, F.: Face recognition for human-robot interaction applications: A comparative study. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)Google Scholar
  18. 18.
    Knox, W.B., Lee, J., Stone, P.: Domestic interaction on a segway base. In: Proceedings of the International RoboCup Symposium 2008 (CD-ROM Proceedings) (2008)Google Scholar
  19. 19.
    Belle, V., Deselaers, T., Schiffer, S.: Randomized trees for real-time one-step face detection and recognition. In: Proceedings of the 19th International Conference on Pattern Recognition (ICPR 2008), December 8-11. IEEE Computer Society, Los Alamitos (2008)Google Scholar
  20. 20.
    Loncomilla, P., Ruiz-del-Solar, J.: Robust object recognition using wide baseline matching for robocup applications. In: Visser, U., Ribeiro, F., Ohashi, T., Dellaert, F. (eds.) RoboCup 2007. LNCS (LNAI), vol. 5001, pp. 441–448. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  21. 21.
    Francke, H., Ruiz-del-Solar, J., Verschae, R.: Real-time hand gesture detection and recognition using boosted classifiers and active learning. In: Mery, D., Rueda, L. (eds.) PSIVT 2007. LNCS, vol. 4872, pp. 533–547. Springer, Heidelberg (2007)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Thomas Wisspeintner
    • 1
  • Tijn van der Zan
    • 2
  • Luca Iocchi
    • 3
  • Stefan Schiffer
    • 4
  1. 1.Department of Mathematics and Computer ScienceFreie Universität BerlinBerlinGermany
  2. 2.Department of Artificial IntelligenceUniversity of GroningenGroningenThe Netherlands
  3. 3.Dipartimento di Informatica e SistemisticaUniversity ”La Sapienza”RomeItaly
  4. 4.Knowledge-Based Systems GroupRWTH Aachen UniversityAachenGermany

Personalised recommendations