Synthetical Benchmarking of Service Robots: A First Effort on Domestic Mobile Platforms

  • Min ChengEmail author
  • Xiaoping Chen
  • Keke Tang
  • Feng Wu
  • Andras Kupcsik
  • Luca Iocchi
  • Yingfeng Chen
  • David Hsu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9513)


Most of existing benchmarking tools for service robots are basically qualitative, in which a robot’s performance on a task is evaluated based on completion/incompletion of actions contained in the task. In the effort reported in this paper, we tried to implement a synthetical benchmarking system on domestic mobile platforms. Synthetical benchmarking consists of both qualitative and quantitative aspects, such as task completion, accuracy of task completions and efficiency of task completions, about performance of a robot. The system includes a set of algorithms for collecting, recording and analyzing measurement data from a MoCap system. It was used as the evaluator in a competition called the BSR challenge, in which 10 teams participated, at RoboCup 2015. The paper presents our motivations behind synthetical benchmarking, the design considerations on the synthetical benchmarking system, the realization of the competition as a comparative study on performance evaluation of domestic mobile platforms, and an analysis of the teams’ performance.



A special thank is given to Intel China for its sponsorship of the BSR challenge and workshop. The authors from USTC are supported by the Natural Science Foundation of China under grants 60745002 and 61175057, as well as the USTC Key Direction Project. All authors are thankful for contributions from all collaborators who are not an author of the paper and all participants in the event.


  1. 1.
    del Pobil, A.P.: Why do we need benchmarks in robotics research? In: Proceedings of the Workshop on Benchmarks in Robotics Research, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006)Google Scholar
  2. 2.
    Nardi, L., Bodin, B., Zia, M.Z., Mawer, J., Nisbet, A., Kelly, P.H., Davison, A.J., Luján, M., O’Boyle, M.F., Riley, G., et al.: Introducing slambench, a performance and accuracy benchmarking methodology for slam. arXiv preprint arXiv:1410.2167 (2014)
  3. 3.
    Fontana, G., Matteucci, M., Sorrenti, D.G.: Rawseeds: building a benchmarking toolkit for autonomous robotics. In: Amigoni, F., Schiaffonati, V. (eds.) Methods and Experimental Techniques in Computer Engineering, pp. 55–68. Springer, New York (2014)CrossRefGoogle Scholar
  4. 4.
    Behnke, S.: Robot competitions-ideal benchmarks for robotics research. In: Proceedings of IROS-2006 Workshop on Benchmarks in Robotics Research (2006)Google Scholar
  5. 5.
    Schultz, A.C.: The 2000 AAAI mobile robot competition and exhibition. AI Mag. 22(1), 67 (2001)Google Scholar
  6. 6.
    Maxwell, B.A., Smart, W., Jacoff, A., Casper, J., Weiss, B., Scholtz, J., Yanco, H., Micire, M., Stroupe, A., Stormont, D., et al.: 2003 AAAI robot competition and exhibition. AI Mag. 25(2), 68 (2004)Google Scholar
  7. 7.
    Balch, T., Yanco, H.: Ten years of the AAAI mobile robot competition and exhibition. AI Mag. 23(1), 13 (2002)Google Scholar
  8. 8.
    Firby, R.J., Prokopowicz, P.N., Swain, M.J., Kahn, R.E., Franklin, D.: Programming CHIP for the IJCAI-95 robot competition. AI Mag. 17(1), 71 (1996)Google Scholar
  9. 9.
    Pratt, G., Manzo, J.: The DARPA robotics challenge [competitions]. IEEE Rob. Autom. Mag. 20(2), 10–12 (2013)CrossRefGoogle Scholar
  10. 10.
    Wisspeintner, T., Van Der Zant, T., Iocchi, L., Schiffer, S.: Robocup@ home: scientific competition and benchmarking for domestic service robots. Interact. Stud. 10(3), 392–426 (2009)CrossRefGoogle Scholar
  11. 11.
    Wisspeintner, T., van der Zan, T., Iocchi, L., Schiffer, S.: RoboCup@Home: results in benchmarking domestic service robots. In: Baltes, J., Lagoudakis, M.G., Naruse, T., Ghidary, S.S. (eds.) RoboCup 2009. LNCS, vol. 5949, pp. 390–401. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Holz, D., Iocchi, L., van der Zant, T.: Benchmarking intelligent service robots through scientific competitions: the Robocup@ home approach. In: AAAI Spring Symposium: Designing Intelligent Robots (2013)Google Scholar
  13. 13.
    Stuckler, J., Holz, D., Behnke, S.: Demonstrating everyday manipulation skills in Robocup@ home. IEEE Rob. Autom. Mag. 19(2), 34–42 (2012)CrossRefGoogle Scholar
  14. 14.
    Amigoni, F., Bonarini, A., Fontana, G., Matteucci, M., Schiaffonati, V.: Benchmarking through competitions. In: European Robotics Forum-Workshop on Robot Competitions: Benchmarking, Technology Transfer, and Education (2013)Google Scholar
  15. 15.
    Ahmad, A., Awaad, I., Amigoni, F., Berghofer, J., Bischoff, R., Bonarini, A., Dwiputra, R., Fontana, G., Hegger, F., Hochgeschwender, N., et al.: Specification of general features of scenarios and robots for benchmarking through competitions. RoCKIn Deliverable D 1 (2013)Google Scholar
  16. 16.
    Corazza, S., Muendermann, L., Chaudhari, A., Demattio, T., Cobelli, C., Andriacchi, T.P.: A markerless motion capture system to study musculoskeletal biomechanics: visual hull and simulated annealing approach. Ann. Biomed. Eng. 34(6), 1019–1029 (2006)CrossRefGoogle Scholar
  17. 17.
    Kurihara, K., Hoshino, S., Yamane, K., Nakamura, Y.: Optical motion capture system with pan-tilt camera tracking and realtime data processing. In: ICRA, pp. 1241–1248 (2002)Google Scholar
  18. 18.
    Field, M., Stirling, D., Naghdy, F., Pan, Z.: Motion capture in robotics review. In: 2009 IEEE International Conference on Control and Automation. ICCA 2009, pp. 1697–1702, December 2009Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Min Cheng
    • 1
    Email author
  • Xiaoping Chen
    • 1
  • Keke Tang
    • 1
  • Feng Wu
    • 1
  • Andras Kupcsik
    • 3
  • Luca Iocchi
    • 2
  • Yingfeng Chen
    • 1
  • David Hsu
    • 3
  1. 1.University of Science and Technology of ChinaHefeiChina
  2. 2.Sapienza University of RomeRomeItaly
  3. 3.National University of SingaporeSingaporeSingapore

Personalised recommendations