Synthetical Benchmarking of Service Robots: A First Effort on Domestic Mobile Platforms
Most of existing benchmarking tools for service robots are basically qualitative, in which a robot’s performance on a task is evaluated based on completion/incompletion of actions contained in the task. In the effort reported in this paper, we tried to implement a synthetical benchmarking system on domestic mobile platforms. Synthetical benchmarking consists of both qualitative and quantitative aspects, such as task completion, accuracy of task completions and efficiency of task completions, about performance of a robot. The system includes a set of algorithms for collecting, recording and analyzing measurement data from a MoCap system. It was used as the evaluator in a competition called the BSR challenge, in which 10 teams participated, at RoboCup 2015. The paper presents our motivations behind synthetical benchmarking, the design considerations on the synthetical benchmarking system, the realization of the competition as a comparative study on performance evaluation of domestic mobile platforms, and an analysis of the teams’ performance.
A special thank is given to Intel China for its sponsorship of the BSR challenge and workshop. The authors from USTC are supported by the Natural Science Foundation of China under grants 60745002 and 61175057, as well as the USTC Key Direction Project. All authors are thankful for contributions from all collaborators who are not an author of the paper and all participants in the event.
- 1.del Pobil, A.P.: Why do we need benchmarks in robotics research? In: Proceedings of the Workshop on Benchmarks in Robotics Research, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006)Google Scholar
- 2.Nardi, L., Bodin, B., Zia, M.Z., Mawer, J., Nisbet, A., Kelly, P.H., Davison, A.J., Luján, M., O’Boyle, M.F., Riley, G., et al.: Introducing slambench, a performance and accuracy benchmarking methodology for slam. arXiv preprint arXiv:1410.2167 (2014)
- 4.Behnke, S.: Robot competitions-ideal benchmarks for robotics research. In: Proceedings of IROS-2006 Workshop on Benchmarks in Robotics Research (2006)Google Scholar
- 5.Schultz, A.C.: The 2000 AAAI mobile robot competition and exhibition. AI Mag. 22(1), 67 (2001)Google Scholar
- 6.Maxwell, B.A., Smart, W., Jacoff, A., Casper, J., Weiss, B., Scholtz, J., Yanco, H., Micire, M., Stroupe, A., Stormont, D., et al.: 2003 AAAI robot competition and exhibition. AI Mag. 25(2), 68 (2004)Google Scholar
- 7.Balch, T., Yanco, H.: Ten years of the AAAI mobile robot competition and exhibition. AI Mag. 23(1), 13 (2002)Google Scholar
- 8.Firby, R.J., Prokopowicz, P.N., Swain, M.J., Kahn, R.E., Franklin, D.: Programming CHIP for the IJCAI-95 robot competition. AI Mag. 17(1), 71 (1996)Google Scholar
- 12.Holz, D., Iocchi, L., van der Zant, T.: Benchmarking intelligent service robots through scientific competitions: the Robocup@ home approach. In: AAAI Spring Symposium: Designing Intelligent Robots (2013)Google Scholar
- 14.Amigoni, F., Bonarini, A., Fontana, G., Matteucci, M., Schiaffonati, V.: Benchmarking through competitions. In: European Robotics Forum-Workshop on Robot Competitions: Benchmarking, Technology Transfer, and Education (2013)Google Scholar
- 15.Ahmad, A., Awaad, I., Amigoni, F., Berghofer, J., Bischoff, R., Bonarini, A., Dwiputra, R., Fontana, G., Hegger, F., Hochgeschwender, N., et al.: Specification of general features of scenarios and robots for benchmarking through competitions. RoCKIn Deliverable D 1 (2013)Google Scholar
- 17.Kurihara, K., Hoshino, S., Yamane, K., Nakamura, Y.: Optical motion capture system with pan-tilt camera tracking and realtime data processing. In: ICRA, pp. 1241–1248 (2002)Google Scholar
- 18.Field, M., Stirling, D., Naghdy, F., Pan, Z.: Motion capture in robotics review. In: 2009 IEEE International Conference on Control and Automation. ICCA 2009, pp. 1697–1702, December 2009Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.