RoboCup 2015: RoboCup 2015: Robot World Cup XIX pp 377-388 | Cite as
Synthetical Benchmarking of Service Robots: A First Effort on Domestic Mobile Platforms
Abstract
Most of existing benchmarking tools for service robots are basically qualitative, in which a robot’s performance on a task is evaluated based on completion/incompletion of actions contained in the task. In the effort reported in this paper, we tried to implement a synthetical benchmarking system on domestic mobile platforms. Synthetical benchmarking consists of both qualitative and quantitative aspects, such as task completion, accuracy of task completions and efficiency of task completions, about performance of a robot. The system includes a set of algorithms for collecting, recording and analyzing measurement data from a MoCap system. It was used as the evaluator in a competition called the BSR challenge, in which 10 teams participated, at RoboCup 2015. The paper presents our motivations behind synthetical benchmarking, the design considerations on the synthetical benchmarking system, the realization of the competition as a comparative study on performance evaluation of domestic mobile platforms, and an analysis of the teams’ performance.
Keywords
Task Completion Robot Performance Service Robot Laser Range Finder Synthetical BenchmarkingNotes
Acknowledgments
A special thank is given to Intel China for its sponsorship of the BSR challenge and workshop. The authors from USTC are supported by the Natural Science Foundation of China under grants 60745002 and 61175057, as well as the USTC Key Direction Project. All authors are thankful for contributions from all collaborators who are not an author of the paper and all participants in the event.
References
- 1.del Pobil, A.P.: Why do we need benchmarks in robotics research? In: Proceedings of the Workshop on Benchmarks in Robotics Research, IEEE/RSJ International Conference on Intelligent Robots and Systems (2006)Google Scholar
- 2.Nardi, L., Bodin, B., Zia, M.Z., Mawer, J., Nisbet, A., Kelly, P.H., Davison, A.J., Luján, M., O’Boyle, M.F., Riley, G., et al.: Introducing slambench, a performance and accuracy benchmarking methodology for slam. arXiv preprint arXiv:1410.2167 (2014)
- 3.Fontana, G., Matteucci, M., Sorrenti, D.G.: Rawseeds: building a benchmarking toolkit for autonomous robotics. In: Amigoni, F., Schiaffonati, V. (eds.) Methods and Experimental Techniques in Computer Engineering, pp. 55–68. Springer, New York (2014)CrossRefGoogle Scholar
- 4.Behnke, S.: Robot competitions-ideal benchmarks for robotics research. In: Proceedings of IROS-2006 Workshop on Benchmarks in Robotics Research (2006)Google Scholar
- 5.Schultz, A.C.: The 2000 AAAI mobile robot competition and exhibition. AI Mag. 22(1), 67 (2001)Google Scholar
- 6.Maxwell, B.A., Smart, W., Jacoff, A., Casper, J., Weiss, B., Scholtz, J., Yanco, H., Micire, M., Stroupe, A., Stormont, D., et al.: 2003 AAAI robot competition and exhibition. AI Mag. 25(2), 68 (2004)Google Scholar
- 7.Balch, T., Yanco, H.: Ten years of the AAAI mobile robot competition and exhibition. AI Mag. 23(1), 13 (2002)Google Scholar
- 8.Firby, R.J., Prokopowicz, P.N., Swain, M.J., Kahn, R.E., Franklin, D.: Programming CHIP for the IJCAI-95 robot competition. AI Mag. 17(1), 71 (1996)Google Scholar
- 9.Pratt, G., Manzo, J.: The DARPA robotics challenge [competitions]. IEEE Rob. Autom. Mag. 20(2), 10–12 (2013)CrossRefGoogle Scholar
- 10.Wisspeintner, T., Van Der Zant, T., Iocchi, L., Schiffer, S.: Robocup@ home: scientific competition and benchmarking for domestic service robots. Interact. Stud. 10(3), 392–426 (2009)CrossRefGoogle Scholar
- 11.Wisspeintner, T., van der Zan, T., Iocchi, L., Schiffer, S.: RoboCup@Home: results in benchmarking domestic service robots. In: Baltes, J., Lagoudakis, M.G., Naruse, T., Ghidary, S.S. (eds.) RoboCup 2009. LNCS, vol. 5949, pp. 390–401. Springer, Heidelberg (2010)CrossRefGoogle Scholar
- 12.Holz, D., Iocchi, L., van der Zant, T.: Benchmarking intelligent service robots through scientific competitions: the Robocup@ home approach. In: AAAI Spring Symposium: Designing Intelligent Robots (2013)Google Scholar
- 13.Stuckler, J., Holz, D., Behnke, S.: Demonstrating everyday manipulation skills in Robocup@ home. IEEE Rob. Autom. Mag. 19(2), 34–42 (2012)CrossRefGoogle Scholar
- 14.Amigoni, F., Bonarini, A., Fontana, G., Matteucci, M., Schiaffonati, V.: Benchmarking through competitions. In: European Robotics Forum-Workshop on Robot Competitions: Benchmarking, Technology Transfer, and Education (2013)Google Scholar
- 15.Ahmad, A., Awaad, I., Amigoni, F., Berghofer, J., Bischoff, R., Bonarini, A., Dwiputra, R., Fontana, G., Hegger, F., Hochgeschwender, N., et al.: Specification of general features of scenarios and robots for benchmarking through competitions. RoCKIn Deliverable D 1 (2013)Google Scholar
- 16.Corazza, S., Muendermann, L., Chaudhari, A., Demattio, T., Cobelli, C., Andriacchi, T.P.: A markerless motion capture system to study musculoskeletal biomechanics: visual hull and simulated annealing approach. Ann. Biomed. Eng. 34(6), 1019–1029 (2006)CrossRefGoogle Scholar
- 17.Kurihara, K., Hoshino, S., Yamane, K., Nakamura, Y.: Optical motion capture system with pan-tilt camera tracking and realtime data processing. In: ICRA, pp. 1241–1248 (2002)Google Scholar
- 18.Field, M., Stirling, D., Naghdy, F., Pan, Z.: Motion capture in robotics review. In: 2009 IEEE International Conference on Control and Automation. ICCA 2009, pp. 1697–1702, December 2009Google Scholar