Evaluation of the RoboCup Logistics League and Derived Criteria for Future Competitions

  • Tim NiemuellerEmail author
  • Sebastian Reuter
  • Alexander Ferrein
  • Sabina Jeschke
  • Gerhard Lakemeyer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9513)


In the RoboCup Logistics League (RCLL), games are governed by a semi-autonomous referee box. It also records tremendous amounts of data about state changes of the game or communication with the robots. In this paper, we analyze the data of the 2014 competition by means of Key Performance Indicators (KPI). KPIs are used in industrial environments to evaluate the performance of production systems. Applying adapted KPIs to the RCLL provides interesting insights about the strategies of the robot teams. When aiming for more realistic industrial properties with a 24/7 production, where teams perform shifts (without intermediate environment reset), KPIs could be a means to score the game. This could be tried first in a simulation sub-league.


  1. 1.
    Zwilling, F., Lakemeyer, G., Niemueller, T.: Simulation for the RoboCup logistics league with real-world environment agency and multi-level abstraction. In: Bianchi, R.A.C., Akin, H.L., Ramamoorthy, S., Sugiura, K. (eds.) RoboCup 2014. LNCS, vol. 8992, pp. 220–232. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  2. 2.
    Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E.: Robocup: the robot world cup initiative. In: 1st International Conference on Autonomous Agents (1997)Google Scholar
  3. 3.
    Niemueller, T., Lakemeyer, G., Ferrein, A., Reuter, S., Ewert, D., Jeschke, S., Pensky, D., Karras, U.: Proposal for advancements to the LLSF in 2014 and beyond. In: ICAR - 1st Workshop on Developments in RoboCup Leagues (2013)Google Scholar
  4. 4.
    Niemueller, T., Lakemeyer, G., Ferrein, A.: The RoboCup logistics league as a benchmark for planning in robotics. In: WS on Planning and Robotics (PlanRob) at International Conference on Automated Planning and Scheduling (ICAPS) (2015, to appear)Google Scholar
  5. 5.
    Wygant, R.M.: CLIPS: a powerful development and delivery expert system tool. Comput. Ind. Eng. 17, 1–4 (1989)CrossRefGoogle Scholar
  6. 6.
    Long, D., Kautz, H., Selman, B., Bonet, B., Geffner, H., Koehler, J., Brenner, M., Hoffmann, J., Rittinger, F., Anderson, C.R., et al.: The AIPS-98 planning competition. AI Mag. 21(2), 13–33 (2000)Google Scholar
  7. 7.
    Howe, A.E., Dahlman, E.: A critical assessment of benchmark comparison in planning. J. Artif. Intell. Res. 17(1), 1–33 (2002)CrossRefGoogle Scholar
  8. 8.
    Balint, A., Belov, A., Järvisalo, M., Sinz, C.: Sat challenge 2012 random sat track: description of benchmark generation. In: Proceedings of SAT CHALLENGE 2012, p. 72 (2012)Google Scholar
  9. 9.
    Karakovskiy, S., Togelius, J.: The mario AI benchmark and competitions. IEEE Trans. Comput. Intell. AI Games 4(1), 55–67 (2012)CrossRefGoogle Scholar
  10. 10.
    Balch, T.R., Yanco, H.A.: Ten years of the AAAI mobile robot competition and exhibition. AI Mag. 23(1), 13–22 (2002)Google Scholar
  11. 11.
    Anderson, M., Jenkins, O., Osentoski, S.: Recasting robotics challenges as experiments [competitions]. IEEE Robot Autom. Mag. 18(2), 10–11 (2011)CrossRefGoogle Scholar
  12. 12.
    Anderson, J., Baltes, J., Cheng, C.T.: Robotics competitions as benchmarks for AI research. Knowl. Eng. Rev. 26(01), 11–17 (2011)CrossRefGoogle Scholar
  13. 13.
    Kitano, H., Tadokoro, S.: Robocup rescue: a grand challenge for multiagent and intelligent systems. AI Mag. 22(1), 11–17 (2001)Google Scholar
  14. 14.
    Wisspeintner, T., Van Der Zant, T., Iocchi, L., Schiffer, S.: Robocup@home: scientific competition and benchmarking for domestic service robots. Interact. Stud. 10(3), 392–426 (2009)CrossRefGoogle Scholar
  15. 15.
    Madhavan, R., Lakaemper, R., Kalmár-Nagy, T.: Benchmarking and standardization of intelligent robotic systems. In: International Conference on Advanced Robotics (ICAR) (2009)Google Scholar
  16. 16.
    Calisi, D., Iocchi, L., Nardi, D.: A unified benchmark framework for autonomous mobile robots and vehicles motion algorithms (movema benchmarks). In: WS on Experimental Methodology and Benchmarking in Robotics Research at RSS (2008)Google Scholar
  17. 17.
    Ahmad, A., Awaad, I., Amigoni, F., Berghofer, J., Bischoff, R., Bonarini, A., Dwiputra, R., Fontana, G., Hegger, F., Hochgeschwender, N., et al.: Specification of general features of scenarios and robots for benchmarking through competitions. RoCKIn Deliverable D 1 (2013)Google Scholar
  18. 18.
    Amigoni, F., Bonarini, A., Fontana, G., Matteucci, M., Schiaffonati, V.: Benchmarking through competitions. In: European Robotics Forum-Workshop on Robot Competitions: Benchmarking, Technology Transfer, and Education (2013)Google Scholar
  19. 19.
    Nyhuis, P., Wiendahl, H.P.: Fundamentals of Production Logistics: Theory Tools and Applications. Springer Science & Business Media, Heidelberg (2008)Google Scholar
  20. 20.
    Wriggers, F., Busse, T., Nyhuis, P.: Modeling and deriving strategic logistic measures. In: IEEE International Conference on Industrial Engineering and Engineering Management (2007)Google Scholar
  21. 21.
    Lödding, H.: Handbook of Manufacturing Control: Fundamentals, Description, Configuration. Springer Science & Business Media, Heidelberg (2012)Google Scholar
  22. 22.
    Nyhuis, P., Wiendahl, H.P.: Logistic production operating curves-basic model of the theory of logistic operating curves. CIRP Annals-Manufact. Tech. 55(1), 441–444 (2006)CrossRefGoogle Scholar
  23. 23.
    Niemueller, T., Lakemeyer, G., Srinivasa, S.S.: A generic robot database and its application in fault analysis and performance evaluation. In: International Conference on Intelligent Robots and Systems (IROS) (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (, which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Authors and Affiliations

  • Tim Niemueller
    • 1
    Email author
  • Sebastian Reuter
    • 2
  • Alexander Ferrein
    • 3
  • Sabina Jeschke
    • 2
  • Gerhard Lakemeyer
    • 1
  1. 1.Knowledge-Based Systems GroupRWTH Aachen UniversityAachenGermany
  2. 2.Institute Cluster IMA/ZLW & IfU, RWTH Aachen UniversityAachenGermany
  3. 3.MASCOR Institute, Aachen University of Applied SciencesAachenGermany

Personalised recommendations