A Layered Metric Definition and Evaluation Framework for Multirobot Systems

  • Çetin Meriçli
  • H. Levent Akın
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5399)


In order to accomplish it successfully, the top-level goal of a multi-robot team should be decomposed into a sequence of sub-goals and proper sequences of actions for achieving these subgoals should be selected and refined through execution. Selecting the proper actions at any given time requires the ability to evaluate the current state of the environment, which can be achieved by using metrics that give quantitative information about the environment. Defining appropriate metrics is already a challenging problem; however, it is even harder to assess the performance of individual metrics. This work proposes a layered evaluation scheme for robot soccer where the environment is represented in different time resolutions at each layer. A set of metrics defined on these layers together with a novel metric validation method for assessing the performance of the defined metrics are proposed.


Convex Hull Evaluation Framework Markov Chain Monte Carlo Method Soccer Game Robot Soccer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    VerDuin, W.H., Kothamasu, R., Huang, S.H.: Analysis of performance evaluation metrics to combat the model selection problem. In: PERMISA Workshop (2003)Google Scholar
  2. 2.
    Yavnai, A.: An information-based approach for system autonomy metrics part i: Metrics definition. In: PERMISA Workshop (2003)Google Scholar
  3. 3.
    Horst, J.A.: Exercising a native intelligence metric on an autonomous on-road driving system. In: PERMISA Workshop (2003)Google Scholar
  4. 4.
    Olsen, D.R., Goodrich, M.A.: Metrics for evaluating human-robot interactions. In: PERMISA Workshop (2003)Google Scholar
  5. 5.
    Balch, T.: Social entropy: a new metric for learning multi-robot teams. In: 10th International FLAIRS Conference (FLAIRS 1997) (1997)Google Scholar
  6. 6.
    Rue, H., Salvesen, Ø.: Predicting and retrospective analysis of soccer matches in a leagueGoogle Scholar
  7. 7.
    Yanco, H.A.: Designing metrics for comparing the performance of robotic systems in robot competitions. In: Workshop on Measuring Performance and Intelligence of Intelligent Systems (PERMIS) (2001)Google Scholar
  8. 8.
    Kok, J., de Boer, R., Vlassis, N.: Towards an optimal scoring policy for simulated soccer agents. Technical report (2001)Google Scholar
  9. 9.
    Dylla, F., Ferrein, A., Lakemeyer, G., Murray, J., Obst, O., Röfer, T., Stolzenburg, F., Visser, U., Wagner, T.: Towards a league-independent qualitative soccer theory for robocup. In: Nardi, D., Riedmiller, M., Sammut, C., Santos-Victor, J. (eds.) RoboCup 2004. LNCS, vol. 3276, pp. 611–618. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Veloso, M., Lenser, S., Vail, D., Roth, M., Stroupe, A., Chernova, S.: Cmpack 2002: Cmu’s legged robot soccer team. Technical report (2002)Google Scholar
  11. 11.
    Röfer, T., et al.: Germanteam 2006. Technical report, The GermanTeam (2007)Google Scholar
  12. 12.
    Quinlan, M.J., Henderson, N., Middleton, R.H.: The 2006 nubots team report. Technical report, Newcastle Robotics Laboratory (2007)Google Scholar
  13. 13.
    Balch, T.: Teambots (2000),
  14. 14.
    Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Massachusetts (1995)zbMATHGoogle Scholar
  15. 15.
    Hodrick, R.J., Prescott, E.C.: Postwar u.s. business cycles: An empirical investigation. Journal of Money, Credit and Banking 29 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Çetin Meriçli
    • 1
  • H. Levent Akın
    • 1
  1. 1.Department of Computer EngineeringBoğaziçi UniversityÍstanbulTurkey

Personalised recommendations