Advertisement

Metrics for Multiagent Systems

  • Robert N. LassEmail author
  • Evan A. Sultanik
  • William C. Regli
Chapter

Abstract

A Multiagent System (MAS) is a software paradigm for building large scale intelligent distributed systems. Increasingly these systems are being deployed on handheld computing devices that rely on non-traditional communications mediums such as mobile ad hoc networks and satellite links. These systems present new challenges for computer scientists in describing system performance and analyzing competing systems. This chapter surveys existing metrics that can be used to describe MASs and related components. A framework for analyzing MASs is provided and an example of how this framework might be employed is given for the domain of distributed constraint reasoning.

Keywords

Transmission Control Protocol Multiagent System Session Initiation Protocol Graph Coloring Constraint Graph 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Mayk, I., Regli, W.C., eds.: Agent Systems Reference Model. Intelligent Agents Integrated Product Sub-Team, Networking Integrated Product Team, Command and Control Directorate, Headquarters, US Army Research, Development, and Engineering Command, Communications-Electronics Research, Development, and Engineering Center, Department of the Army (November 2006) http://gicl.cs.drexel.edu/people/regli/reference_model-v1a.pdf.
  2. 2.
    Sirer, E.G.: Sextant Deployment (accessed 10/31/2007) http://www.cs.cornell.edu/People/egs/sextant/deployment.php.
  3. 3.
    Haeberlen, A., Mislove, A., Post, A., Druschel, P.: Fallacies in Evaluating Decentralized Systems. In: Proceedings of the 5th International Workshop on Peer-to-Peer Systems (February 2006)Google Scholar
  4. 4.
    Sultanik, E.A., Lass, R.N., Regli, W.C.: eDCOPolis: A Framework for Simulating and Deploying Distributed Constraint Optimization Algorithms. In: Proceedings of the Ninth Annual Workshop on Distributed Constraint Reasoning (September 2007)Google Scholar
  5. 5.
    Wooldridge, M.: Introduction to Multiagent Systems. John Wiley & Sons, Inc. New York, NY, USA (2001)Google Scholar
  6. 6.
    Italia, T.: Java Agent DEvelopment Framework (JADE) http://jade.tilab.com/.
  7. 7.
    Corporation, B.: Cognitive Agent Architecture (Cougaar) http://www.cougaar.org/.
  8. 8.
  9. 9.
    Zimmerman, H.: OSI Reference Model–-the ISO Model of Architecture for Open System Interconnection. IEEE Transactions on Communications 28(4) (April 1980) 425–432CrossRefGoogle Scholar
  10. 10.
    Stevens, S.S.: On the Theory of Scales of Measurement. Science (1946)Google Scholar
  11. 11.
    Hollingsworth, J.K., Lumpp, J., Miller, B.P.: Techniques for Performance Measurement of Parellel Programs. In: Parallel Computers: Theory and Practice. IEEE Press, Hoboken, NJ, USA (1994)Google Scholar
  12. 12.
    Hollingsworth, J.K., Miller, B.P.: Parallel Program Performance Metrics: A Comparison and Validation. In: Proceedings of the 1992 ACM/IEEE Conference on Supercomputing. IEEE Computer Society Press, Los Alamitos, CA, USA (1992) 4–13Google Scholar
  13. 13.
    Huang, H.M.: The Autonomy Levels for Unmanned Systems ALFUS Framework. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006) 47–51Google Scholar
  14. 14.
    Barber, K.S., Martin, C.E.: Agent Autonomy: Specification, Measurement, and Dynamic Adjustment. In: Proceedings of the Autonomy Control Software Workshop. (1999) 8–15Google Scholar
  15. 15.
    Davin, J., Modi, P.J.: Impact of problem centralization in distributed constraint optimization algorithms. In: Proceedings of the Fourth International Joint Conference on Autonomous Agents and Multiagent Systems. ACM Press, New York, NY, USA (2005) 1057–1063Google Scholar
  16. 16.
    Meisels, A., Kaplansky, E., Razgon, I., Zivan, R.: Comparing Performance of Distributed Constraints Processing Algorithms. In: Proceedings of the Third International Workshop on Distributed Constraint Reasoning, Bologna, Italy (July 2002)Google Scholar
  17. 17.
    Silaghi, M., Lass, R.N., Sultanik, E.A., Regli, W.C., Matsui, T., Yokoo, M.: Constant Cost of the Computation-Unit in Efficiency Graphs for DCOPs. In: Proceedings of the International Conference on Intelligent Agent Technology (December 2008)Google Scholar
  18. 18.
    Anderson, M.L.: A Flexible Approach to Quantifying Various Dimensions of Environmental Complexity. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop (2004)Google Scholar
  19. 19.
    Molino, V., Madhavan, R., Messina, E., Downs, T., Jacoff, A., Balakirsky, S.: Treversability Metrics for Urban Search and Rescue Robots on Rough Terrain. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  20. 20.
    Freed, M., Harris, R., Shafto, M.: Measuring Autonomous UAV Surveillence. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop. (2004)Google Scholar
  21. 21.
    Commuri, S., Li, Y., Hougen, D., Fierro, R.: Evaluating Intelligence in Unmanned Ground Vehicle Teams. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop (2004)Google Scholar
  22. 22.
    Brendle, B.: 3D Data Registrnation Based on Human Perception. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  23. 23.
    Burke, J.L., Murphy, R.R., Riddle, D.R., Fincannon, T.: Task Performance Metrics in Human-Robot Interaction: Taking a Systems Approach. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop (2004)Google Scholar
  24. 24.
    Freedy, A., McDonough, J., Jacobs, R., Freedy, E., Thayer, S., Weltman, G.: A Mixed Initiative Human-Robots Team Performance Assessment System for Use in Operational and Training Environments. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop (2004)Google Scholar
  25. 25.
    Kramer, J.A., Murphy, R.R.: Endurance Testing for Safety, Security and Rescue Robots. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006) 247–254Google Scholar
  26. 26.
    Foedisch, M., Schlenoff, C., Madhavan, R.: Performance Analysis of Symbolic Road Recognition for On-road Driving. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  27. 27.
    Stanton, B., Antonishek, B., Scholtz, J.: Development of an Evaluation Method for Acceptable Usability. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006) 263–267Google Scholar
  28. 28.
    Tunstel, E.: Performance Metrics for Operational Mars Rovers. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006) 69–76Google Scholar
  29. 29.
    Lee, L., Nwana, H., Ndumu, D., Wilde, P.D.: The Stability, Scalability and Performance of Multi-agent Systems. BT Technology Journal 16(3) (1998) 94–103CrossRefGoogle Scholar
  30. 30.
    Rodriguez, G., Weisbin, C.R.: A New Method to Evaluate Human-Robot System Performance. Autonomous Robots 14(2) (2003) 165–178zbMATHCrossRefGoogle Scholar
  31. 31.
    Balakirsky, S., Scrapper, C., Carpin, S., Lewis, M.: USARSim: Providing a Framework for Multi-Robot Performance Evaluation. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  32. 32.
    Lundberg, C., Christensen, H., Reinhold, R.: Long Term Study of a Portable Field Robot in Urban Terrain. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  33. 33.
    Kannan, B., Parker, L.E.: Fault Tolerance Based Metrics for Evaluating System Performance in Multi-Robot Teams. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006)Google Scholar
  34. 34.
    Ference, J.J., Szabo, S., Najm, W.G.: Performance Evaluation of Integrated Vehicle-Based Safety Systems. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop, NIST Special Publication Gaithersburg, MD, USA (2006) 85–89Google Scholar
  35. 35.
    Chellappa, R., Roy-Chowdhury, A.K.: An Information Theoretic Evaluation Criterion for 3D Reconstruction Algorithms. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop. (2004)Google Scholar
  36. 36.
    Corson, S., Macker, J.: Mobile Ad hoc Networking (MANET): Routing Protocol Performance Issues and Evaluation Considerations. RFC 2501 (January 1999)Google Scholar
  37. 37.
    Johansson, P., Larsson, T., Hedman, N., Mielczarek, B., Degermark, M.: Scenario-Based Performance Analysis of Routing Protocols for Mobile Ad-Hoc Networks. In: Proceedings of the 5th Annual ACM/IEEE International Conference on Mobile Computing and Networking. ACM Press, New York, NY, USA (1999) 195–206Google Scholar
  38. 38.
    Lamport, L.: Time, Clocks and the Ordering of Events in a Distributed System. Communications of the ACM 21(7) (July 1978) 558–565zbMATHCrossRefGoogle Scholar
  39. 39.
    Zhang, Y., Mackworth, A.K.: Parallel and Distributed Algorithms for Finite Constraint Satisfaction Problems. In: Proceedings of the Third IEEE Symposium on Parallel and Distributed Processing (1991) 394–397Google Scholar
  40. 40.
    Hamadi, Y., Bessière, C., Quinqueton, J.: Backtracking in Distributed Constraint Networks. In: Proceedings of the European Conference on Artificial Intelligence (1998) 219–223Google Scholar
  41. 41.
    Chechetka, A., Sycara, K.: No-Commitment Branch and Bound Search for Distributed Constraint Optimization. In: Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems. ACM Press, New York, NY, USA (2006) 1427–1429Google Scholar
  42. 42.
  43. 43.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann San Fransisco, CA, USA (2005)zbMATHGoogle Scholar
  44. 44.
    Smith, R.G.: The Contract Net Protocol: High Level Communication and Control in a Distributed Problem Solver. IEEE Transactions on Computers 12(C-29) (December 1980) 1004–1113Google Scholar
  45. 45.
    Basili, V., Caldiera, G., Rombach, H.: The goal question metric approach. Encyclopedia of Software Engineering 1 (1994) 528–532Google Scholar
  46. 46.
    Yokoo, M., Durfee, E., Ishida, T., Kuwabara, K.: The Distributed Constraint Satisfaction Problem: Formalization and Algorithms. IEEE Transactions on Knowledge and Data Engineering 10(5) (1998) 673–685CrossRefGoogle Scholar
  47. 47.
    Balakirsky, S., Kramer, T.R.: Comparing Algorithms: Rules of Thumb and an Example. In: Proceedings of the Performance Metrics for Intelligent Systems Workshop (2004)Google Scholar
  48. 48.
    Cohen, P.R.: Empirical Methods for Artificial Intelligence. MIT Press, Cambridge, MA, USA (1995)zbMATHGoogle Scholar
  49. 49.
    Modi, P.J., Shen, W.M., Tambe, M., Yokoo, M.: An Asynchronous Complete Method for Distributed Constraint Optimization. In: Proceedings of the Second International Joint Conference on Autonomous Agents and Multiagent Systems. ACM Press, New York, NY, USA (2003) 161–168Google Scholar
  50. 50.
    Mailler, R., Lesser, V.: Solving Distributed Constraint Optimization Problems Using Cooperative Mediation. In: Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems. IEEE Computer Society, Washington, DC, USA (2004) 438–445Google Scholar
  51. 51.
    Petcu, A., Faltings, B.: A Distributed, Complete Method for Multi-Agent Constraint Optimization. In: Proceedings of the Fifth International Workshop on Distributed Constraint Reasoning, Toronto, Canada (September 2004)Google Scholar
  52. 52.
    Anderson, T.E., Lazowska, E.D.: Quartz: a Tool for Tuning Parallel Program Performance. In: Proceedings of the ACM SIGMETRICS Conference on Measurement and Modeling of Computer Systems. ACM Press, New York, NY, USA (1990) 115–125Google Scholar
  53. 53.
    Pearce, J.P.: University of Southern California DCOP Repository (2007) http://teamcore.usc.edu/dcop/.

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Robert N. Lass
    • 1
    Email author
  • Evan A. Sultanik
    • 1
  • William C. Regli
    • 1
  1. 1.Department of Computer ScienceCollege of Engineering, Drexel UniversityPhiladelphiaUSA

Personalised recommendations