Autonomous Agents and Multi-Agent Systems

, Volume 6, Issue 1, pp 77–107 | Cite as

Predicting the Expected Behavior of Agents that Learn About Agents: The CLRI Framework

  • José M. Vidal
  • Edmund H. Durfee

Abstract

We describe a framework and equations used to model and predict the behavior of multi-agent systems (MASs) with learning agents. A difference equation is used for calculating the progression of an agent's error in its decision function, thereby telling us how the agent is expected to fare in the MAS. The equation relies on parameters which capture the agent's learning abilities, such as its change rate, learning rate and retention rate, as well as relevant aspects of the MAS such as the impact that agents have on each other. We validate the framework with experimental results using reinforcement learning agents in a market system, as well as with other experimental results gathered from the AI literature. Finally, we use PAC-theory to show how to calculate bounds on the values of the learning parameters.

multi-agent systems machine learning complex systems 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    W. B. Arthur, S. Durlauf, and D. Lane (ed.), The Economy as an Evolving Complex System II, Series in the Sciences of Complexity, Addison-Wesley: Reading, MA, 1997.Google Scholar
  2. 2.
    W. B. Arthur, J. H. Holland, B. LeBaron, R. Palmer, and P. Tayler. “Asset pricing under endogenous expectations in an artificial stock market, ” in W. B. Arthur, S. Durlauf, and D. Lane (eds.), The Economy as an Evolving Complex System II, Addison-Wesley: Reading, MA, 1997.Google Scholar
  3. 3.
    K. Binmore, “Modeling rational players, part 2, ” Economics and Philosophy, vol. 4, pp. 9–55, 1988.Google Scholar
  4. 4.
    D. Carmel and S. Markovitch, “Exploration and adaptation in multiagent systems: A model-based approach, ” in Proceedings of the International Joint Conference on AI, 1997.Google Scholar
  5. 5.
    A. Chavez and P. M. Kasbah, “An agent marketplace for buying and selling goods, ” in First International Conference on the Practical Application of Intelligent Agents and Multi-Agent Technology, pp. 75–90, 1996.Google Scholar
  6. 6.
    C. Claus and C. Boutilier, “The dynamics of reinforcement learning in cooperative multiagent systems, ” in Proceedings of Workshop on Multiagent Learning, AAAI Press, 1997.Google Scholar
  7. 7.
    G. A. Cowan, D. Pines, and D. Meltzer (eds.), Complexity: Metaphors, Models and Reality, Addison-Wesley: Reading, MA, 1995.Google Scholar
  8. 8.
    V. M. Darley and S. Kauffman, “Natural rationality, ” in W. B. Arthur, S. Durlaf, and D. Lane (eds.), The Economy as an Evolving Complex System II, Addison-Wesley: Reading, MA, 1997.Google Scholar
  9. 9.
    E. H. Durfee, D. L. Kiskis, and W. P. Birmingham, “The agent architecture of the University of Michigan Digital Library, ” IEE Proceedings on Software Engineering, vol. 144, no.1, pp. 61–71, 1997.Google Scholar
  10. 10.
    O. Etzioni, “Moving up the information food chain: Deploying softbots on the world wide web, ” in Proceedings of the Thirteenth National Conference on Artificial Intelligence and the Eighth Innovative Applications of Artificial Intelligence Conference, Menlo Park: AAAI Press, MIT Press, pp. 1322–1326, 1996.Google Scholar
  11. 11.
    J. H. Holland, Hidden Order, Addison-Wesley, 1995.Google Scholar
  12. 12.
    J. Hu and M. P. Wellman, “Self-fulfilling bias in multiagent learning, ” in Proceedings of the Second International Conference on Multi-Agent Systems, pp. 118–125, 1996.Google Scholar
  13. 13.
    J. Hu and M. P. Wellman, “Multiagent reinforcement learning: Theoretical framework and an algorithm, ” in Proceedings of the Fifteenth International Conference on Machine Learning, 1998.Google Scholar
  14. 14.
    J. Hu and M. P. Wellman, “Online learning about other agents in a dynamic multiagent system, ” in Proceedings of the Second International Conference on Autonomous Agents, 1998.Google Scholar
  15. 15.
    A. Hübler and D. Pines, “Prediction and Adaptation in an Evolving Chaotic Environment, ” in G. A. Cowan, D. Pines, and D. Meltzer (eds.), Complexity: Methaphors, Models and Reality, pp. 343–379, 1994.Google Scholar
  16. 16.
    T. Ishida, Real-Time Search for Learning Autonomous Agents, Kluwer Academic Publishers, 1997.Google Scholar
  17. 17.
    M. J. Kearns and U. V. Vazirani, An Introduction to Computational Learning Theory, The MIT Press: Cambridge, MA, 1994.Google Scholar
  18. 18.
    J. O. Kephart, T. Hogg, and B. A. Huberman, “Collective behavior of predictive agents, ” Physica D, vol. 42, no.2, pp. 48–65, 1990.Google Scholar
  19. 19.
    V. R. Lesser and D. D. Corkill, “Functionally accurate, cooperative distributed systems, ” IEEE Transactions on Systems, Man, and Cybernetics, vol. 11, no.1, pp. 81–96, 1981.Google Scholar
  20. 20.
    M. L. Littman, “Markov games as a framework for multi-agent reinforcement learning, ” in Proceedings of the Eleventh International Conference on Machine Learning, Morgan Kaufmann, pp. 157–163, 1994.Google Scholar
  21. 21.
    J. S. Liu and K. P. Sycara, “Multiagent coordination in tightly coupled real-time environments, ” in Victor Lesser (ed.), Proceedings of the First International Conference on Multi-Agent Systems, MIT Press, 1995.Google Scholar
  22. 22.
    M. J. Matarić, “Issues and approaches in the design of collective autonomous agents, ” Robotics and Autonomous Systems, vol. 16, nos.2–4, pp. 321–331, 1995.Google Scholar
  23. 23.
    M. J. Matarić, “Learning social behaviors, ” Robotics and Autonomous Agents, vol. 20, pp. 191–204, 1997.Google Scholar
  24. 24.
    T. M. Mitchell, Machine Learning, McGraw-Hill, 1997.Google Scholar
  25. 25.
    J. S. Rosenschein and G. Zlotkin, Rules of Encounter, The MIT Press: Cambridge, MA, 1994.Google Scholar
  26. 26.
    S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach, Prentice Hall, 1995.Google Scholar
  27. 27.
    S. Sen (ed.), Working Notes from the AAAI Symposium on Adaptation, Co-evolution and Learning in Multiagent Systems, 1996.Google Scholar
  28. 28.
    S. Sen and M. Sekaran, “Individual learning of coordination knowledge, ” Journal of Experimental and Theoretical AI, pp. 156–170, 1998.Google Scholar
  29. 29.
    S. Sen, M. Sekaran, and J. Hale, “Learning to coordinate without sharing information, ” in Proceedings of the Twelfth National Conference on Artificial Intelligence, 1994.Google Scholar
  30. 30.
    Y. Shoham and M. Tennenholtz, “On the emergence of social conventions: modeling, analysis, and simulations, ” Artificial Intelligence, vol. 94, no.1, pp. 139–166, 1997.Google Scholar
  31. 31.
    P. Stone and M. Velos, “Multiagent systems: A survery from a machine learning perspective, ” Technical Report CMU-CS–97–193, Carnegie Mellon University, December 1997.Google Scholar
  32. 32.
    P. Stone and M. Veloso, “Team-partitioned, opaque-transition reinforcement learning, ” in Proceedings of the Third International Conference on Autonomous Agents, Springer-Verlag, 1999.Google Scholar
  33. 33.
    J. M. Vidal, “Computational agents that learn about agents: Algorithms for their design and a predictive theory of their behavior, ” Ph.D. thesis, University of Michigan, 1998.Google Scholar
  34. 34.
    J. M. Vidal and E. H. Durfee, “Learning nested models in an information economy, ” Journal of Experimental and Theoretical Artificial Intelligence, vol. 10, no.3, pp. 291–308, 1998.Google Scholar
  35. 35.
    C. J. Watkins and P. Dayan, “Q-learning, ” Machine Learning, vol. 8, pp. 279–292, 1992.Google Scholar
  36. 36.
    G. Weiß (ed.), Distributed Artificial Intelligence Meets Machine Learning, Springer, 1997.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • José M. Vidal
    • 1
  • Edmund H. Durfee
    • 2
  1. 1.Swearingen Engineering CenterUniversity of South CarolinaColumbia
  2. 2.Advanced Technology LaboratoryUniversity of MichiganAnn Arbor

Personalised recommendations