Skip to main content

QueryPOMDP: POMDP-Based Communication in Multiagent Systems

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 7541)

Abstract

Decentralized Partially Observable Markov Decision Processes (Dec-POMDPs) provide powerful modeling tools for multiagent decision-making in the face of uncertainty, but solving these models comes at a very high computational cost. Two avenues for side-stepping the computational burden can be identified: structured interactions between agents and intra-agent communication. In this paper, we focus on the interplay between these concepts, namely how sparse interactions impact the communication needs. A key insight is that in domains with local interactions the amount of communication necessary for successful joint behavior can be heavily reduced, due to the limited influence between agents. We exploit this insight by deriving local POMDP models that optimize each agent’s communication behavior. Our experimental results show that our approach successfully exploits sparse interactions: we can effectively identify the situations in which it is beneficial to communicate, as well as trade off the cost of communication with overall task performance.

Keywords

  • Multiagent System
  • Markov Decision Process
  • Partially Observable Markov Decision Process
  • Partial Observability
  • Primitive Action

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This work was funded in part by Fundação para a Ciência e a Tecnologia (INESC-ID multiannual funding) through the PIDDAC Program funds and the project CMU-PT/SIA/0023/2009 under the Carnegie Mellon-Portugal Program. M.S. is funded by the FP7 Marie Curie Actions Individual Fellowship #275217 (FP7-PEOPLE-2010-IEF).

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   54.99
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   72.00
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Allen, M., Zilberstein, S.: Agent influence as a predictor of difficulty for decentralized problem-solving. In: Proc. 22nd AAAI Conf. Artificial Intelligence, pp. 688–693 (2007)

    Google Scholar 

  2. Becker, R., Zilberstein, S., Lesser, V., Goldman, C.: Transition-independent decentralized Markov decision processes. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 41–48 (2003)

    Google Scholar 

  3. Becker, R., Lesser, V., Zilberstein, S.: Decentralized Markov decision processes with event-driven interactions. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 302–309 (2004)

    Google Scholar 

  4. Oliehoek, F., Spaan, M., Whiteson, S., Vlassis, N.: Exploiting locality of interaction in factored Dec-POMDPs. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems (2008)

    Google Scholar 

  5. Spaan, M., Melo, F.: Interaction-driven Markov games for decentralized multiagent planning under uncertainty. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 525–532 (2008)

    Google Scholar 

  6. Witwicki, S., Durfee, E.: Influence-based policy abstraction for weakly-coupled Dec-POMDPs. In: Int. Conf. Automated Planning and Scheduling (2010)

    Google Scholar 

  7. Varakantham, P., Kwak, J., Taylor, M., Marecki, J., Scerri, P., Tambe, M.: Exploiting coordination locales in distributed POMDPs via social model shaping. In: Proc. 19th Int. Conf. Automated Planning and Scheduling, pp. 313–320 (2009)

    Google Scholar 

  8. Goldman, C., Zilberstein, S.: Optimizing information exchange in cooperative multiagent systems. In: Proc. 2nd Int. Conf. Autonomous Agents and Multiagent Systems, pp. 137–144 (2003)

    Google Scholar 

  9. Goldman, C., Zilberstein, S.: Communication-based decomposition mechanisms for decentralized MDPs. J. Artificial Intelligence Res. 32, 169–202 (2008)

    MathSciNet  MATH  Google Scholar 

  10. Roth, M., Simmons, R., Veloso, M.: Decentralized communication strategies for coordinated multiagent policies. In: Multi-Robot Systems: From Swarms to Intelligent Automata, pp. 93–106 (2005)

    Google Scholar 

  11. Roth, M., Simmons, R., Veloso, M.: Exploiting factored representations for decentralized execution in multiagent teams. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems, pp. 469–475 (2007)

    Google Scholar 

  12. Spaan, M., Gordon, G., Vlassis, N.: Decentralized planning under uncertainty for teams of communicating agents. In: Proc. Int. Conf. Auton. Agents and Multiagent Systems (2006)

    Google Scholar 

  13. Tasaki, M., Yabu, Y., Iwanari, Y., Yokoo, M., Tambe, M., Marecki, J., Varakantham, P.: Introducing communication in Dis-POMDPs with locality of interaction. In: IEEE/WIC/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology, vol. 2, pp. 169–175 (2008)

    Google Scholar 

  14. Wu, F., Zilberstein, S., Chen, X.: Multi-agent online planning with communication. In: Proc. Int. Conf. Automated Planning and Scheduling, pp. 321–329 (2009)

    Google Scholar 

  15. Xuan, P., Lesser, V., Zilberstein, S.: Communication decisions in multiagent cooperation: Model and experiments. In: Proc. 5th Int. Conf. Autonomous Agents, pp. 616–623 (2001)

    Google Scholar 

  16. Mostafa, H., Lesser, V.: Offline planning for communication by exploiting structured interactions in decentralized MDPs. In: IEEE/WIC/ACM Int. Conf. Web Intelligence and Intelligent Agent Technology, pp. 193–200 (2009)

    Google Scholar 

  17. Melo, F., Veloso, M.: Decentralized MDPs with sparse interactions. Artificial Intelligence 175(11), 1757–1789 (2011)

    CrossRef  MathSciNet  MATH  Google Scholar 

  18. Pynadath, D., Tambe, M.: The communicative multiagent team decision problem: Analyzing teamwork theories and models. J. Artificial Intelligence Res. 16, 389–423 (2002)

    MathSciNet  MATH  Google Scholar 

  19. Becker, R., Carlin, A., Lesser, V., Zilberstein, S.: Analyzing myopic approaches for multi-agent communications. Computational Intelligence 25(1), 31–50 (2009)

    CrossRef  MathSciNet  Google Scholar 

  20. Seuken, S., Zilberstein, S.: Formal models and algorithms for decentralized decision making under uncertainty. Auton. Agents and Multi-Agent Systems (2008)

    Google Scholar 

  21. Nair, R., Tambe, M., Yokoo, M., Pynadath, D., Marsella, S.: Taming decentralized POMDPs: Towards efficient policy computation for multiagent settings. In: Proc. 18th Int. Joint Conf. Artificial Intelligence, pp. 705–711 (2003)

    Google Scholar 

  22. Doshi, P., Gmytrasiewicz, P.: On the difficulty of achieving equilibrium in interactive POMDPs. In: Proc. 21st AAAI Conf. Artificial Intelligence, pp. 1131–1136 (2006)

    Google Scholar 

  23. Spaan, M.T.J., Vlassis, N.: Perseus: Randomized point-based value iteration for POMDPs. J. Artificial Intelligence Res. 24, 195–220 (2005)

    MATH  Google Scholar 

  24. Oliehoek, F., Spaan, M., Vlassis, N.: Optimal and approximate Q-value functions for decentralized POMDPs. J. Artificial Intelligence Res. 32, 289–353 (2008)

    MathSciNet  MATH  Google Scholar 

  25. Spaan, M., Oliehoek, F., Amato, C.: Scaling up optimal heuristic search in Dec-POMDPs via incremental expansion. In: Proc. Int. Joint Conf. Artificial Intelligence, pp. 2027–2032 (2011)

    Google Scholar 

  26. Becker, R., Zilberstein, S., Lesser, V., Goldman, C.: Solving transition independent decentralized Markov decision processes. J. Artificial Intelligence Res. 22, 423–455 (2004)

    MathSciNet  MATH  Google Scholar 

  27. Mostafa, H., Lesser, V.: A compact mathematical formulation for problems with structured agent interactions. In: Proc. AAMAS MSDM Workshop (2011)

    Google Scholar 

  28. Goldmann, C., Allen, M., Zilberstein, S.: Learning to communicate in a decentralized environment. J. Auton. Agents and Multiagent Systems 15(1), 47–90 (2007)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Melo, F.S., Spaan, M.T.J., Witwicki, S.J. (2012). QueryPOMDP: POMDP-Based Communication in Multiagent Systems. In: Cossentino, M., Kaisers, M., Tuyls, K., Weiss, G. (eds) Multi-Agent Systems. EUMAS 2011. Lecture Notes in Computer Science(), vol 7541. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34799-3_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34799-3_13

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34798-6

  • Online ISBN: 978-3-642-34799-3

  • eBook Packages: Computer ScienceComputer Science (R0)