Towards an Approach for Modelling Uncertain Theory of Mind in Multi-Agent Systems

  • Ştefan Sarkadi
  • Alison R. PanissonEmail author
  • Rafael H. Bordini
  • Peter McBurney
  • Simon Parsons
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11327)


Applying Theory of Mind to multi-agent systems enables agents to model and reason about other agents’ minds. Recent work shows that this ability could increase the performance of agents, making them more efficient than agents that lack this ability. However, modelling others agents’ minds is a difficult task, given that it involves many factors of uncertainty, e.g., the uncertainty of the communication channel, the uncertainty of reading other agents correctly, and the uncertainty of trust in other agents. In this paper, we explore how agents acquire and update Theory of Mind under conditions of uncertainty. To represent uncertain Theory of Mind, we add probability estimation on a formal semantics model for agent communication based on the BDI architecture and agent communication languages.


Multi-agent systems Theory of Mind Uncertainty Socially-aware AI 



We gratefully acknowledge the partial support from CAPES and CNPq. Special thanks to Francesca Mosca for the support and for the feedback on this paper.


  1. 1.
    Apperly, I.A.: What is theory of mind? concepts, cognitive processes and individual differences. Q. J. Exp. Psychol. 65(5), 825–839 (2012)CrossRefGoogle Scholar
  2. 2.
    Barlassina, L., Gordon, R.M.: Folk psychology as mental simulation. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, summer 2017 edn. (2017)Google Scholar
  3. 3.
    Black, E., Atkinson, K.: Choosing persuasive arguments for action. In: The 10th International Conference on Autonomous Agents and Multiagent Systems, pp. 905–912 (2011)Google Scholar
  4. 4.
    El Fallah Seghrouchni, A., Dix, J., Dastani, M., Bordini, R.H. (eds.): Multi-Agent Programming. Springer, Boston, MA (2009). Scholar
  5. 5.
    Bordini, R.H., Hübner, J.F., Wooldridge, M.: Programming Multi-Agent Systems in AgentSpeak using Jason (Wiley Series in Agent Technology). Wiley, Hoboken (2007)CrossRefGoogle Scholar
  6. 6.
    Chwe, M.S.Y.: Rational Ritual. Culture, Coordination, and Common Knowledge. Princeton University Press, Princeton (2001)Google Scholar
  7. 7.
    Cohen, P.R., Perrault, C.R.: Elements of a plan-based theory of speech acts. In: Readings in Distributed Artificial Intelligence, pp. 169–186. Elsevier (1988)Google Scholar
  8. 8.
    de Weerd, H., Verheij, B.: The advantage of higher-order theory of mind in the game of limited bidding. In: Proceedings of the Workshop on Reasoning About Other Minds, CEUR Workshop Proceedings, vol. 751, pp. 149–164 (2011)Google Scholar
  9. 9.
    de Weerd, H., Verbrugge, R., Verheij, B.: Higher-order social cognition in rock-paper-scissors: a simulation study. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 1195–1196 (2012)Google Scholar
  10. 10.
    Finin, T., Fritzson, R., McKay, D., McEntire, R.: KQML as an agent communication language. In: Proceedings of the 3rd international conference on Information and knowledge management, pp. 456–463. ACM (1994)Google Scholar
  11. 11.
    FIPA, T.: FIPA communicative act library specification. Foundation for Intelligent Physical Agents (15.02.2018) (2008).
  12. 12.
    Goldman, A.I.: Theory of mind. In: The Oxford Handbook of Philosophy of Cognitive Science, 2012 edn. vol. 1, Oxford Handbooks Online (2012)Google Scholar
  13. 13.
    Gopnik, A., Glymour, C., Sobel, D.M., Schulz, L.E., Kushnir, T., Danks, D.: A theory of causal learning in children: causal maps and bayes nets. Psychol. Rev. 111(1), 3 (2004)CrossRefGoogle Scholar
  14. 14.
    Gopnik, A., Wellman, H.M.: Reconstructing constructivism: causal models, Bayesian learning mechanisms, and the theory theory. Psychol. Bull. 138(6), 1085 (2012)CrossRefGoogle Scholar
  15. 15.
    Hadidi, N., Dimopoulos, Y., Moraitis, P., et al.: Tactics and concessions for argumentation-based negotiation. In: COMMA, pp. 285–296 (2012)Google Scholar
  16. 16.
    Hadjinikolis, C., Siantos, Y., Modgil, S., Black, E., McBurney, P.: Opponent modelling in persuasion dialogues. In: International Joint Conference on Artificial Intelligence, pp. 164–170 (2013)Google Scholar
  17. 17.
    Hyslop, A.: Other minds. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, spring 2016 edn. (2016)Google Scholar
  18. 18.
    Jaynes, E.T.: Probability theory as logic. In: Fougère, P.F. (ed.) Maximum Entropy and Bayesian Methods. Springer, Dordrecht (1990). Scholar
  19. 19.
    Kumar, S., Cohen, P.R.: STAPLE: an agent programming language based on the joint intention theory. In: Proceedings of the 3rd International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 1390–1391 (2004)Google Scholar
  20. 20.
    Kumar, S., Cohen, P.R., Huber, M.J.: Direct execution of team specifications in STAPLE. In: Proceedings of the 1st International Joint Conference on Autonomous Agents and Multiagent Systems, pp. 567–568 (2002)Google Scholar
  21. 21.
    Kumar, S., Cohen, P.R., Levesque, H.J.: The adaptive agent architecture: achieving fault-tolerance using persistent broker teams. In: Proceedings of Fourth International Conference on MultiAgent Systems, pp. 159–166 (2000)Google Scholar
  22. 22.
    Labrou, Y., Finin, T.: A semantics approach for KQML - a general purpose communication language for software agents. In: Proceedings of the Third International Conference on Information and knowledge Management, pp. 447–455. ACM (1994)Google Scholar
  23. 23.
    Leudar, I., Costall, A.: On the persistence of the problem of other minds in psychology: chomsky, grice and theory of mind. Theory Psychol. 14(5), 601–621 (2004)CrossRefGoogle Scholar
  24. 24.
    Luck, M., McBurney, P.: Computing as interaction: agent and agreement technologies. In: IEEE International Conference on Distributed Human-machine Systems. IEEE Press, Citeseer (2008)Google Scholar
  25. 25.
    Mayfield, J., Labrou, Y., Finin, T.: Evaluation of KQML as an agent communication language. In: Wooldridge, M., Müller, J.P., Tambe, M. (eds.) ATAL 1995. LNCS, vol. 1037, pp. 347–360. Springer, Heidelberg (1996). Scholar
  26. 26.
    Melo, V.S., Panisson, A.R., Bordini, R.H.: Argumentation-based reasoning using preferences over sources of information. In: 15th International Conference on Autonomous Agents and Multiagent Systems (2016)Google Scholar
  27. 27.
    Oren, N., Norman, T.J.: Arguing using opponent models. In: McBurney, P., Rahwan, I., Parsons, S., Maudet, N. (eds.) ArgMAS 2009. LNCS (LNAI), vol. 6057, pp. 160–174. Springer, Heidelberg (2010). Scholar
  28. 28.
    Paglieri, F., Castelfranchi, C., da Costa Pereira, C., Falcone, R., Tettamanzi, A., Villata, S.: Trusting the messenger because of the message: feedback dynamics from information quality to source evaluation. Comput. Math. Organ. Theory 20(2), 176–194 (2014)Google Scholar
  29. 29.
    Panisson, A.R., Sarkadi, S., McBurney, P., Parsons, S., Bordini, R.H.: On the formal semantics of theory of mind in agent communication. In: 6th International Conference on Agreement Technologies (2018)Google Scholar
  30. 30.
    Panisson, A.R., Melo, V.S., Bordini, R.H.: Using preferences over sources of information in argumentation-based reasoning. In: 5th Brazilian Conference on Intelligent Systems, pp. 31–26 (2016)Google Scholar
  31. 31.
    Panisson, A.R., Meneguzzi, F., Fagundes, M., Vieira, R., Bordini, R.H.: Formal semantics of speech acts for argumentative dialogues. In: 13th International Conference on Autonomous Agents and Multiagent Systems, pp. 1437–1438 (2014)Google Scholar
  32. 32.
    Panisson, A.R., Meneguzzi, F., Vieira, R., Bordini, R.H.: Towards practical argumentation in multi-agent systems. In: Brazilian Conference on Intelligent Systems (2015)Google Scholar
  33. 33.
    Panisson, A.R., Sarkadi, S., McBurney, P., Parsons, S., Bordini, R.H.: Lies, bullshit, and deception in agent-oriented programming languages. In: Proceedings of the 20th International Trust Workshop, pp. 50–61 (2018)Google Scholar
  34. 34.
    Parsons, S., Sklar, E., McBurney, P.: Using argumentation to reason with and about trust. In: McBurney, P., Parsons, S., Rahwan, I. (eds.) ArgMAS 2011. LNCS (LNAI), vol. 7543, pp. 194–212. Springer, Heidelberg (2012). Scholar
  35. 35.
    Parsons, S., Tang, Y., Sklar, E., McBurney, P., Cai, K.: Argumentation-based reasoning in agents with varying degrees of trust. In: The 10th International Conference on Autonomous Agents and Multiagent Systems, pp. 879–886 (2011)Google Scholar
  36. 36.
    Rahwan, I., Cebrian, M.: Machine behavior needs to be an academic discipline (2018).
  37. 37.
    Rao, A.S.: AgentSpeak(L): BDI agents speak out in a logical computable language. In: Van de Velde, W., Perram, J.W. (eds.) MAAMAW 1996. LNCS, vol. 1038, pp. 42–55. Springer, Heidelberg (1996). Scholar
  38. 38.
    Rienstra, T., Thimm, M., Oren, N.: Opponent models with uncertainty for strategic argumentation. In: International Joint Conference on Artificial Intelligence, pp. 332–338 (2013)Google Scholar
  39. 39.
    Rosenschein, J.S.: Rational interaction: cooperation among intelligent agents (1986)Google Scholar
  40. 40.
    Sarkadi, S.: Deception. In: IJCAI, pp. 5781–5782 (2018)Google Scholar
  41. 41.
    Searle, J.R.: Speech Acts: An Essay in the Philosophy of Language. Cambridge University Press, Cambridge (1969)CrossRefGoogle Scholar
  42. 42.
    Thimm, M.: Strategic argumentation in multi-agent systems. KI-Künstliche Intelligenz 28(3), 159–168 (2014)CrossRefGoogle Scholar
  43. 43.
    Vieira, R., Moreira, A., Wooldridge, M., Bordini, R.H.: On the formal semantics of speech-act based communication in an agent-oriented programming language. J. Artif. Int. Res. 29(1), 221–267 (2007)zbMATHGoogle Scholar
  44. 44.
    Wooldridge, M.: Semantic issues in the verification of agent communication languages. Auton. Agent. Multi-Agent Syst. 3(1), 9–31 (2000)CrossRefGoogle Scholar
  45. 45.
    Wooldridge, M.: An Introduction to Multiagent Systems. Wiley, Hoboken (2009)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ştefan Sarkadi
    • 1
  • Alison R. Panisson
    • 2
    Email author
  • Rafael H. Bordini
    • 2
  • Peter McBurney
    • 1
  • Simon Parsons
    • 1
  1. 1.Department of InformaticsKing’s College LondonLondonUK
  2. 2.School of TechnologyPUCRSPorto AlegreBrazil

Personalised recommendations