Abstract
One of the most common applications of human intelligence is social interaction, where people must make effective decisions despite uncertainty about the potential behavior of others around them. Reinforcement learning (RL) provides one method for agents to acquire knowledge about such interactions. We investigate different methods of multiagent reinforcement learning within the Sigma cognitive architecture. We leverage Sigma’s architectural mechanism for gradient descent to realize four different approaches to multiagent learning: (1) with no explicit model of the other agent, (2) with a model of the other agent as following an unknown stationary policy, (3) with prior knowledge of the other agent’s possible reward functions, and (4) through inverse reinforcement learning (IRL) of the other agent’s reward function. While the first three variations re-create existing approaches from the literature, the fourth represents a novel combination of RL and IRL for social decision-making. We show how all four styles of adaptive Theory of Mind are realized through Sigma’s same gradient descent algorithm, and we illustrate their behavior within an abstract negotiation task.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: ICML, pp. 1–8. ACM (2004)
Brown, G.W.: Iterative solution of games by fictitious play. Activity Analysis of Production and Allocation 13(1), 374–376 (1951)
Busoniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics 38(2), 156–172 (2008)
Gmytrasiewicz, P., Doshi, P.: A framework for sequential planning in multi-agent settings. JAIR 24, 49–79 (2005)
Hu, J., Wellman, M.P.: Multiagent reinforcement learning: theoretical framework and an algorithm. In: ICML, pp. 242–250 (1998)
Hu, J., Wellman, M.P.: Learning about other agents in a dynamic multiagent system. Journal of Cognitive Systems Research 2, 67–79 (2001)
Koller, D., Friedman, N.: Probabilistic graphical models: principles and techniques. MIT Press (2009)
Kschischang, F.R., Frey, B.J., Loeliger, H.A.: Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory 47(2), 498–519 (2001)
Langley, P., Laird, J.E., Rogers, S.: Cognitive architectures: Research issues and challenges. Cognitive Systems Research 10(2), 141–160 (2009)
Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: ICML, vol. 94, pp. 157–163 (1994)
Ng, A.Y., Russell, S.J.: Algorithms for inverse reinforcement learning. In: ICML, pp. 663–670 (2000)
Pynadath, D.V., Marsella, S.C.: PsychSim: Modeling theory of mind with decision-theoretic agents. In: IJCAI, pp. 1181–1186 (2005)
Pynadath, D.V., Rosenbloom, P.S., Marsella, S.C., Li, L.: Modeling two-player games in the Sigma graphical cognitive architecture. In: Kühnberger, K.-U., Rudolph, S., Wang, P. (eds.) AGI 2013. LNCS (LNAI), vol. 7999, pp. 98–108. Springer, Heidelberg (2013)
Rosenbloom, P.S.: From memory to problem solving: Mechanism reuse in a graphical cognitive architecture. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), vol. 6830, pp. 143–152. Springer, Heidelberg (2011)
Rosenbloom, P.S.: Deconstructing reinforcement learning in Sigma. In: Bach, J., Goertzel, B., Iklé, M. (eds.) AGI 2012. LNCS (LNAI), vol. 7716, pp. 262–271. Springer, Heidelberg (2012)
Rosenbloom, P.S.: Extending mental imagery in Sigma. In: Bach, J., Goertzel, B., Iklé, M. (eds.) AGI 2012. LNCS (LNAI), vol. 7716, pp. 272–281. Springer, Heidelberg (2012)
Rosenbloom, P.S., Demski, A., Han, T., Ustun, V.: Learning via gradient descent in Sigma. In: ICCM (2013)
Russell, S., Binder, J., Koller, D., Kanazawa, K.: Local learning in probabilistic networks with hidden variables. In: IJCAI, pp. 1146–1152 (1995)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)
Watkins, C.J., Dayan, P.: Q-learning. Machine Learning 8(3-4), 279–292 (1992)
Whiten, A. (ed.): Natural Theories of Mind. Basil Blackwell, Oxford (1991)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Pynadath, D.V., Rosenbloom, P.S., Marsella, S.C. (2014). Reinforcement Learning for Adaptive Theory of Mind in the Sigma Cognitive Architecture. In: Goertzel, B., Orseau, L., Snaider, J. (eds) Artificial General Intelligence. AGI 2014. Lecture Notes in Computer Science(), vol 8598. Springer, Cham. https://doi.org/10.1007/978-3-319-09274-4_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-09274-4_14
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09273-7
Online ISBN: 978-3-319-09274-4
eBook Packages: Computer ScienceComputer Science (R0)