Modeling the Opponent’s Action Using Control-Based Reinforcement Learning
In this paper, we propose an alternative to model-free reinforcement learning approaches that recently have demonstrated Theory-of-Mind like behaviors. We propose a game theoretic approach to the problem in which pure RL has demonstrated to perform below the standards of human-human interaction. In this context, we propose alternative learning architectures that complement basic RL models with the ability to predict the other’s actions. This architecture is tested in different scenarios where agents equipped with similar or varying capabilities compete in a social game. Our different interaction scenarios suggest that our model-based approaches are especially effective when competing against models of equivalent complexity, in contrast to our previous results with more basic predictive architectures. We conclude that the evolution of mechanisms that allow for the control of other agents provide different kinds of advantages that can become significant when interacting with different kinds of agents. We argue that no single proposed addition to the learning architecture is sufficient to optimize performance in these scenarios, but a combination of the different mechanisms suggested is required to achieve near-optimal performance in any case.
KeywordsMulti-agent models Cognitive architectures Theory of mind Game theory Social decision-making Reinforcement learning
The research leading to these results has received funding from the European Commission’s Horizon 2020 socSMC project (socSMC-641321H2020-FETPROACT-2014) and by the European Research Council’s CDAC project (ERC-2013-ADG341196).
- 1.Rabinowitz, N.C., Perbet, F., Song, H.F., Zhang, C., Eslami, S.M., Botvinick, M.: Machine Theory of Mind (2018). arXiv preprint arXiv:1802.07740
- 2.Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations (2017). arXiv preprint arXiv:1703.04908
- 4.Freire, I.T., Moulin-Frier, C., Sanchez-Fibla, M., Arsiwalla, X.D., Verschure, P.: Modeling the Formation of Social Conventions in Multi-Agent Populations (2018). arXiv preprint arXiv:1802.06108
- 6.Moulin-Frier, C., Arsiwalla, X.D., Puigbo, J.Y., Sanchez-Fibla, M., Duff, A., Verschure, P.F.: Top-down and bottom-up interactions between low-level reactive control and symbolic rule learning in embodied agents. In: CoCo@ NIPS (2016)Google Scholar
- 7.Braitenberg, V.: Vehicles: Experiments in Synthetic Psychology. MIT Press, Cambridge (1986)Google Scholar
- 8.Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9–44 (1988)Google Scholar
- 9.Moulin-Frier, C., Puigbo, J.Y., Arsiwalla, X.D., Sanchez-Fibla, M., Verschure, P.F.: Embodied artificial intelligence through distributed adaptive control: An integrated framework (2017). arXiv preprint arXiv:1704.01407
- 10.Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Sanchez, M., Verschure, P.F.: Is consciousness a control process? Artificial Intelligence Research and Development, pp. 233–238. IOS Press, Amsterdam (2016)Google Scholar
- 12.Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Verschure, P.: Consciousness as an Evolutionary Game-Theoretic, Strategy, pp. 509–514 (2017)Google Scholar
- 13.Arsiwalla, X.D., Moulin-Frier, C., Herreros, I., Sanchez-Fibla, M., Verschure, P.: The Morphospace of Consciousness (2017). ArXiv preprint arXiv:1705.11190