Modeling the Opponent’s Action Using Control-Based Reinforcement Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10928)


In this paper, we propose an alternative to model-free reinforcement learning approaches that recently have demonstrated Theory-of-Mind like behaviors. We propose a game theoretic approach to the problem in which pure RL has demonstrated to perform below the standards of human-human interaction. In this context, we propose alternative learning architectures that complement basic RL models with the ability to predict the other’s actions. This architecture is tested in different scenarios where agents equipped with similar or varying capabilities compete in a social game. Our different interaction scenarios suggest that our model-based approaches are especially effective when competing against models of equivalent complexity, in contrast to our previous results with more basic predictive architectures. We conclude that the evolution of mechanisms that allow for the control of other agents provide different kinds of advantages that can become significant when interacting with different kinds of agents. We argue that no single proposed addition to the learning architecture is sufficient to optimize performance in these scenarios, but a combination of the different mechanisms suggested is required to achieve near-optimal performance in any case.


Multi-agent models Cognitive architectures Theory of mind Game theory Social decision-making Reinforcement learning 



The research leading to these results has received funding from the European Commission’s Horizon 2020 socSMC project (socSMC-641321H2020-FETPROACT-2014) and by the European Research Council’s CDAC project (ERC-2013-ADG341196).


  1. 1.
    Rabinowitz, N.C., Perbet, F., Song, H.F., Zhang, C., Eslami, S.M., Botvinick, M.: Machine Theory of Mind (2018). arXiv preprint arXiv:1802.07740
  2. 2.
    Mordatch, I., Abbeel, P.: Emergence of grounded compositional language in multi-agent populations (2017). arXiv preprint arXiv:1703.04908
  3. 3.
    Hawkins, R.X.D., Goldstone, R.L.: The formation of social conventions in real-time environments. PLoS One 11, e0151670 (2016)CrossRefGoogle Scholar
  4. 4.
    Freire, I.T., Moulin-Frier, C., Sanchez-Fibla, M., Arsiwalla, X.D., Verschure, P.: Modeling the Formation of Social Conventions in Multi-Agent Populations (2018). arXiv preprint arXiv:1802.06108
  5. 5.
    Verschure, P.F.M.J., Voegtlin, T., Douglas, R.J.: Environmentally mediated synergy between perception and behaviour in mobile robots. Nature 425, 620–624 (2003)CrossRefGoogle Scholar
  6. 6.
    Moulin-Frier, C., Arsiwalla, X.D., Puigbo, J.Y., Sanchez-Fibla, M., Duff, A., Verschure, P.F.: Top-down and bottom-up interactions between low-level reactive control and symbolic rule learning in embodied agents. In: CoCo@ NIPS (2016)Google Scholar
  7. 7.
    Braitenberg, V.: Vehicles: Experiments in Synthetic Psychology. MIT Press, Cambridge (1986)Google Scholar
  8. 8.
    Sutton, R.S.: Learning to predict by the methods of temporal differences. Mach. Learn. 3, 9–44 (1988)Google Scholar
  9. 9.
    Moulin-Frier, C., Puigbo, J.Y., Arsiwalla, X.D., Sanchez-Fibla, M., Verschure, P.F.: Embodied artificial intelligence through distributed adaptive control: An integrated framework (2017). arXiv preprint arXiv:1704.01407
  10. 10.
    Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Sanchez, M., Verschure, P.F.: Is consciousness a control process? Artificial Intelligence Research and Development, pp. 233–238. IOS Press, Amsterdam (2016)Google Scholar
  11. 11.
    Arsiwalla, X.D., Herreros, I., Verschure, P.: On three categories of conscious machines. In: Conference on Biomimetic and Biohybrid Systems, pp. 389–392 (2016)CrossRefGoogle Scholar
  12. 12.
    Arsiwalla, X.D., Herreros, I., Moulin-Frier, C., Verschure, P.: Consciousness as an Evolutionary Game-Theoretic, Strategy, pp. 509–514 (2017)Google Scholar
  13. 13.
    Arsiwalla, X.D., Moulin-Frier, C., Herreros, I., Sanchez-Fibla, M., Verschure, P.: The Morphospace of Consciousness (2017). ArXiv preprint arXiv:1705.11190

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.IBEC, Institute for BioEngineering of CataloniaBarcelonaSpain
  2. 2.UPF, Universitat Pompeu FabraBarcelonaSpain
  3. 3.BIST, Barcelona Institue of Science and TechnologyBarcelonaSpain
  4. 4.ICREA, Catalan Institute for Research and Advanced StudiesBarcelonaSpain

Personalised recommendations