Effects of Input Addition in Learning for Adaptive Games: Towards Learning with Structural Changes

  • Iago BonniciEmail author
  • Abdelkader Gouaïch
  • Fabien Michel
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11454)


Adaptive Games (AG) involve a controller agent that continuously feeds from player actions and game state to tweak a set of game parameters in order to maintain or achieve an objective function such as the flow measure defined by Csíkszentmihályi. This can be considered a Reinforcement Learning (RL) situation, so that classical Machine Learning (ML) approaches can be used. On the other hand, many games naturally exhibit an incremental gameplay where new actions and elements are introduced or removed progressively to enhance player’s learning curve or to introduce variety within the game. This makes the RL situation unusual because the controller agent input/output signature can change over the course of learning. In this paper, we get interested in this unusual “protean” learning situation (PL). In particular, we assess how the learner can rely on its past shapes and experience to keep improving among signature changes without needing to restart the learning from scratch on each change. We first develop a rigorous formalization of the PL problem. Then, we address the first elementary signature change: “input addition”, with Recurrent Neural Networks (RNNs) in an idealized PL situation. As a first result, we find that it is possible to benefit from prior learning in RNNs even if the past controller agent signature has less inputs. The use of PL in AG thus remains encouraged. Investigating output addition, input/output removal and translating these results to generic PL will be part of future works.


Adaptive games Reinforcement Learning Transfer Learning Recurrent networks 


  1. 1.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series, 2nd edn. MIT Press, Cambridge (2018)zbMATHGoogle Scholar
  2. 2.
    Hanna, C.J., Hickey, R.J., Charles, D.K., Black, M.M.: Modular reinforcement learning architectures for artificially intelligent agents in complex game environments. In: Computational Intelligence and Games. pp. 380–387. IEEE, Copenhagen, August 2010Google Scholar
  3. 3.
    Elman, J.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)CrossRefGoogle Scholar
  4. 4.
    Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)CrossRefGoogle Scholar
  5. 5.
    Taylor, M.E., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10(7), 1633–1685 (2009)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Lazaric, A.: Transfer in reinforcement learning: a framework and a survey. In: Wiering, M., van Otterlo, M. (eds.) Reinforcement Learning. Adaptation, Learning, and Optimization, vol. 12, pp. 143–173. Springer, Heidelberg (2012). Scholar
  7. 7.
    Tanaka, F., Yamamura, M.: Multitask reinforcement learning on the distribution of MDPs. In: International Symposium on Computational Intelligence in Robotics and Automation. Computational Intelligence in Robotics and Automation for the New Millennium, vol. 3, pp. 1108–1113. IEEE, Kobe (2003)Google Scholar
  8. 8.
    Devin, C., Gupta, A., Darrell, T., Abbeel, P., Levine, S.: Learning modular neural network policies for multi-task and multi-robot transfer. CoRR abs/1609.07088 (2016)Google Scholar
  9. 9.
    Frans, K., Ho, J., Chen, X., Abbeel, P., Schulman, J.: Meta learning shared hierarchies. CoRR abs/1710.09767 (2017)Google Scholar
  10. 10.
    Teh, Y.W., et al.: Distral: robust multitask reinforcement learning. CoRR abs/1707.04175 (2017)Google Scholar
  11. 11.
    Tsymbal, A.: The Problem of Concept Drift: Definitions and Related Work (2004)Google Scholar
  12. 12.
    Wang, H., Abraham, Z.: Concept drift detection for streaming data. In: International Joint Conference on Neural Networks, pp. 1–9. IEEE, Killarney, July 2015Google Scholar
  13. 13.
    Ring, M.B.: Continual learning in reinforcement environments. Ph.D. thesis, University of Texas at Austin, Austin (1994)Google Scholar
  14. 14.
    Xu, J., Zhu, Z.: Reinforced continual learning. CoRR abs/1805.12369 (2018)Google Scholar
  15. 15.
    Sweetser, P., Wyeth, P.: GameFlow: a model for evaluating player enjoyment in games. Comput. Entertainment 3(3), 3 (2005)CrossRefGoogle Scholar
  16. 16.
    Holt, R., Mitterer, J.: Examining video game immersion as a flow state. In: 108th Annual Psychological Association, Washington, DC (2000)Google Scholar
  17. 17.
    Chen, J.: flOw, January 2019.
  18. 18.
    Chen, J.: Flow in games (and everything else). ACM Commun. 50(4), 31 (2007)CrossRefGoogle Scholar
  19. 19.
    Bonnici, I., Gouaïch, A.: Formalisation of metamorph reinforcement learning. Technical report, LIRMM, Montpellier, November 2018.
  20. 20.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078 (2014)Google Scholar
  21. 21.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2014)Google Scholar
  22. 22.
    Paszke, A., et al.: Automatic differentiation in PyTorch (2017)Google Scholar
  23. 23.
    R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2018)Google Scholar
  24. 24.
    Le Hy, R., Arrigoni, A., Bessiere, P., Lebeltel, O.: Teaching Bayesian behaviours to video game characters. Robot. Auton. Syst. 47, 177–185 (2004)CrossRefGoogle Scholar
  25. 25.
    Tencé, F., Buche, C.: Automatable evaluation method oriented toward behaviour believability for video games. CoRR abs/1009.0501 (2010)Google Scholar
  26. 26.
    Polceanu, M., Mora, A., Jimenez, J., Buche, C., Fernandez-Leiva, A.: The believability gene in virtual bots. In: 29th International Flairs, p. 4. AAAI Press, Key Largo (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Iago Bonnici
    • 1
    Email author
  • Abdelkader Gouaïch
    • 1
  • Fabien Michel
    • 1
  1. 1.LIRMM, Université de Montpellier, CNRSMontpellierFrance

Personalised recommendations