Memory-Based Explainable Reinforcement Learning

  • Francisco CruzEmail author
  • Richard Dazeley
  • Peter Vamplew
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11919)


Reinforcement learning (RL) is a learning approach based on behavioral psychology used by artificial agents to learn autonomously by interacting with their environment. An open issue in RL is the lack of visibility and understanding for end-users in terms of decisions taken by an agent during the learning process. One way to overcome this issue is to endow the agent with the ability to explain in simple terms why a particular action is taken in a particular situation. In this work, we propose a memory-based explainable reinforcement learning (MXRL) approach. Using an episodic memory, the RL agent is able to explain its decisions by using the probability of success and the number of transactions to reach the goal state. We have performed experiments considering two variations of a simulated scenario, namely, an unbounded grid world with aversive regions and a bounded grid world. The obtained results show that the agent, using information extracted from the memory, is able to explain its behavior in an understandable manner for non-expert end-users at any moment during its operation.


Reinforcement learning Explainable reinforcement learning Human-aligned artificial intelligence 


  1. 1.
    Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018)Google Scholar
  2. 2.
    Conrad, B., Gross, D., Fogg, L., Ruchala, P.: Maternal confidence, knowledge, and quality of mother-toddler interactions: a preliminary study. Infant Mental Health J. 13(4), 353–362 (1992)Google Scholar
  3. 3.
    Cruz, F., Acuña, G., Cubillos, F., Moreno, V., Bassi, D.: Indirect training of grey-box models: application to a bioprocess. In: Liu, D., Fei, S., Hou, Z., Zhang, H., Sun, C. (eds.) ISNN 2007. LNCS, vol. 4492, pp. 391–397. Springer, Heidelberg (2007). Scholar
  4. 4.
    Cruz, F., Magg, S., Nagai, Y., Wermter, S.: Improving interactive reinforcement learning: what makes a good teacher? Connect. Sci. 30(3), 306–325 (2018)CrossRefGoogle Scholar
  5. 5.
    Dulac-Arnold, G., Mankowitz, D., Hester, T.: Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 (2019)
  6. 6.
    Gunning, D.: Explainable artificial intelligence (XAI). Defense Advanced Research Projects Agency (DARPA), nd Web (2017)Google Scholar
  7. 7.
    Hein, D., Udluft, S., Runkler, T.A.: Interpretable policies for reinforcement learning by genetic programming. Eng. Appl. Artif. Intell. 76, 158–169 (2018)CrossRefGoogle Scholar
  8. 8.
    Langley, P., Meadows, B., Sridharan, M., Choi, D.: Explainable agency for intelligent autonomous systems. In: Twenty-Ninth IAAI Conference, pp. 4762–4763 (2017)Google Scholar
  9. 9.
    Lim, B.Y., Dey, A.K., Avrahami, D.: Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2119–2128. ACM (2009)Google Scholar
  10. 10.
    Madumal, P., Miller, T., Sonenberg, L., Vetere, F.: Explainable reinforcement learning through a causal lens. arXiv preprint arXiv:1905.10958 (2019)
  11. 11.
    Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2018)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Niv, Y.: Reinforcement learning in the brain. J. Math. Psychol. 53, 139–154 (2009)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Pocius, R., Neal, L., Fern, A.: Strategic tasks for explainable reinforcement learning. In: The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI 2019), p. 2 (2019)CrossRefGoogle Scholar
  14. 14.
    Robertson, S.B., Weismer, S.E.: Effects of treatment on linguistic and social skills in toddlers with delayed language development. J. Speech Lang. Hearing Res. 42(5), 1234–1248 (1999)CrossRefGoogle Scholar
  15. 15.
    Sequeira, P., Yeh, E., Gervasio, M.T.: Interestingness elements for explainable reinforcement learning through introspection. In: IUI Workshops, p. 7 (2019)Google Scholar
  16. 16.
    Shu, T., Xiong, C., Socher, R.: Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv:1712.07294 (2017)
  17. 17.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. Bradford Book, Cambridge (1998)zbMATHGoogle Scholar
  18. 18.
    Tabrez, A., Hayes, B.: Improving human-robot interaction through explainable reinforcement learning. In: 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pp. 751–753. IEEE (2019)Google Scholar
  19. 19.
    Verma, A., Murali, V., Singh, R., Kohli, P., Chaudhuri, S.: Programmatically interpretable reinforcement learning. arXiv preprint arXiv:1804.02477 (2018)
  20. 20.
    Wang, X., Chen, Y., Yang, J., Wu, L., Wu, Z., Xie, X.: A reinforcement learning framework for explainable recommendation. In: 2018 IEEE International Conference on Data Mining (ICDM), pp. 587–596. IEEE (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Francisco Cruz
    • 1
    Email author
  • Richard Dazeley
    • 1
  • Peter Vamplew
    • 2
  1. 1.School of Information TechnologyDeakin UniversityGeelongAustralia
  2. 2.School of Science, Engineering and Information TechnologyFederation UniversityBallaratAustralia

Personalised recommendations