Abstract
Today’s advanced Reinforcement Learning algorithms produce black-box policies, that are often difficult to interpret and trust for a person. We introduce a policy distilling algorithm, building on the CN2 rule mining algorithm, that distills the policy into a rule-based decision system. At the core of our approach is the fact that an RL process does not just learn a policy, a mapping from states to actions, but also produces extra meta-information, such as action values indicating the quality of alternative actions. This meta-information can indicate whether more than one action is near-optimal for a certain state. We extend CN2 to make it able to leverage knowledge about equally-good actions to distill the policy into fewer rules, increasing its interpretability by a person. Then, to ensure that the rules explain a valid, non-degenerate policy, we introduce a refinement algorithm that fine-tunes the rules to obtain good performance when executed in the environment. We demonstrate the applicability of our algorithm on the Mario AI benchmark, a complex task that requires modern reinforcement learning algorithms including neural networks. The explanations we produce capture the learned policy in only a few rules, that allow a person to understand what the black-box agent learned. Source code: https://gitlab.ai.vub.ac.be/yocoppen/svcn2.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Another illustration can be found on pages 8 and 9 of https://www.ida.liu.se/~frehe08/tailor2020/TAILOR_2020_paper_48.pdf.
- 2.
- 3.
Since BDPI is an actor-critic algorithm, we use the actor predictions as output, i.e. probabilities over actions. With value-based algorithms, e.g. DQN [18], one could record the Q-values instead.
References
Agogino, A.K., Tumer, K.: Analyzing and visualizing multiagent rewards in dynamic and stochastic domains. Auton Agents Multi-Agent Syst. 17(2), 320–338 (2008). https://doi.org/10.1007/s10458-008-9046-9
Alharin, A., Doan, T.N., Sartipi, M.: Reinforcement learning interpretation methods: a survey. IEEE Access 8, 171058–171077 (2020). https://doi.org/10.1109/ACCESS.2020.3023394
Brys, T., Nowé, A., Kudenko, D., Taylor, M.E.: Combining multiple correlated reward and shaping signals by measuring confidence. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pp. 1687–1693. AAAI Press, Palo Alto (2014)
Clark, P., Niblett, T.: The CN2 induction algorithm. Machine Learn. 3(4), 261–283 (1989). https://doi.org/10.1007/BF00116835
Coppens, Y., Efthymiadis, K., Lenaerts, T., Nowé, A.: Distilling deep reinforcement learning policies in soft decision trees. In: Miller, T., Weber, R., Magazzeni, D. (eds.) Proceedings of the IJCAI 2019 Workshop on Explainable Artificial Intelligence, Macau, pp. 1–6 (2019)
De Giacomo, G., Iocchi, L., Favorito, M., Patrizi, F.: Restraining Bolts for reinforcement learning agents. In: Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. vol. 9, pp. 13659–13662. AAAI Press, Palo Alto (2020). https://doi.org/10.1609/aaai.v34i09.7114
Frosst, N., Hinton, G.: Distilling a neural network into a soft decision tree. In: Besold, T.R., Kutz, O. (eds.) Proceedings of the First International Workshop on Comprehensibility and Explanation in AI and ML 2017. AI*IA Series, vol. 2071. CEUR Workshop Proceedings, Aachen (2017)
Fürnkranz, J., Gamberger, D., Lavrač, N.: Foundations of Rule Learning. Cognitive Technologies. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-540-75197-7
Gevaert, A., Peck, J., Saeys, Y.: Distillation of deep reinforcement learning models using fuzzy inference systems. In: Beuls, K., et al. (eds.) Proceedings of the 31st Benelux Conference on Artificial Intelligence (BNAIC 2019) and the 28th Belgian Dutch Conference on Machine Learning (Benelearn 2019), vol. 2491. CEUR Workshop Proceedings, Aachen (2019)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv e-prints arXiv:1503.02531 (2015)
Huang, J., Angelov, P.P., Yin, C.: Interpretable policies for reinforcement learning by empirical fuzzy sets. Eng. Appl. Artif. Intell. 91 (2020). https://doi.org/10.1016/j.engappai.2020.103559
Karakovskiy, S., Togelius, J.: The Mario AI benchmark and competitions. IEEE Trans. Comput. Intell. AI Games 4(1), 55–67 (2012). https://doi.org/10.1109/TCIAIG.2012.2188528
Lavrač, N., Flach, P., Zupan, B.: Rule evaluation measures: a unifying view. In: Džeroski, S., Flach, P. (eds.) ILP 1999. LNCS (LNAI), vol. 1634, pp. 174–185. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48751-4_17
Libin, P.J.K., et al.: Deep reinforcement learning for large-scale epidemic control. In: Dong, Y., Ifrim, G., Mladenić, D., Saunders, C., Van Hoecke, S. (eds.) ECML PKDD 2020. LNCS (LNAI), vol. 12461, pp. 155–170. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67670-4_10
Madumal, P., Miller, T., Sonenberg, L., Vetere, F.: Explainable reinforcement learning through a causal lens. In: Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, vol. 3, pp. 2493–2500. AAAI Press, Palo Alto (2020). https://doi.org/10.1609/aaai.v34i03.5631
Maes, P.: Computational reflection. In: Morik, K. (ed.) GWAI-87 11th German Workshop on Artifical Intelligence. Informatik-Fachberichte, vol. 152, pp. 251–265. Springer, Heidelberg (1987). https://doi.org/10.1007/978-3-642-73005-4_27
Miller, T.: Explanation in artificial intelligence: insights from the social sciences. Artif. Intell. 267, 1–38 (2019). https://doi.org/10.1016/j.artint.2018.07.007
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Molnar, C.: Interpretable Machine Learning. Leanpub, Victoria (2019)
Rusu, A.A., et al.: Policy distillation. In: International Conference on Learning Representations (2016). arXiv:1511.06295
Rückstieß, T., Sehnke, F., Schaul, T., Wierstra, D., Sun, Y., Schmidhuber, J.: Exploring parameter space in reinforcement learning. Paladyn, J. Behav. Robot. 1(1), 14–24 (2010). doi: https://doi.org/10.2478/s13230-010-0002-4
Steckelmacher, D., Plisnier, H., Roijers, D.M., Nowé, A.: Sample-efficient model-free reinforcement learning with off-policy critics. In: Brefeld, U., Fromont, E., Hotho, A., Knobbe, A., Maathuis, M., Robardet, C. (eds.) ECML PKDD 2019. LNCS (LNAI), vol. 11908, pp. 19–34. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46133-1_2
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge (2018)
Sutton, R.S., McAllester, D., Singh, S., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation. In: Neural Information Processing Systems (NIPS), pp. 1057–1063 (2000)
Tadepalli, P., Givan, R., Driessens, K.: Relational reinforcement learning: an overview. In: Tadepalli, P., Givan, R., Driessens, K. (eds.) Proceedings of the ICML-2004 Workshop on Relational Reinforcement Learning, Banff, Canada, pp. 1–9 (2004)
Todorovski, L., Flach, P., Lavrač, N.: Predictive performance of weighted relative accuracy. In: Zighed, D.A., Komorowski, J., Żytkow, J. (eds.) PKDD 2000. LNCS (LNAI), vol. 1910, pp. 255–264. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45372-5_25
Zambaldi, V., et al.: Deep reinforcement learning with relational inductive biases. In: International Conference on Learning Representations (2019)
Acknowledgements
This work is supported by the Research Foundation Flanders (FWO) [grant numbers G062819N and 1129319N], the AI Research Program from the Flemish Government (Belgium) and the Francqui Foundation. This work is part of the research program Hybrid Intelligence with project number 024.004.022, which is (partly) financed by the Dutch Ministry of Education, Culture and Science (OCW).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Coppens, Y., Steckelmacher, D., Jonker, C.M., Nowé, A. (2021). Synthesising Reinforcement Learning Policies Through Set-Valued Inductive Rule Learning. In: Heintz, F., Milano, M., O'Sullivan, B. (eds) Trustworthy AI - Integrating Learning, Optimization and Reasoning. TAILOR 2020. Lecture Notes in Computer Science(), vol 12641. Springer, Cham. https://doi.org/10.1007/978-3-030-73959-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-73959-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-73958-4
Online ISBN: 978-3-030-73959-1
eBook Packages: Computer ScienceComputer Science (R0)