Skip to main content

Modelling Agent Policies with Interpretable Imitation Learning

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12641)


As we deploy autonomous agents in safety-critical domains, it becomes important to develop an understanding of their internal mechanisms and representations. We outline an approach to imitation learning for reverse-engineering black box agent policies in MDP environments, yielding simplified, interpretable models in the form of decision trees. As part of this process, we explicitly model and learn agents’ latent state representations by selecting from a large space of candidate features constructed from the Markov state. We present initial promising results from an implementation in a multi-agent traffic environment.


  • Explainable artificial intelligence
  • Interpretability
  • Imitation learning
  • Representation learning
  • Decision tree
  • Traffic modelling

Supported by an EPSRC/Thales industrial CASE award in autonomous systems.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

    Icons from users Freepik and Pixel Perfect at


  1. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth & Brooks, Cole Statistics/Probability Series, California (1984)

    Google Scholar 

  2. Coppens, Y., Efthymiadis, K., Lenaerts, T., Nowé, A., Miller, T., Weber, R., Magazzeni, D.: Distilling deep reinforcement learning policies in soft decision trees. In: Proceedings of the IJCAI 2019 Workshop on Explainable AI, pp. 1–6 (2019)

    Google Scholar 

  3. Guidotti, R., Monreale, A., Giannotti, F., Pedreschi, D., Ruggieri, S., Turini, F.: Factual and counterfactual explanations for black box decision making. IEEE Intell. Syst. (2019)

    Google Scholar 

  4. Lesort, T., Díaz-Rodríguez, N., Goudou, J.F., Filliat, D.: State representation learning for control: an overview. Neural Networks 108, 379–392 (2018)

    CrossRef  Google Scholar 

  5. Samek, W., Wiegand, T., Müller, K.R.: Explainable AI: understanding, visualizing and interpreting deep learning models. arXiv:1708.08296 (2017)

  6. Turnbull, O., Lawry, J., Lowenberg, M., Richards, A.: A cloned linguistic decision tree controller for real-time path planning in hostile environments. Fuzzy Sets Syst. 293, 1–29 (2016)

    CrossRef  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Tom Bewley .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bewley, T., Lawry, J., Richards, A. (2021). Modelling Agent Policies with Interpretable Imitation Learning. In: Heintz, F., Milano, M., O'Sullivan, B. (eds) Trustworthy AI - Integrating Learning, Optimization and Reasoning. TAILOR 2020. Lecture Notes in Computer Science(), vol 12641. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-73958-4

  • Online ISBN: 978-3-030-73959-1

  • eBook Packages: Computer ScienceComputer Science (R0)