DeepRED – Rule Extraction from Deep Neural Networks

  • Jan Ruben Zilke
  • Eneldo Loza Mencía
  • Frederik Janssen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9956)


Neural network classifiers are known to be able to learn very accurate models. In the recent past, researchers have even been able to train neural networks with multiple hidden layers (deep neural networks) more effectively and efficiently. However, the major downside of neural networks is that it is not trivial to understand the way how they derive their classification decisions. To solve this problem, there has been research on extracting better understandable rules from neural networks. However, most authors focus on nets with only one single hidden layer. The present paper introduces a new decompositional algorithm – DeepRED – that is able to extract rules from deep neural networks.

The evaluation of the proposed algorithm shows its ability to outperform a pedagogical baseline on several tasks, including the successful extraction of rules from a neural network realizing the XOR function.


  1. 1.
    Andrews, R., Diederich, J., Tickle, A.B.: Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl. Based Syst. 8(6), 373–389 (1995)CrossRefMATHGoogle Scholar
  2. 2.
    Augasta, M.G., Kathirvalavakumar, T.: Reverse engineering the neural networks for rule extraction in classification problems. Neural Process. Lett. 35(2), 131–150 (2012)CrossRefGoogle Scholar
  3. 3.
    Benítez, J.M., Castro, J.L., Requena, I.: Are artificial neural networks black boxes? IEEE Trans. Neural Netw. 8(5), 1156–1164 (1997)CrossRefGoogle Scholar
  4. 4.
    Craven, M., Shavlik, J.W.: Using sampling and queries to extract rules from trained neural networks. In: ICML, pp. 37–45 (1994)Google Scholar
  5. 5.
    Craven, M.W., Shavlik, J.W.: Extracting tree-structured representations of trained networks. In: Advances in Neural Information Processing Systems, pp. 24–30 (1996)Google Scholar
  6. 6.
    Frey, P.W., Slate, D.J.: Letter recognition using Holland-style adaptive classifiers. Mach. Learn. 6(2), 161–182 (1991)Google Scholar
  7. 7.
    Fu, L.: Rule generation from neural networks. IEEE Trans. Syst. Man Cybern. 24(8), 1114–1124 (1994)CrossRefGoogle Scholar
  8. 8.
    Johansson, U., Lofstrom, T., Konig, R., Sonstrod, C., Niklasson, L.: Rule extraction from opaque models-a slightly different perspective. In: 5th International Conference on Machine Learning and Applications, ICMLA 2006, pp. 22–27. IEEE (2006)Google Scholar
  9. 9.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  10. 10.
    Quinlan, J.R.: C4.5: Programs for Machine Learning, vol. 1. Morgan Kaufmann, San Francisco (1993)Google Scholar
  11. 11.
    Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education, New York (1995)MATHGoogle Scholar
  12. 12.
    Sato, M., Tsukimoto, H.: Rule extraction from neural networks via decision tree induction. In: Proceedings of the International Joint Conference on Neural Networks, IJCNN 2001, vol. 3, pp. 1870–1875. IEEE (2001)Google Scholar
  13. 13.
    Schmitz, G.P., Aldrich, C., Gouws, F.S.: ANN-DT: an algorithm for extraction of decision trees from artificial neural networks. IEEE Trans. Neural Netw. 10(6), 1392–1401 (1999)CrossRefGoogle Scholar
  14. 14.
    Sethi, K.K., Mishra, D.K., Mishra, B.: KDRuleEx: a novel approach for enhancing user comprehensibility using rule extraction. In: 2012 Third International Conference on Intelligent Systems, Modelling and Simulation (ISMS), pp. 55–60. IEEE (2012)Google Scholar
  15. 15.
    Setiono, R., Leow, W.K.: FERNN: an algorithm for fast extraction of rules from neural networks. Appl. Intell. 12(1–2), 15–25 (2000)CrossRefGoogle Scholar
  16. 16.
    Taha, I.A., Ghosh, J.: Symbolic interpretation of artificial neural networks. IEEE Trans. Knowl. Data Eng. 11(3), 448–463 (1999)CrossRefGoogle Scholar
  17. 17.
    Thrun, S.: Extracting provably correct rules from artificial neural networks. Technical report, University of Bonn, Institut für Informatik III (1993)Google Scholar
  18. 18.
    Thrun, S.: Extracting rules from artificial neural networks with distributed representations. In: Advances in neural information processing systems, pp. 505–512 (1995)Google Scholar
  19. 19.
    Towell, G.G., Shavlik, J.W.: Extracting refined rules from knowledge-based neural networks. Mach. Learn. 13(1), 71–101 (1993)Google Scholar
  20. 20.
    Tsukimoto, H.: Extracting rules from trained neural networks. IEEE Trans. Neural Netw. 11(2), 377–389 (2000)CrossRefGoogle Scholar
  21. 21.
    Zhou, Z.H., Chen, S.F., Chen, Z.Q.: A statistics based approach for extracting priority rules from trained neural networks. In: Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks, IJCNN 2000, vol. 3, pp. 401–406. IEEE (2000)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Jan Ruben Zilke
    • 1
  • Eneldo Loza Mencía
    • 1
  • Frederik Janssen
    • 1
  1. 1.Knowledge Engineering GroupTechnische Universität DarmstadtDarmstadtGermany

Personalised recommendations