Skip to main content
SpringerLink
Log in

Review of Research in the Field of Developing Methods to Extract Rules From Artificial Neural Networks

  • ARTIFICIAL INTELLIGENCE
  • Published:
Journal of Computer and Systems Sciences International Aims and scope

Abstract

A large-scale review and analysis of the existing methods and approaches to extract rules from artificial neural networks, including deep learning neural networks, is carried out. A wide range of methods and approaches to extract rules and related approaches to develop explainable artificial intelligence (AI) systems are considered. The taxonomy and several directions in studies of explainable neural networks related to the extraction of rules from neural networks, which allow the user to get an idea of how the neural network uses the input data, and also, using rules, to reveal the hidden relationships of the input data and the results found, are explored. This review focuses on the relationship of the most common rule-based explanation systems in AI with the most powerful machine learning algorithms using neural networks. In addition to rule extraction, other methods of constructing explainable AI systems are considered based on the construction of special modules that interpret each step of changing the neural network’s weights. A comprehensive analysis of the existing research makes it possible to draw conclusions about the appropriateness of using certain approaches. The results of the analysis will allow us to get a detailed picture of the state of research in this area and create our own applications based on neural networks, the results of which can be studied in detail and their reliability evaluated. The development of such systems is necessary for the development of the digital economy in Russia and the creation of applications that allow making responsible and explainable management decisions in critical areas of the national economy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.

Similar content being viewed by others

REFERENCES

  1. M. Bilgic and R. J. Mooney, “Explaining recommendations: Satisfaction vs. promotion,” in Proceedings of the Beyond Personalization Workshop, 2005, Vol. 5, p. 153.

  2. W. R. Swartout and J. D. Moore, “Explanation in second generation expert systems,” in Second Generation Expert Systems (Springer, Berlin, 1993), pp. 543–585.

    Book  Google Scholar 

  3. B. Chandrasekaran, M. C. Tanner, and J. R. Josephson, “Explaining control strategies in problem solving,” IEEE Expert 4 (1), 9–15 (1989).

    Article  Google Scholar 

  4. J. S. Dhaliwal and I. Benbasat, “The use and effects of knowledge-based system explanations: theoretical foundations and a framework for empirical evaluation,” Inform. Syst. Res. 7, 342–362 (1996).

    Article  Google Scholar 

  5. M. M. Eining and P. B. Dorr, “The impact of expert system usage on experiential learning in an auditing setting,” Inform. Syst. 5, 1–16 (1991).

    Google Scholar 

  6. D. S. Murphy, “Expert system use and the development of expertise in auditing: A preliminary investigation,” Inform. Syst. 4, 18–35 (1990).

    Google Scholar 

  7. D. M. Lamberti and W. A. Walace, “Intelligent interface design: An empirical assessment of knowledge presentation in expert systems,” MIS Quart. 14, 279–311 (1990).

    Article  Google Scholar 

  8. Artificial Intelligence, The Reference Book, Ed. by V. N. Zakharov, E. V. Popov, D. A. Pospelov, and V. F. Khoroshevskii (Radio Svyaz’, Moscow, 1990) [in Russian].

    Google Scholar 

  9. E. V. Popov, Expert Systems: Solving Unformalized Tasks in a Dialogue with a Computer (Nauka, Moscow, 1987) [in Russian].

    Google Scholar 

  10. W. R. Swartout, “A digitalis therapy advisor with explanations,” in Proceedings of the 5th International Joint Conference on Artificial Intelligence (Cambridge, 1977), Vol. 2, pp. 819–825.

  11. J. L. Weiner, “BLAH, A system that explains its reasoning,” Artif. Intell. 15, 19–48 (1980).

    Article  Google Scholar 

  12. W. R. Swartout, C. Paris, and J. Moore, “Explanations in knowledge systems: Design for explainable expert systems,” IEEE Expert 6 (3), 58–64 (1991).

    Article  Google Scholar 

  13. W. J. Clancey, Intelligent Tutoring Systems: A Tutorial Survey (Stanford Univ. Dep. Comput. Sci., Stanford, 1986).

  14. R. Sinha and K. Swearingen, “The role of transparency in recommender systems,” in Extended Abstracts of CHI'02 Conference on Human Factors in Computing Systems (Minneapolis, 2002), pp. 830–831.

  15. T. Gruber, “Learning why by being told what,” IEEE Expert 6 (4), 65–75 (1991).

    Article  Google Scholar 

  16. W. J. Clancey, “Details of the revised therapy algorithm,” in Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project (Addison-Wesley, Reading, MA, 1984), pp. 133–146.

    Google Scholar 

  17. W. J. Clancey, “From GUIDON to NEOMYCIN and HERACLES in twenty short lessons,” AI Magazine 7 (3), 40 (1986).

    Google Scholar 

  18. A. Arioua, P. Buche, and M. Croitoru, “Explanatory dialogs with argumentative faculties over inconsistent knowledge bases,” Expert Syst. Appl. 80, 244–262 (2017).

    Article  Google Scholar 

  19. U. Johansson, T. Lofstrom, R. Konig, C. Sonstro, and L. Nilsson, “Rule extraction from opaque models-a slightly different perspective,” in Proceedings of the 5th International Conference on Machine Learning and Applications (ICMLA’06), Orlando, FL, USA, 2006, pp. 22–27.

  20. M. Craven and J. Shavlik, “Rule extraction: Where do we go from here,” in University of Wisconsin Machine Learning Research Group Working Paper (Wisconsin, 1999), pp. 99–108.

  21. R. Andrew, J. Diederich, and A. B. Tickle, “Survey and critique of techniques for extracting rules from trained artificial networks,” Knowledge-based Syst. 8, 373–389 (1995).

    Article  Google Scholar 

  22. S. Thrum, “Extracting provably correct rules from artificial neural networks,” Technical Report (Inst. Inform. III, Bonn, 1993).

  23. M. Craven and J. W. Shavlik, “Using sampling and queries to extract rules from trained neural networks,” in Proceedings of the 11th International Conference, Rutgers Univ., New Brunswick, USA, 1994, pp. 37–45.

  24. L. Fu, “Rule generation from neural networks,” IEEE Trans. Syst. Man Cybern. 24, 1114–1124 (1994).

    Article  Google Scholar 

  25. M. Sato and H. Tsukimoto, “Rule extraction from neural networks via decision tree induction,” in Proceedings of International Joint Conference on Neural Networks (IJCNN'01) (Washington, DC, 2001), Vol. 3, pp. 1870–1875.

  26. A. B. Tickle, R. Andrew, M. Golea, and J. Diederich, “The truth will come to light: Directions and challenges in extracting the knowledge embedded within trained artificial neural networks,” IEEE Trans. Neural Networks 9, 1057–1068 (1998).

    Article  Google Scholar 

  27. K. K. Sethi, D. K. Mishra, and B. Mishra, “KDRuleEx: A novel approach for enhancing user comprehensibility using rule extraction,” in Intelligent Systems, Modelling and Simulation (ISMS), Proceedings of the 3rd International Conference, Kota Kinabalu, Malaysia, 2012, pp. 55–60.

  28. U. Johansson, T. Lofstrom, R. Konig, C. Sonstrod, and L. Niklasson, “Rule extraction from opaque models-a slightly different perspective,” in Proceedings of the 5th International Conference on Machine Learning and Applications, ICMLA’06, Orlando, FL, USA, 2006, pp. 22–27.

  29. M. Rangwala and G. R. Weckman, “Extracting rules from artificial neural networks utilizing TREPAN,” in Proceedings of IIE Annual Conference, Orlando, Florida, 2006.

  30. T. Hailesilassie, “Extraction algorithm for deep neural networks: A review,” Int. J. Comput. Sci. Inform. Secur. 14, 376–381 (2016).

    Google Scholar 

  31. A. Averkin and S. Yarushev, “Hybrid neural networks and time series forecasting,” Commun. Comput. Inform. Sci. 934, 230–239 (2018).

    Article  Google Scholar 

  32. G. Pilato, S. A. Yarushev, and A. N. Averkin, “Prediction and detection of user emotions based on neuro-fuzzy neural networks in social networks,” in Proceedings of the 3rd International Scientific Conference on Intelligent Information Technologies for Industry IITI'18, Sochi, Russia, Adv. Intell. Syst. Comput. 875, 118–126 (2018).

  33. A. N. Averkin, G. Pilato, and S. A. Yarushev, “An approach for prediction of user emotions based on ANFIS in social networks,” in Proceedings of the 2nd International Scientific and Practical Conference on Fuzzy Technologies in the Industry FTI 2018–CEUR Workshop, Ostrava-Prague, Czech Republic, 2018, pp. 126–134.

  34. X.-H. Jin, “Neurofuzzy decision support system for efficient risk allocation in public-private partnership infrastructure projects,” J. Comput. Civ. Eng. 24, 525–538 (2010).

    Article  Google Scholar 

  35. X.-H. Jin, “Model for efficient risk allocation in privately financed public infrastructure projects using neuro-fuzzy techniques,” J. Constr. Eng. Manag., 1003–1014 (2011).

  36. V. V. Borisov, A. S. Fedulov, and M. M. Zernov, Fundamentals of Hybridization of Fuzzy Models, Vol. 9 of Fundamentals of Fuzzy Mathematics Series (Goryachaya Liniya-Telekom, Moscow, 2017) [in Russian].

  37. D. Rutkowska, M. Piliński, and L. Rutkowski, Neural Networks, Genetic Algorithms and Fuzzy Systems (Naukowe PWN, Warszawa, 2008) [in Polish].

    Google Scholar 

  38. S. Rajab and V. Sharma, “A review on the applications of neuro-fuzzy systems in business,” Artif. Intell. Rev. 49, 481–510 (2018).

    Article  Google Scholar 

  39. S. Mitra and Y. Hayashi, “Neuro-fuzzy rule generation: Survey in soft computing framework,” IEEE Trans. Neural Network 11, 748–768 (2000).

    Article  Google Scholar 

  40. J. Vieira, F. Morgado-Dias, and A. Mota, “Neuro-fuzzy systems: A survey,” WSEAS Trans. Syst. 3, 414–419 (2004).

    Google Scholar 

  41. J. Kim and N. Kasabov, “HyFIS: Adaptive neuro-fuzzy inference systems and their application to nonlinear dynamical systems,” Neural Network 12, 1301–1319 (1999).

    Article  Google Scholar 

  42. K. V. Shihabudheen and G. N. Pillai, “Recent advances in neuro-fuzzy system: A survey,” Knowl.-Based Syst. 152, 136–162 (2018).

    Article  Google Scholar 

  43. I. Z. Batyrshin, A. O. Nedosekin, A. A. Stetsko, V. B. Tarasov, A. V. Yazenin, and N. G. Yarushkina, Fuzzy Hybride Systems: Theory and Practice, Ed. by N. G. Yarushkina (Fizmatlit, Moscow, 2007) [in Russian].

    MATH  Google Scholar 

  44. Z. J. Viharos and K. B. Kis, “Survey on neuro-fuzzy systems and their applications in technical diagnostics and measurement,” Measurement 67, 126–136 (2015).

    Article  Google Scholar 

  45. C. T. Lin and C. S. G. Lee, “Neural network based fuzzy logic control and decision system,” IEEE Trans. Comput. 40, 1320–1336 (1991).

    Article  MathSciNet  Google Scholar 

  46. J.-S. R. Jang, “ANFIS: Adaptive-network-based fuzzy inference system,” IEEE Trans. Syst. Cybern. 23, 665–685 (1993).

    Article  Google Scholar 

  47. H. Naderpour and M. Mirrashid, “Shear failure capacity prediction of concrete beam-column joints in terms of ANFIS and GMDH,” Pract. Period. Struct. Des. Constr. 24 (2) (2019).

  48. L. Fan, “Revisit fuzzy neural network: Demystifying batch normalization and ReLU with generalized hamming network,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, CA, 2017, pp. 1920–1929.

  49. H. R. Bherenji and P. Khedkar, “Learning and tuning fuzzy logic controllers through reinforcements,” IEEE Trans. Neural Networks 3, 724–740 (1992).

    Article  Google Scholar 

  50. D. Nauck and R. Kruse, “Neuro-fuzzy systems for function approximation,” Fuzzy Sets Syst. 101, 261–271 (1999).

    Article  MathSciNet  Google Scholar 

  51. S. Tano, T. Oyama, and T. Arnould, “Deep combination of fuzzy inference and neural network in fuzzy inference,” Fuzzy Sets Syst. 82, 151–160 (1996).

    Article  Google Scholar 

  52. J. Ch. Feng and L. Ch. Teng, “An online self constructing neural fuzzy inference network and its applications,” IEEE Trans. Fuzzy Syst. 6, 12–32 (1998).

    Article  Google Scholar 

  53. N. Kasabov and Qun Song, “Dynamic evolving fuzzy neural networks with 'm-out-of-n' activation nodes for on-line adaptive systems,” Technical Report TR99/04 (Department of Inform. Sci., Univ. Otago, Otago, 1999).

  54. D. Gunning and D. Aha, “DARPA’S explainable artificial intelligence (XAI) program,” AI Magazine 40 (2), 44–58 (2019).

    Article  Google Scholar 

  55. A. N. Gorban’, “Errors of data-based intelligence,” in Proceedings of the International Conference on Intelligent Systems in Science and Technology, and the 6th All-Russian Scientific and Practical Conference on Artificial Intelligence in Solving Urgent Social and Economic Problems of the XXI Century, Perm, 2020, pp. 11–13.

  56. R. Hu, J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko, “Learning to reason: End-to-end module networks for visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, New York, 2017), pp. 804–813.

  57. J. Kim and J. Canny, “Interpretable learning for self-driving cars by visualizing causal attention,” in Proceedings of the International Conference on Computer Vision (IEEE, New York, 2017), pp. 2942–2950.

  58. L. A. Hendricks, T. Darrell, and Z. Akata, “Grounding visual explanations,” in Proceedings of the European Conference of Computer Vision (ECCV), Munich, Germany (Springer, 2018).

  59. K. Marazopoulou, M. Maier, and D. Jensen, “Learning the structure of causal models with relational and temporal dependence,” in Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, Association for Uncertainty in Artificial Intelligence, Amsterdam, Netherlands, 2015, pp. 572–581.

  60. A. Pfeffer, Practical Probabilistic Programming (Manning, Greenwich, CT, 2016).

    Google Scholar 

  61. M. Harradon, J. Druce, and B. Ruttenberg, “Causal learning and explanation of deep neural networks via autoencoded activations,” arXiv: 1802.00541v1 [cs.AI] (2018).

  62. L. She and J. Y. Chai, “Interactive learning for acquisition of grounded verb semantics towards human-robot communication,” in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017, Vol. 1, pp. 1634–1644.

  63. Z. Qi and F. Li, “Learning explainable embeddings for deep networks,” in Proceedings of the NIPS Workshop on Interpreting, Explaining and Visualizing Deep Learning, Long Beach, 2017.

  64. J. Dodge, S. Penney, C. Hilderbrand, A. Anderson, and M. Burnett, “How the experts do it: Assessing and explaining agent behaviors in real-time strategy games,” in Proceedings of the CHI Conference on Human Factors in Computing Systems (Assoc. for Comput. Machinery, New York, 2018), pp. 1–12.

  65. F. Belbute-Peres and J. Z. Kolter, “A modular differentiable rigid body physics engine,” in Neural Information Processing Systems, Deep Reinforcement Learning Symposium, Long Beach, CA, 2017.

  66. A. Hefny, Z. Marinho, W. Sun, S. Srinivasa, and G. Gordon, “Recurrent predictive state policy networks,” in Proceedings of the 35th International Conference on Machine Learning (Int. Machine Learning Soc., Stockholm, Sweden, 2018), pp. 1949–1958.

  67. P. Vicol, M. Tapaswi, L. Castrejon, and S. Fidler, “MovieGraphs: Towards understanding human-centric situations from videos,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (IEEE, New York, 2018), pp. 4631–4640.

  68. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, “Object detectors emerge in deep scene CNNs,” in Proceedings of the International Conference on Learning Representations, San Diego, CA, 2015.

  69. V. Gogate and P. Domingos, “Probabilistic theorem proving,” Commun. ACM 59 (7), 107–15 (2016).

    Article  Google Scholar 

  70. M. Du, N. Liu, Q. Song, and X. Hu, “Towards Explanation of DNN-based prediction and guided feature inversion,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Assoc. Comput. Machinery, New York, 2018), pp. 1358–1367.

  71. J. Gao, N. Liu, M. Lawley, and X. Hu, “An interpretable classification framework for information extraction from online healthcare forums,” J. Healthcare Eng. 2017, 2460174 (2017).

    Google Scholar 

  72. S. C.-H. Yang and P. Shafto, “Explainable artificial intelligence via Bayesian teaching,” in Proceedings of the 31st Conference on Neural Information Processing Systems Workshop on Teaching Machines, Robots and Humans, Long Beach, CA, 2017.

Download references

Funding

This study was supported by the Russian Foundation for Basic Research (grant no. 20-17-50199) under the Expansion Program.

Author information

Authors and Affiliations

  1. Federal Research Center “Computer Science and Control,” Russian Academy of Sciences, Moscow, Russia

    A. N. Averkin

  2. Plekhanov Russian University of Economics, Moscow, Russia

    A. N. Averkin & S. A. Yarushev

Authors
  1. A. N. Averkin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. A. Yarushev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to A. N. Averkin or S. A. Yarushev.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Averkin, A.N., Yarushev, S.A. Review of Research in the Field of Developing Methods to Extract Rules From Artificial Neural Networks. J. Comput. Syst. Sci. Int. 60, 966–980 (2021). https://doi.org/10.1134/S1064230721060046

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1064230721060046

Search

Navigation