Skip to main content
SpringerLink
Log in

Using deep reinforcement learning to search reachability properties in systems specified through graph transformation

Soft Computing Aims and scope Submit manuscript

Cite this article

Abstract

Today, model checking is one of the essential techniques in the verification of software systems. This technique can verify some properties such as reachability in which the entire state space is searched to find the desired state. However, model checking may lead to the state space explosion problem in which all states cannot be generated due to the exponential resource usage. Although the results of recent model checking approaches are promising, there is still room for improvement in terms of accuracy and the number of explored states. In this paper, using deep reinforcement learning and two neural networks, we propose an approach to increase the accuracy of the generated witnesses and reduce the use of hardware resources. In this approach, at first, an agent starts to explore the state space without any knowledge and gradually identifies the proper and improper actions by receiving different rewards/penalties from the environment to achieve the goal. Once the dataset is fulfilled with the agent's experiences, two neural networks evaluate the quality of each operation in each state, and afterwards, the best action is selected. The significant difficulties and challenges in the implementation are encoding the states, feature engineering, feature selection, reward engineering, handling invalid actions, and configuring the neural network. Finally, the proposed approach has been implemented in the Groove toolset, and as a result, in most of the case studies, it overcame the problem of state space explosion. Also, this approach outperforms the existing solutions in terms of generating shorter witnesses and exploring fewer states. On average, the proposed approach is nearly 400% better than other approaches in exploring fewer states and 300% better than the others in generating shorter witnesses. Also, on average, the proposed approach is 37% more accurate than the others in terms of finding the goals state.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34

Similar content being viewed by others

Data availability

Enquiries about data availability should be directed to the authors.

References

  • Amiri M, Amnieh HB, Hasanipanah M, Khanli LM (2016) A new combination of artificial neural network and K-nearest neighbors models to predict blast-induced ground vibration and air-overpressure. Eng Comput 32(4):631–644. https://doi.org/10.1007/s00366-016-0442-5

    Article  Google Scholar 

  • Baier C, Katoen J-P (2008) Principles of model checking. MIT Press, London

    MATH  Google Scholar 

  • Bergmann G, et al., (2015) Viatra 3: A reactive model transformation platform. In: international conference on theory and practice of model transformations, 2015: Springer, pp. 101–110. [Online]. Available: https://link.springer.com/chapter/https://doi.org/10.1007/978-3-319-21155-8_8

  • Bertsekas DP (2018) Feature-based aggregation and deep reinforcement learning: a survey and some new implementations. IEEE/CAA J Automatica Sinica 6(1):1–31

    Article  MathSciNet  Google Scholar 

  • Corradini A, F. L. Dotti, L. Foss, and L. Ribeiro, (2004) Translating Java code to graph transformation systems. In: international conference on graph transformation, Springer, pp. 383–398. [Online]. Available: https://link.springer.com/chapter/https://doi.org/10.1007/978-3-540-30203-2_27

  • Demri S, Laroussinie F, Schnoebelen P (2006) A parametric analysis of the state-explosion problem in model checking. Comput Syst Sci 72(4):547–575

    Article  MathSciNet  Google Scholar 

  • Duboue P (2020) The art of feature engineering: essentials for machine learning. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Fulton N, and A. Platzer, (2018) Safe reinforcement learning via formal methods: Toward safe control through proof and learning. In: thirty-second AAAI conference on artificial intelligence, 2018. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/download/17376/16225

  • Hasselt van H, A. Guez, and D. Silver, (2016) Deep reinforcement learning with double q-learning. In: Thirtieth AAAI conference on artificial intelligence, 2016. [Online]. Available: https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/download/12389/11847

  • Hosu I.A., and T. Rebedea, "Playing atari games with deep reinforcement learning and human checkpoint replay. arXiv.org e-Print archive, 2016. [Online]. Available: https://arxiv.org/abs/1607.05077

  • Jansen N, B. Könighofer, S. Junges, and R. Bloem, (2018) Shielded decision-making in MDPs. arXiv.org e-Print archive, vol. abs/1807.06096, 2018. [Online]. Available: https://www.researchgate.net/profile/Nils_Jansen/publication/326459531_Shielded_Decision-Making_in_MDPs/links/5be0224e4585150b2b9faeed/Shielded-Decision-Making-in-MDPs.pdf?origin=publication_detail.

  • Lara de J and H. Vangheluwe, (2002) AToM 3: a Tool for Multi-formalism and Meta-modelling. In: international conference on fundamental approaches to software engineering, Springer, pp. 174–188. [Online]. Available: https://link.springer.com/chapter/https://doi.org/10.1007/3-540-45923-5_12

  • Liu T, Tian B, Ai Y, Li L, Cao D, Wang F-Y (2018) Parallel reinforcement learning: a framework and case study. IEEE/CAA J Automatica Sinica 5(4):827–835

    Article  MathSciNet  Google Scholar 

  • Maeoka J, Y. Tanabe, and F. Ishikawa, (2015) Depth-first heuristic search for software model checking. In: Computer and Information Science, Springer, 2016, pp. 75-96

  • Partabian J, Rafe V, Parvin H, Nejatian S (2020) An approach based on knowledge exploration for state space management in checking reachability of complex software systems. Soft Comput 24(10):7181–7196

    Article  Google Scholar 

  • Pira E (2020) A novel approach to solve AI planning problems in graph transformations. Eng Appl Artif Intell 92:103684

    Article  Google Scholar 

  • Pira E, Rafe V, Nikanjam A (2016) EMCDM: efficient model checking by data mining for verification of complex software systems specified through architectural styles. Appl Soft Comput 49:1185–1201

    Article  Google Scholar 

  • Pira E, Rafe V, Nikanjam A (2017) Deadlock detection in complex software systems specified through graph transformation using Bayesian optimization algorithm. Syst Softw 131:181–200

    Article  Google Scholar 

  • Pira E, Rafe V, Nikanjam A (2018) Searching for violation of safety and liveness properties using knowledge discovery in complex systems specified through graph transformations. Inf Softw Technol 97:110–134

    Article  Google Scholar 

  • Pira E, Rafe V, Nikanjam A (2019) Using evolutionary algorithms for reachability analysis of complex software systems specified through graph transformation. Reliabil Eng Syst Saf 191:106577

    Article  Google Scholar 

  • Pira E, "Using knowledge discovery to propose a two-phase model checking for safety analysis of graph transformations," Software Quality Journal, pp. 1–28, 2021.

  • Rafe V, Moradi M, Yousefian R, Nikanjam A (2015) A meta-heuristic solution for automated refutation of complex software systems specified through graph transformations. Appl Soft Comput 33:136–149

    Article  Google Scholar 

  • Rafe V, Darghayedi M, Pira E (2019) MS-ACO: a multi-stage ant colony optimization to refute complex software systems specified through graph transformation. Soft Comput 23(12):4531–4556

    Article  Google Scholar 

  • Rensink A, (2003) The GROOVE simulator: a tool for state space generation. In: international workshop on applications of graph transformations with industrial relevance, Springer, pp. 479–485. [Online]. Available: https://link.springer.com/chapter/https://doi.org/10.1007/978-3-540-25959-6_40

  • Rozenberg G (1997) Handbook of graph grammars and computing. World Scientific, Singapore

    MATH  Google Scholar 

  • Snippe E, (2011) Using heuristic search to solve planning problems in GROOVE. In: 14th Twente Student Conference on IT, University of Twente, [Online]. Available: https://www.researchgate.net/publication/228977418_Using_Heuristic_Search_to_Solve_Planning_Problems_in_GROOVE

  • Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press, London

    MATH  Google Scholar 

  • Taentzer G, (2003) AGG: A graph transformation environment for modeling and validation of software. In: international workshop on applications of graph transformations with industrial relevance, Springer, pp. 446–453. [Online]. Available: https://link.springer.com/chapter/https://doi.org/10.1007/978-3-540-25959-6_35

  • Wilcoxon F (1992) Individual comparisons by ranking methods. In: Jhonson NL, Kotz S (eds) Breakthroughs in statistics. Springer, New York, pp 196–202

    Chapter  Google Scholar 

  • Yasrebi M, Rafe V, Nejatian S (2020) An efficient approach to state space management in model checking of complex software systems using machine learning techniques. J Intell Fuzzy Syst 38(2):1761–1773

    Article  Google Scholar 

  • Yousefian R, Rafe V, Rahmani M (2014) A heuristic solution for model checking graph transformation systems. Appl Soft Comput 24:169–180

    Article  Google Scholar 

  • Yousefian R, Aboutorabi S, Rafe V (2016) A greedy algorithm versus metaheuristic solutions to deadlock detection in Graph Transformation Systems. Intell Fuzzy Syst 31(1):137–149

    Article  Google Scholar 

Download references

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Engineering, Arak University, 38156-8-8349, Arak, Iran

    Mohammad Javad Mehrabi & Vahid Rafe

Authors
  1. Mohammad Javad Mehrabi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vahid Rafe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vahid Rafe.

Ethics declarations

Conflict of interest

The authors have not disclosed any competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mehrabi, M.J., Rafe, V. Using deep reinforcement learning to search reachability properties in systems specified through graph transformation. Soft Comput 26, 9635–9663 (2022). https://doi.org/10.1007/s00500-022-06815-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-022-06815-4

Keywords

search

Navigation