Abstract
Reinforcement-Learning-based solutions have achieved many successes in numerous complex tasks. However, their training process may be unstable, and achieving convergence can be difficult, expensive, and in some instances impossible. We propose herein an approach that enables the integration of strong formal verification methods in order to improve the learning process as well as prove convergence. During the learning process, formal methods serve as experts to identify weaknesses in the learned model, improve it, and even lead it to converge. By evaluating our approach on several common problems, which have already been studied and solved by classical methods, we demonstrate the strength and potential of our core idea of incorporating formal methods into the training process of Reinforcement Learning methods.
Link to our code: https://github.com/eliyabron/Formal_verification_with_RL.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction. MIT press (2018)
Kiran, B.R., et al.: Deep reinforcement learning for autonomous driving: a survey. IEEE Trans. Intell. Transp. Sys.23, 4909–4926 (2021)
Xiong, C., Zhong, V., Socher, R.: DCN+: mixed objective and deep residual coattention for question answering. arXiv preprint arXiv:1711.00106 (2017)
Weile, D.S., Michielssen, E.: Genetic algorithm optimization applied to electromagnetics: a review. IEEE Trans. Antennas Propag. 45(3), 343–353 (1997)
Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. ACM Comput. Surv. (CSUR) 50(2), 1–35 (2017)
Pomerleau, D.A.: Alvinn: an autonomous land vehicle in a neural network. In: Advances in Neural Information Processing Systems 1 (1988)
Abbeel, P., Ng, A.Y.: Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the Twenty-First International Conference on Machine Learning, p. 1 (2004)
Menzies, T., Pecheur, C.: Verification and validation and artificial intelligence 65, 153–201 (2005)
Seshia, S.A., Sadigh, D., Sastry, S.S.: Towards verified artificial intelligence. ArXiv e-prints (2016)
García, J., Fernández, F.: A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res. 16(1), 1437–1480 (2015)
Fulton, N., Platzer, A.: Safe reinforcement learning via formal methods: toward safe control through proof and learning. In: AAAI (2018)
Kazak, Y., Barrett, C., Katz, G., Schapira, M.: Verifying deep-rl-driven systems. In: Proceedings of the 2019 Workshop on Network Meets AI & ML, NetAI’19, pp. 83–89 New York, NY, USA (2019). Association for Computing Machinery
Yang, Z., et al.: An iterative scheme of safe reinforcement learning for nonlinear systems via barrier certificate generation. In: Silva, A., Leino, K.R.M. (eds.) CAV 2021. LNCS, vol. 12759, pp. 467–490. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-81685-8_22
Jin, P., Zhang, M., Li, J., Han, L., Wen, X.: Learning on abstract domains: a new approach for verifiable guarantee in reinforcement learning. CoRR. arXiv:2106.06931 (2021)
Zhu, H., Magill, S.: Systems support for hardware anti-rop. Technical report, Galois Inc (2017). https://galois.com/reports/formal-methods-for-reinforcement-learning/
Watkins, C.J.C.H., Dayan, P.: Q-learning. In: Machine Learning, vol. 8, pp. 279–292 (1992). https://doi.org/10.1007/BF00992698
Aigner, M., Fromme, M.: A game of cops and robbers. Discret. Appl. Math. 8(1), 1–12 (1984)
Parsons, T.D.: Pursuit-evasion in a graph. In: Theory and Applications of Graphs, pp. 426–441. Springer (1978). https://doi.org/10.1007/BFb0070400
Acknowledgments
This work is supported by the Horizon 2020 research and innovation programme for the Bio4Comp project under grant agreement number 732482 and by the ISRAEL SCIENCE FOUNDATION (Grant No. 190/19). We would like to thank Assaf Grundman and Shlomi Mamman for their work and feedback on an early version of this project, and the Data Science Institute at Bar-Ilan University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Raviv, A., Bronshtein, E., Reginiano, O., Aluf-Medina, M., Kugler, H. (2023). Learning Through Imitation by Using Formal Verification. In: Gąsieniec, L. (eds) SOFSEM 2023: Theory and Practice of Computer Science. SOFSEM 2023. Lecture Notes in Computer Science, vol 13878. Springer, Cham. https://doi.org/10.1007/978-3-031-23101-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-031-23101-8_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23100-1
Online ISBN: 978-3-031-23101-8
eBook Packages: Computer ScienceComputer Science (R0)