Skip to main content
SpringerLink
Log in

Performance Fuzzing with Reinforcement-Learning and Well-Defined Constraints for the B Method

  • Conference paper
  • First Online:
Integrated Formal Methods (iFM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14300))

Included in the following conference series:

  • 154 Accesses

Abstract

The B method is a formal method supported by a variety of tools. Those tools, like any complex piece of software, may suffer from performance issues and vulnerabilities, especially for potentially undiscovered, pathological cases. To find such cases and assess their performance impacts within a single tool, we leverage the performance fuzzing algorithm BanditFuzz for the constraint solving backends of the ProB model checker. BanditFuzz utilises two multi-armed bandits to generate and mutate benchmark inputs for the ProB backends in a targeted manner. We describe how we adapted BanditFuzz for the B method, which differences exist to the original implementation for the SMT-LIB standard, and how we ensure well-definedness of the randomly generated benchmarks. Our experiments successfully uncovered performance issues in specific backends and even external tooling, providing valuable insights into areas which required improvement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Reported in https://github.com/Z3Prover/z3/issues/6734.

References

  1. Abrial, J.R.: The B-book: assigning programs to meanings. Cambridge Univ. Press (1996). https://doi.org/10.1017/CBO9780511624162

    Article  Google Scholar 

  2. Abrial, J.R.: Modeling in event-B: system and software engineering. Cambridge Univ. Press (2010). https://doi.org/10.1017/CBO9781139195881

    Article  Google Scholar 

  3. Abrial, J.-R., Mussat, L.: On using conditional definitions in formal theories. In: Bert, D., Bowen, J.P., Henson, M.C., Robinson, K. (eds.) ZB 2002. LNCS, vol. 2272, pp. 242–269. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45648-1_13

    Chapter  Google Scholar 

  4. Agrawal, S., Goyal, N.: Analysis of Thompson sampling for the multi-armed bandit problem. In: Proceedings of the 25th Annual Conference on Learning Theory. Proceedings of Machine Learning Research, vol. 23, pp. 39.1-39.26. PMLR (2012)

    Google Scholar 

  5. Back, R.J.R.: On correct refinement of programs. J. Comput. Syst. Sci. 23(1), 49–68 (1981). https://doi.org/10.1016/0022-0000(81)90005-2

    Article  MathSciNet  Google Scholar 

  6. Back, R.J., Wright, J.: Refinement calculus: a systematic introduction. Texts in Computer Science, Springer (2012). https://doi.org/10.1007/978-1-4612-1674-2

  7. Barbosa, H., et al.: cvc5: a versatile and industrial-strength SMT solver. In: Fisman, D., Rosu, G. (eds.) Tools and Algorithms for the Construction and Analysis of Systems: 28th International Conference, TACAS 2022, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2022, Munich, Germany, April 2–7, 2022, Proceedings, Part I, pp. 415–442. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-99524-9_24

  8. Barrett, C., Stump, A., Tinelli, C.: The SMT-LIB standard: Version 2.0. In: Proceedings of the 8th International Workshop on Satisfiability Modulo Theories (Edinburgh, UK) (2010)

    Google Scholar 

  9. Butler, M., et al.: The first twenty-five years of industrial use of the B-method. In: ter Beek, M.H., Ničković, D. (eds.) FMICS 2020. LNCS, vol. 12327, pp. 189–209. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58298-2_8

    Chapter  Google Scholar 

  10. Carlsson, M., Mildner, P.: SICStus prolog-the first 25 years. Theory Pract. Logic Program. 12(1–2), 35–66 (2012). https://doi.org/10.1017/S1471068411000482

    Article  MathSciNet  Google Scholar 

  11. Carlsson, M., Ottosson, G., Carlson, B.: An open-ended finite domain constraint solver. In: Glaser, H., Hartel, P., Kuchen, H. (eds.) PLILP 1997. LNCS, vol. 1292, pp. 191–206. Springer, Heidelberg (1997). https://doi.org/10.1007/BFb0033845

    Chapter  Google Scholar 

  12. Carlsson, M., Widen, J., Andersson, J., Andersson, S., Boortz, K., Nilsson, H., Sjöland, T.: SICStus Prolog user’s manual, vol. 3. Swedish Institute of Computer Science, Kista, Sweden (1988)

    Google Scholar 

  13. Chapelle, O., Li, L.: An empirical evaluation of Thompson sampling. Adv. Neural. Inf. Process. Syst. 24, 2249–2257 (2011)

    Google Scholar 

  14. Chen, Y., Bradbury, M., Suri, N.: Towards effective performance fuzzing. In: 2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), pp. 128–129 (2022). https://doi.org/10.1109/ISSREW55968.2022.00055

  15. de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  16. Dunkelau, J., Schmidt, J., Leuschel, M.: Analysing ProB’s constraint solving backends. In: Raschke, A., Méry, D., Houdek, F. (eds.) ABZ 2020. LNCS, vol. 12071, pp. 107–123. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-48077-6_8

    Chapter  Google Scholar 

  17. Dutertre, B.: Yices 2.2. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 737–744. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08867-9_49

    Chapter  Google Scholar 

  18. Koo, J., Saumya, C., Kulkarni, M., Bagchi, S.: PYSE: automatic worst-case test generation by reinforcement learning. In: 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), pp. 136–147 (2019). https://doi.org/10.1109/ICST.2019.00023

  19. Krings, S., Leuschel, M.: SMT solvers for validation of B and event-B models. In: Ábrahám, E., Huisman, M. (eds.) IFM 2016. LNCS, vol. 9681, pp. 361–375. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33693-0_23

    Chapter  Google Scholar 

  20. Le, X.B.D., Pasareanu, C., Padhye, R., Lo, D., Visser, W., Sen, K.: Saffron: adaptive grammar-based fuzzing for worst-case analysis. SIGSOFT Softw. Eng. Notes 44(4), 14 (2019). https://doi.org/10.1145/3364452.3364455

    Article  Google Scholar 

  21. Lemieux, C., Padhye, R., Sen, K., Song, D.: Perffuzz: automatically generating pathological inputs. In: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 254–265 (2018). https://doi.org/10.1145/3213846.3213861

  22. Leuschel, M.: Fast and effective well-definedness checking. In: Dongol, B., Troubitsyna, E. (eds.) IFM 2020. LNCS, vol. 12546, pp. 63–81. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63461-2_4

    Chapter  Google Scholar 

  23. Leuschel, M., Bendisposto, J., Dobrikov, I., Krings, S., Plagge, D.: From animation to data validation: the ProB constraint solver 10 years on. In: Formal Methods Applied to Complex Systems: Implementation of the B Method, chap. 14, pp. 427–446. Wiley ISTE (2014). https://doi.org/10.1002/9781119002727.ch14

  24. Leuschel, M., Butler, M.: ProB: a model checker for B. In: Araki, K., Gnesi, S., Mandrioli, D. (eds.) FME 2003. LNCS, vol. 2805, pp. 855–874. Springer, Heidelberg (2003). https://doi.org/10.1007/978-3-540-45236-2_46

    Chapter  Google Scholar 

  25. Leuschel, M., Butler, M.: ProB: an automated analysis toolset for the B method. Int. J. Softw. Tools Technol. Transfer 10(2), 185–203 (2008). https://doi.org/10.1007/s10009-007-0063-9

    Article  Google Scholar 

  26. Liang, H., Pei, X., Jia, X., Shen, W., Zhang, J.: Fuzzing: state of the art. IEEE Trans. Reliab. 67(3), 1199–1218 (2018). https://doi.org/10.1109/TR.2018.2834476

    Article  Google Scholar 

  27. Manès, V.J., Han, H., Han, C., Cha, S.K., Egele, M., Schwartz, E.J., Woo, M.: The art, science, and engineering of fuzzing: a survey. IEEE Trans. Software Eng. 47(11), 2312–2331 (2021). https://doi.org/10.1109/TSE.2019.2946563

    Article  Google Scholar 

  28. Marques-Silva, J., Lynce, I., Malik, S.: Conflict-driven clause learning SAT solvers. In: Handbook of Satisfiability, Frontiers in Artificial Intelligence and Applications, vol. 185, pp. 131–153. IOS press (2009). https://doi.org/10.3233/978-1-58603-929-5-131

  29. Miller, B.P., Fredriksen, L., So, B.: An empirical study of the reliability of UNIX utilities. Commun. ACM 33(12), 32–44 (1990). https://doi.org/10.1145/96267.96279

    Article  Google Scholar 

  30. Niemetz, A., Preiner, M.: Bitwuzla at the SMT-COMP 2020. CoRR abs/2006.01621 (2020)

    Google Scholar 

  31. Petsios, T., Zhao, J., Keromytis, A.D., Jana, S.: Slowfuzz: automated domain-independent detection of algorithmic complexity vulnerabilities. In: Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 2155–2168 (2017). https://doi.org/10.1145/3133956.3134073

  32. Plagge, D., Leuschel, M.: Validating B,Z and TLA+ using ProB and Kodkod. In: Giannakopoulou, D., Méry, D. (eds.) FM 2012. LNCS, vol. 7436, pp. 372–386. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32759-9_31

    Chapter  Google Scholar 

  33. Robbins, H.: Some aspects of the sequential design of experiments. Bull. Am. Math. Soc. 55, 527–535 (1952)

    Article  MathSciNet  Google Scholar 

  34. Schmidt, J., Krings, S., Leuschel, M.: Repair and generation of formal models using synthesis. In: Furia, C.A., Winter, K. (eds.) IFM 2018. LNCS, vol. 11023, pp. 346–366. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98938-9_20

    Chapter  Google Scholar 

  35. Schmidt, J., Leuschel, M.: Improving SMT solver integrations for the validation of B and Event-B Models. In: Lluch Lafuente, A., Mavridou, A. (eds.) FMICS 2021. LNCS, vol. 12863, pp. 107–125. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85248-1_7

    Chapter  Google Scholar 

  36. Schmidt, J., Leuschel, M.: SMT solving for the validation of B and Event-B models. Int. J. Softw. Tools Technol. Transfer 24, 1043–1077 (2022). https://doi.org/10.1007/s10009-022-00682-y

    Article  Google Scholar 

  37. Scott, J., Mora, F., Ganesh, V.: BanditFuzz: a reinforcement-learning based performance fuzzer for SMT solvers. In: Christakis, M., Polikarpova, N., Duggirala, P.S., Schrammel, P. (eds.) NSV/VSTTE -2020. LNCS, vol. 12549, pp. 68–86. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63618-0_5

    Chapter  Google Scholar 

  38. Scott, J., Sudula, T., Rehman, H., Mora, F., Ganesh, V.: BanditFuzz: fuzzing SMT solvers with multi-agent reinforcement learning. In: Huisman, M., Păsăreanu, C., Zhan, N. (eds.) FM 2021. LNCS, vol. 13047, pp. 103–121. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-90870-6_6

    Chapter  Google Scholar 

  39. Thompson, W.R.: On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25(3–4), 285–294 (1933). https://doi.org/10.1093/biomet/25.3-4.285

    Article  Google Scholar 

  40. Watkins, C.J., Dayan, P.: Q-learning. Machine Learn. 8, 279–292 (1992). https://doi.org/10.1007/BF00992698

  41. Wen, C., et al.: Memlock: memory usage guided fuzzing. In: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 765–777. ICSE ’20, Association for Computing Machinery (2020). https://doi.org/10.1145/3377811.3380396

  42. Zhang, Y., et al.: Demystifying performance regressions in string solvers. IEEE Trans. Software Eng. 49(3), 947–961 (2023). https://doi.org/10.1109/TSE.2022.3168373

    Article  Google Scholar 

Download references

Acknowledgements

We want to thank our colleague Joshua Schmidt for his input and ideas regarding more targeted fuzz generation for the ProB CDCL(T) and Z3 backends. Computational support and infrastructure was provided by the “Centre for Information and Media Technology” (ZIM) at the University of Düsseldorf (Germany).

Author information

Authors and Affiliations

  1. Institut Für Informatik, Heinrich-Heine-Universität Düsseldorf, Universitätsstraße 1, 40225, Düsseldorf, Germany

    Jannik Dunkelau & Michael Leuschel

Authors
  1. Jannik Dunkelau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Leuschel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jannik Dunkelau .

Editor information

Editors and Affiliations

  1. Embedded Systems Group, University of Münster, Münster, Germany

    Paula Herber

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Anton Wijs

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dunkelau, J., Leuschel, M. (2024). Performance Fuzzing with Reinforcement-Learning and Well-Defined Constraints for the B Method. In: Herber, P., Wijs, A. (eds) Integrated Formal Methods. iFM 2023. Lecture Notes in Computer Science, vol 14300. Springer, Cham. https://doi.org/10.1007/978-3-031-47705-8_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47705-8_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47704-1

  • Online ISBN: 978-3-031-47705-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation