Skip to main content

Configurable Benchmarks for C Model Checkers

  • Conference paper
  • First Online:
NASA Formal Methods (NFM 2022)


Software model checkers employ many different techniques. During various competitions, the capabilities of these verification tools are compared on a wide variety of benchmarks. Our aim is to get insight into which code characteristics are “hard” for software model checkers. To that end, we present a software tool that automatically generates C benchmark programs that are intended as stress tests for software model checkers. The parameters of the generated C programs, e.g., program size, types of operation, are controllable, and programs can be tweaked, e.g., floats can be replaced by integers and pointer dereferencing can be used for variable accesses. Our tool enables a systematic comparison of software verifiers. We illustrate its usage by evaluating the top verifiers from the SV-COMP 2022 reachability category and analyze what makes benchmarks hard for these tools and how well these tools scale, both in terms of code related to the property at hand as well as in terms of code that is unrelated to it.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions


  1. 1.

  2. 2.

  3. 3.

  4. 4.

  5. 5.


  1. Berger, P., Katoen, J.-P., Ábrahám, E., Waez, M.T.B., Rambow, T.: Verifying auto-generated C code from simulink. In: Havelund, K., Peleska, J., Roscoe, B., de Vink, E. (eds.) FM 2018. LNCS, vol. 10951, pp. 312–328. Springer, Cham (2018).

    Chapter  Google Scholar 

  2. Beyer, D.: Software verification: 10th comparative evaluation (SV-COMP 2021). In: TACAS 2021. LNCS, vol. 12652, pp. 401–422. Springer, Cham (2021).

    Chapter  Google Scholar 

  3. Beyer, D., Keremoglu, M.E.: CPAchecker: a tool for configurable software verification. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 184–190. Springer, Heidelberg (2011).

    Chapter  Google Scholar 

  4. Beyer, D., Lemberger, T.: Software verification: testing vs. model checking. In: HVC 2017. LNCS, vol. 10629, pp. 99–114. Springer, Cham (2017).

    Chapter  Google Scholar 

  5. Beyer, D., Löwe, S., Wendler, P.: Reliable benchmarking: requirements and solutions. Int. J. Softw. Tools Technol. Transf. 21(1), 1–29 (2019)

    Article  Google Scholar 

  6. Chalupa, M., Novák, J., Strejcek, J.: Symbiotic 8: parallel and targeted test generation - (competition contribution). In: FASE. LNCS, vol. 12649, pp. 368–372. Springer (2021)

    Google Scholar 

  7. Chalupa, M., Strejček, J.: Evaluation of program slicing in software verification. In: Ahrendt, W., Tapia Tarifa, S.L. (eds.) IFM 2019. LNCS, vol. 11918, pp. 101–119. Springer, Cham (2019).

    Chapter  Google Scholar 

  8. Cordeiro, L.C., Kroening, D., Schrammel, P.: Benchmarking of java verification tools at the software verification competition (SV-COMP). ACM SIGSOFT Softw. Eng. Notes 43(4), 56 (2018)

    Article  Google Scholar 

  9. Gadelha, M.Y.R., Menezes, R., Monteiro, F.R., Cordeiro, L.C., Nicole, D.A.: ESBMC: scalable and precise test generation based on the floating-point theory - (competition contribution). In: FASE. LNCS, vol. 12076, pp. 525–529. Springer (2020)

    Google Scholar 

  10. Groce, A., Havelund, K., Holzmann, G., Joshi, R., Xu, R.-G.: Establishing flight software reliability: testing, model checking, constraint-solving, monitoring and learning. Ann. Math. Artif. Intell. 70(4), 315–349 (2014).

    Article  MathSciNet  MATH  Google Scholar 

  11. Heizmann, M., et al.: Ultimate automizer and the search for perfect interpolants. In: Beyer, D., Huisman, M. (eds.) TACAS 2018. LNCS, vol. 10806, pp. 447–451. Springer, Cham (2018).

    Chapter  Google Scholar 

  12. Howar, F., Jasper, M., Mues, M., Schmidt, D., Steffen, B.: The RERS challenge: towards controllable and scalable benchmark synthesis. Int. J. Softw. Tools Technol. Transf. 23(6), 917–930 (2021).

    Article  Google Scholar 

  13. Jasper, M.: Synthesizing realistic verification tasks. Ph.D. thesis, Technical University of Dortmund, Germany (2021)

    Google Scholar 

  14. Jhala, R., Majumdar, R.: Software model checking. ACM Comput. Surv. 41(4), 21:1–21:54 (2009)

    Google Scholar 

  15. Klinger, C., Christakis, M., Wüstholz, V.: Differentially testing soundness and precision of program analyzers. In: ISSTA, pp. 239–250. ACM (2019)

    Google Scholar 

  16. Malík, V., Schrammel, P., Vojnar, T.: 2LS: heap analysis and memory safety. In: TACAS 2020. LNCS, vol. 12079, pp. 368–372. Springer, Cham (2020).

    Chapter  Google Scholar 

  17. Méry, D., Singh, N.K.: Automatic code generation from event-b models. In: SoICT, pp. 179–188. ACM (2011)

    Google Scholar 

  18. Richter, C., Wehrheim, H.: PeSCo: predicting sequential combinations of verifiers. In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) TACAS 2019. LNCS, vol. 11429, pp. 229–233. Springer, Cham (2019).

    Chapter  Google Scholar 

  19. Sharir, M., Pnueli, A., et al.: Two Approaches to Interprocedural Data Flow Analysis. New York University, Courant Institute of Mathematical Sciences (1978)

    Google Scholar 

  20. Vorobyov, K., Krishnan, P.: Comparing model checking and static program analysis: a case study in error detection approaches. In: Proceedings of SSV (2010)

    Google Scholar 

  21. Westhofen, L., Berger, P., Katoen, J.-P.: Benchmarking software model checkers on automotive code. In: Lee, R., Jha, S., Mavridou, A., Giannakopoulou, D. (eds.) NFM 2020. LNCS, vol. 12229, pp. 133–150. Springer, Cham (2020).

    Chapter  Google Scholar 

  22. Zhang, C., Su, T., Yan, Y., Zhang, F., Pu, G., Su, Z.: Finding and understanding bugs in software model checkers. In: ESEC/SIGSOFT FSE, pp. 763–773. ACM (2019)

    Google Scholar 

Download references


We thank Fabian Hippler and Felix Faber for their continuing support and work.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Philipp Berger .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fink, X., Berger, P., Katoen, JP. (2022). Configurable Benchmarks for C Model Checkers. In: Deshmukh, J.V., Havelund, K., Perez, I. (eds) NASA Formal Methods. NFM 2022. Lecture Notes in Computer Science, vol 13260. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06772-3

  • Online ISBN: 978-3-031-06773-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics