Synthesizing Subtle Bugs with Known Witnesses

  • Marc JasperEmail author
  • Bernhard Steffen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11245)


This paper presents a new technique for the generation of verification benchmarks that are automatically guaranteed to be hard, or as we say, to contain subtle bugs/property violations: (i) Identifying a bug requires to match many computation steps and (ii) corresponding counterexamples are sparse among all feasible executions. Key idea is to iteratively synthesize Büchi automata for variations of a set of LTL properties and to combine these automata in a fashion that each property can be individually controlled in the resulting model: Based on our notion of a counterexample handle, it is possible to switch the satisfaction of a given property on and off without affecting that of the other considered properties. This orthogonality of our treatment of counterexamples is vital for the subsequent parts of the benchmark generation process. Together with the mentioned hardness, it helps to overcome the undesired clustering of counterexamples observed during previous iterations of the RERS Challenge. Even more importantly, these handles and associated counterexamples are sufficient to automatically generate the modal contracts required for the parallel decomposition process that allows us to generate parallel verification benchmarks of arbitrary size, for example in form of a Petri net or in Promela.


Benchmark generation Program verification Model checking Error witnesses Temporal logic LTL synthesis Büchi automata Modal transition systems Modal contracts 


  1. 1.
    Baier, C., Katoen, J.P., Larsen, K.G.: Principles of Model Checking. MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  2. 2.
    Bartocci, E., et al.: First international competition on runtime verification: rules, benchmarks, tools, and final results of CRV 2014. STTT 1–40 (2017).
  3. 3.
    Büchi, J.R.: Symposium on decision problems: on a decision method in restricted second order arithmetic. In: Logic, Methodology and Philosophy of Science, Studies in Logic and the Foundations of Mathematics, vol. 44, pp. 1–11. Elsevier (1966)Google Scholar
  4. 4.
    Beyer, D.: Competition on software verification. In: Flanagan, C., König, B. (eds.) TACAS 2012. LNCS, vol. 7214, pp. 504–524. Springer, Heidelberg (2012). Scholar
  5. 5.
    Beyer, D.: Software verification and verifiable witnesses. In: Baier, C., Tinelli, C. (eds.) TACAS 2015. LNCS, vol. 9035, pp. 401–416. Springer, Heidelberg (2015). Scholar
  6. 6.
    Duret-Lutz, A., Poitrenaud, D.: SPOT: an extensible model checking library using transition-based generalized Büchi automata. In: 12th Annual International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunications Systems (MASCOTS 2004), pp. 76–83. IEEE (2004)Google Scholar
  7. 7.
    Erickson, K.T.: Programmable logic controllers. IEEE Potentials 15(1), 14–17 (1996). Scholar
  8. 8.
    Gastin, P., Oddoux, D.: Fast LTL to Büchi automata translation. In: Berry, G., Comon, H., Finkel, A. (eds.) CAV 2001. LNCS, vol. 2102, pp. 53–65. Springer, Heidelberg (2001). Scholar
  9. 9.
    Geske, M., Jasper, M., Steffen, B., Howar, F., Schordan, M., van de Pol, J.: RERS 2016: parallel and sequential benchmarks with focus on LTL verification. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9953, pp. 787–803. Springer, Cham (2016). Scholar
  10. 10.
    Gourcuff, V., Smet, O.D., Faure, J.M.: Efficient representation for formal verification of PLC programs. In: 2006 8th International Workshop on Discrete Event Systems, pp. 182–187, July 2006Google Scholar
  11. 11.
    Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D., Păsăreanu, C.: Rigorous examination of reactive systems. The RERS challenges 2012 and 2013. STTT 16(5), 457–464 (2014)CrossRefGoogle Scholar
  12. 12.
    Huisman, M., Klebanov, V., Monahan, R.: VerifyThis 2012. STTT 17(6), 647–657 (2015)CrossRefGoogle Scholar
  13. 13.
    Jasper, M., et al.: The RERS 2017 challenge and workshop (invited paper). In: Proceedings of the 24th ACM SIGSOFT International SPIN Symposium on Model Checking of Software, SPIN 2017, pp. 11–20. ACM (2017)Google Scholar
  14. 14.
    Jasper, M., Schordan, M.: Multi-core model checking of large-scale reactive systems using different state representations. In: Margaria, T., Steffen, B. (eds.) ISoLA 2016. LNCS, vol. 9952, pp. 212–226. Springer, Cham (2016). Scholar
  15. 15.
    Jasper, M., Steffen, B.: Synthesizing verification benchmarks with subtle bugs for given property profiles. (To appear)Google Scholar
  16. 16.
    Johnson, D.B.: Finding all the elementary circuits of a directed graph. SIAM J. Comput. 4(1), 77–84 (1975)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Kordon, F.: Report on the model checking contest at petri nets 2011. In: Jensen, K., van der Aalst, W.M., Ajmone Marsan, M., Franceschinis, G., Kleijn, J., Kristensen, L.M. (eds.) Transactions on Petri Nets and Other Models of Concurrency VI. LNCS, vol. 7400, pp. 169–196. Springer, Heidelberg (2012). Scholar
  18. 18.
    Larsen, K.G.: Modal specifications. In: Sifakis, J. (ed.) CAV 1989. LNCS, vol. 407, pp. 232–246. Springer, Heidelberg (1990). Scholar
  19. 19.
    Mealy, G.H.: A method for synthesizing sequential circuits. Bell Syst. Tech. J. 34(5), 1045–1079 (1955)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Miller, S.P., Whalen, M.W., Cofer, D.D.: Software model checking takes off. Commun. ACM 53(2), 58–64 (2010)CrossRefGoogle Scholar
  21. 21.
    Moon, I.: Modeling programmable logic controllers for logic verification. IEEE Control Syst. 14(2), 53–59 (1994)CrossRefGoogle Scholar
  22. 22.
    Peterson, J.L.: Petri Net Theory and the Modeling of Systems. Prentice Hall PTR, Upper Saddle River (1981)zbMATHGoogle Scholar
  23. 23.
    Plotkin, G.D.: A structural approach to operational semantics. DAIMI FN-19, Computer Science Department, Aarhus University (1981)Google Scholar
  24. 24.
    Pnueli, A., Rosner, R.: On the synthesis of a reactive module. In: Proceedings of the 16th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 1989, pp. 179–190. ACM, New York (1989)Google Scholar
  25. 25.
    Rausch, M., Krogh, B.H.: Formal verification of PLC programs. In: Proceedings of the 1998 American Control Conference, ACC, vol. 1, pp. 234–238, June 1998Google Scholar
  26. 26.
    Steffen, B., Jasper, M., Meijer, J., van de Pol, J.: Property-preserving generation of tailored benchmark petri nets. In: 17th International Conference on Application of Concurrency to System Design (ACSD), pp. 1–8, June 2017Google Scholar
  27. 27.
    Steffen, B., Howar, F., Isberner, M., Naujokat, S., Margaria, T.: Tailored generation of concurrent benchmarks. STTT 16(5), 543–558 (2014)CrossRefGoogle Scholar
  28. 28.
    Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation: synthesizing programs of realistic structure. STTT 16(5), 465–479 (2014)CrossRefGoogle Scholar
  29. 29.
    Steffen, B., Jasper, M.: Property-preserving parallel decomposition. In: Aceto, L., Bacci, G., Bacci, G., Ingólfsdóttir, A., Legay, A., Mardare, R. (eds.) Models, Algorithms, Logics and Tools. LNCS, vol. 10460, pp. 125–145. Springer, Cham (2017). Scholar
  30. 30.
    Visser, W., Mehlitz, P.: Model checking programs with Java PathFinder. In: Godefroid, P. (ed.) SPIN 2005. LNCS, vol. 3639, p. 27. Springer, Heidelberg (2005). Scholar
  31. 31.
    Wolper, P.: Temporal logic can be more expressive. Inf. Control 56(1), 72–99 (1983)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Zhivich, M., Cunningham, R.K.: The real cost of software errors. IEEE Secur. Priv. 7(2), 87–90 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.TU Dortmund UniversityDortmundGermany

Personalised recommendations