Property-driven benchmark generation: synthesizing programs of realistic structure

  • Bernhard Steffen
  • Malte Isberner
  • Stefan NaujokatEmail author
  • Tiziana Margaria
  • Maren Geske


We present a systematic approach to the automatic generation of platform-independent benchmarks of realistic structure and tailored complexity for evaluating verification tools for reactive systems. The idea is to mimic a systematic constraint-driven software development process by automatically transforming randomly generated temporal-logic-based requirement specifications on the basis of a sequence of property-preserving, randomly generated structural design decisions into executable source code of a chosen target language or platform. Our automated transformation process steps through dedicated representations in terms of Büchi automata, Mealy machines, decision diagram models, and code models. It comprises LTL synthesis, model checking, property-oriented expansion, path condition extraction, theorem proving, SAT solving, and code motion. This setup allows us to address different communities via a growing set of programming languages, tailored sets of programming constructs, different notions of observation, and the full variety of LTL properties—ranging from mere reachability over general safety properties to arbitrary liveness properties. The paper illustrates the corresponding tool chain along accompanying examples, emphasizes the current state of development, and sketches the envisioned potential and impact of our approach.


Benchmark generation LTL synthesis Model checking Property-oriented expansion Path condition extraction Theorem proving SAT solving Code motion 


  1. 1.
    Almeida, E.E., Luntz, J.E., Tilbury, D.M.: Event-condition–action systems for reconfigurable logic control. IEEE Trans. Autom. Sci. Eng. 4(2), 167–181 (2007)Google Scholar
  2. 2.
    Apt, K.R., Olderog, E.-R.: Verification of Sequential and Concurrent Programs. Texts and Monographs in Computer Science. Springer, New York, NY, USA (1991)Google Scholar
  3. 3.
    Bauer, O., Geske, M., Isberner, M.: Analyzing program behavior through active automata learning. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0333-2 (2014)
  4. 4.
    Beyer, D., Stahlbauer, A.: BDD-based software verification. Applications to event-condition–action systems. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0334-1 (2014)
  5. 5.
    Beyer, D. Competition on software verification—(SV-COMP). In: Proceedings of of 18th Int. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2012), vol. 7214 of LNCS, pp. 504–524. Springer (2012)Google Scholar
  6. 6.
    Biere, A., Heule, M.J.H., van Maaren, H., Walsh, T. (editors): Handbook of Satisfiability, vol. 185 of Frontiers in Artificial Intelligence and Applications. IOS Press, Amsterdam, NL (2009)Google Scholar
  7. 7.
    Blom, S.C.C., van de Pol, J.C., Weber, M.: Ltsmin: distributed and symbolic reachability. In: Touili, T., Cook, B., Jackson, P. (eds.) Computer Aided Verification. Edinburgh, vol. 6174 of Lecture Notes in Computer Science, pp. 354–359. Springer, Berlin (2010)Google Scholar
  8. 8.
    Briggs, P., Cooper, K.D.: Effective partial redundancy elimination. In: Proceedings of ACM SIGPLAN Conf. Prog. Lang. Design and Impl. (PLDI’94), pp. 159–170 (1994)Google Scholar
  9. 9.
    Burkart, O., Steffen, B.: Model checking the full modal Mu-Calculus for infinite sequential processes. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) Automata. Languages and Programming, vol. 1256 of Lecture Notes in Computer Science, pp. 419–429. Springer, Berlin Heidelberg (1997)Google Scholar
  10. 10.
    Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. MIT Press, Cambridge, MA, USA (2001)Google Scholar
  11. 11.
    Combe, D., de la Higuera, C., Zulu, J.-C.J.: An interactive learning competition. In: Revised Selected Papers of 8th International Workshop on Finite-State Methods and Natural Language Processing (FSMNLP 2009), vol. 6062, pp. 139–146. Springer (2010)Google Scholar
  12. 12.
    Dhamdhere, D.M.: A new algorithm for composite hoisting and strength reduction optimisation (+ Corrigendum). Int. J. Comp. Math. 27, 1–14 (1989)CrossRefGoogle Scholar
  13. 13.
    Dwyer, M.B., Avrunin, G.S., Corbett, J.C.: Patterns in property specifications for finite-state verification. In: Proceedings of the 1999 Int. Conf. on Software Engineering, pp. 411–420. IEEE (1999)Google Scholar
  14. 14.
    Floyd, R.W.: Assigning meaning to programs. In: Proceedongs of Symposium on Applied Mathematics, vol. 19 of Mathematical aspects of computer science, pp. 19–32. American Mathematical Society (1967)Google Scholar
  15. 15.
    Gastin, P., Oddoux, D.: Fast LTL to Büchi Automata Translation. In: Berry, G., Comon, H., Finkel, A., (eds) Proceedings of the 13th International Conference on Computer Aided Verification (CAV’01), vol. 2102 of Lecture Notes in Computer Science, pp. 53–65, Paris, France. Springer (2001)Google Scholar
  16. 16.
    Giannakopoulou, D., Lerda, F.: From states to transitions: improving translation of LTL formulae to Büchi Automata. In: Proceedings of the 22nd IFIP WG 6.1 International Conference Houston on Formal Techniques for Networked and Distributed Systems, FORTE ’02, pp. 308–326, London, UK. Springer (2002)Google Scholar
  17. 17.
    Hayes-Roth, F.: Rule-based systems. Commun. ACM 28(9), 921–932 (1985)Google Scholar
  18. 18.
    Hoare, C.A.R.: An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)CrossRefzbMATHGoogle Scholar
  19. 19.
    Holzmann, G.J.: The SPIN Model Checker—Primer and Reference Manual. Addison-Wesley, Boston, MA, USA (2004)Google Scholar
  20. 20.
    Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D., Pasareanu, C.S.: Rigorous examination of reactive systems. The RERS challenges 2012 and 2013. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0337-y (2014)
  21. 21.
    Howar, F., Isberner, M., Merten, M., Steffen, B., Beyer, D.: The RERS grey-box challenge 2012: analysis of event-condition–action systems. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change, vol. 7609 of Lecture Notes in Computer Science, pp. 608–614. Springer, Berlin Heidelberg (2012)Google Scholar
  22. 22.
    Huth, M.: Model checking modal transition systems using kripke structures. In: Revised Papers from the Third International Workshop on Verification, Model Checking, and Abstract Interpretation, VMCAI ’02, pp. 302–316, London. Springer (2002)Google Scholar
  23. 23.
    Kaynar, D.K., Lynch, N.A., Segala, R., Vaandrager, F.W.: Timed I/O Automata: a mathematical framework for modeling and analyzing real-time systems. In: Proceedings of the 24th IEEE Real-Time Systems Symposium (RTSS 2003), pp. 166–177. IEEE Computer Society (2003)Google Scholar
  24. 24.
    Klebanov, V., Müller, P., Shankar, N., Leavens, G.T., Wüstholz, V., Alkassar, E., Arthan, R., Bronish, D., Chapman, R., Cohen, E., Hillebrand, M.A., Rustan, J.K., Leino, M., Monahan, R., Piessens, F., Polikarpova, N., Ridge, T., Smans, J., Tobies, S., Tuerk, T., Ulbrich, M., Weiß, B.: The 1st verified software competition: experience report. In: Proceedings of the 17th Int. Symposium on Formal Methods (FM 2011), vol. 6664 of LNCS, pp. 154–168 (2011)Google Scholar
  25. 25.
    Knoop, J., Rüthing, O., Steffen, B.: Expansion-based removal of semantic partial redundancies. In: Compiler Construction, 8th International Conference, CC’99, Held as Part of the European Joint Conferences on the Theory and Practice of Software, ETAPS’99, Amsterdam, The Netherlands, 22–28 March, 1999, Proceedings, vol. 1575 of LNCS, pp. 91–106. Springer (1999)Google Scholar
  26. 26.
    Knoop, J., Rüthing, O., Steffen, B.: Lazy code motion. In: Proceedings of the ACM SIGPLAN’92 Conference on Programming Language Design and Implementation (PLDI), pp. 224–234. ACM (1992)Google Scholar
  27. 27.
    Knoop, J., Rüthing, O., Steffen, B.: Lazy strength reduction. J. Program. Lang. 1, 71–91 (1993)Google Scholar
  28. 28.
    Knoop, J., Rüthing, O., Steffen, B.: Optimal code motion: theory and practice. ACM Trans. Program. Lang. Syst. 16(4), 1117–1155 (1994)Google Scholar
  29. 29.
    Knoop, J., Rüthing, O., Steffen, B.: Partial dead code elimination. In: Proceedings of the ACM SIGPLAN’94 Conference on Programming Language Design and Implementation (PLDI), pp. 147–158. ACM (1994)Google Scholar
  30. 30.
    Knoop, J., Rüthing, O., Steffen, B. The power of assignment motion. In: Proceedings of the ACM SIGPLAN’95 Conference on Programming Language Design and Implementation (PLDI). ACM (1995)Google Scholar
  31. 31.
    Lamport, L.: What good is temporal logic? Inf. Process. 83, 657–668 (1983)Google Scholar
  32. 32.
    Larsen, K.G.: Modal specifications. In: Automatic Verification Methods for Finite State Systems, pp. 232–246 (1989)Google Scholar
  33. 33.
    Margaria, T., Steffen, B.: Continuous model-driven engineering. IEEE Computer 42(10), 106–109 (2009)Google Scholar
  34. 34.
    McCarthy, D.R., Dayal, U.: The architecture of an active data base management system. In: Proceedings of the 1989 ACM SIGMOD International Conference on Management of Data, pp. 215–224. ACM Press (1989)Google Scholar
  35. 35.
    Mealy, G.H.: A method for synthesizing sequential circuits. Bell Syst. Tech. J. 34(5), 1045–1079 (1955)Google Scholar
  36. 36.
    Milner, R.: Communication and Concurrency. PHI Series in Computer Science. Prentice-Hall Inc, Upper Saddle River (1989)Google Scholar
  37. 37.
    Morel, E., Renvoise, C.: Global optimization by suppression of partial redundancies. Commun. ACM 22(2), 96–103 (1979)CrossRefzbMATHMathSciNetGoogle Scholar
  38. 38.
    Morse, J., Cordeiro, L., Nicole, D., Fischer, B.: Applying symbolic bounded model checking to the 2012 RERS greybox challenge. Soft. Tools Technol. Transf. doi: 10.1007/s10009-014-0335-0 (2014)
  39. 39.
    Robinson, A., Voronkov, A. (eds): Handbook of Automated Reasoning, vol. I and II. Elsevier, Amsterdam, NL (2001)Google Scholar
  40. 40.
    Rosen, B.K., Wegman, M.N., Kenneth Zadeck, F.: Global value numbers and redundant computations. In: Conference Record of the Fifteenth Annual ACM Symposium on Principles of Programming Languages. ACM Press (1988)Google Scholar
  41. 41.
    Rüthing, O., Knoop, J., Steffen, B.: Sparse code motion. In: Proceedings of the 27th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL 2000), pp. 170–183. ACM (2000)Google Scholar
  42. 42.
    SCCE: Service Centered Continuous Engineering. Accessed 26 Feb 2014
  43. 43.
    Schordan, M., Prantl, A.: Combining static analysis and state transition graphs for verification of event-condition–action systems in the RERS 2012 and 2013 challenges. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0338-x (2014)
  44. 44.
    Snelting, G., Robschnik, T., Krinke, J.: Efficient path conditions in dependence graphs for software safety analysis. ACM Trans. Softw. Eng. Methodol. (TOSEM) 15(4), 410–457 (2006)Google Scholar
  45. 45.
    Steffen, B., Claßen, A., Klein, M., Knoop, J., Margaria, T.: The fixpoint-analysis machine. In: Lee, I., Smolka, S.A. (eds) CONCUR ’95: Concurrency Theory, volume 962 of Lecture Notes in Computer Science, pp. 72–87. Springer, Berlin, Heidelberg (1995)Google Scholar
  46. 46.
    Steffen, B., Howar, F., Isberner, M., Naujokat, S., Margaria, T.: Tailored generation of concurrent benchmarks. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0339-9 (2014)
  47. 47.
    Steffen, B., Isberner, M., Naujokat, S., Margaria, T., Geske, M.: Property-driven benchmark generation. In: International SPIN Symposium on Model Checking of Software (SPIN2013), vol. 7976 of LNCS, pp. 341–357. Springer (2013)Google Scholar
  48. 48.
    Steffen, B., Knoop, J., Rüthing, O.: Efficient code motion and an adaption to strength reduction. In: Proceedings of the International Joint Conference on Theory and Practice of Software Development (TAPSOFT’91). Volume 2: Advances in Distributed Computing (ADC) and Colloquium on Combining Paradigms for Software Developmemnt (CCPSD), vol. 494 of LNCS, pp. 394–415. Springer (1991)Google Scholar
  49. 49.
    Steffen, B., Knoop, J., Rüthing, O.: The value flow graph: a program representation for optimal program transformations. In: Jones, N.D. (ed.) 3rd European Symposium on Programming (ESOP’90). Lecture Notes in Computer Science, vol. 432, pp. 389–405. Springer, Berlin Heidelberg (1990)Google Scholar
  50. 50.
    Steffen, B., Knoop, J.: Finite constants: characterizations of a new decidable set of constants. In: Kreczmar, A., Mirkowska, G. (eds) Mathematical Foundations of Computer Science (MFCS’89), vol. 379 of LNCS, pp. 481–491. Springer (1989)Google Scholar
  51. 51.
    Steffen, B., Margaria, T., Nagel, R., Jörges, S., Kubczak, C.: Model-driven development with the jABC. In: Bin, E., Ziv, A., Ur, S. (eds) Haifa Verification Conference, vol. 4383 of Lecture Notes in Computer Science, pp. 92–108. Springer (2006)Google Scholar
  52. 52.
    Steffen, B., Narayan, P.: Full life-cycle support for end-to-end processes. IEEE Computer 40(11), 64–73 (2007)Google Scholar
  53. 53.
    Steffen, B., Rüthing, O.: Quality engineering: leveraging heterogeneous information—(Invited Talk). In: Proceedings of the 12th Int. Conf. on Verification, Model Checking, and Abstract Interpretation (VMCAI 2011), LNCS, pp. 23–37 (2011)Google Scholar
  54. 54.
    Steffen, B.: Characteristic formulae. In: Ausiello, G., Dezani-Ciancaglini, M., Rocca, S.R. (eds.) Automata. Languages and Programming, vol. 372 of Lecture Notes in Computer Science, pp. 723–732. Springer, Berlin Heidelberg (1989) Google Scholar
  55. 55.
    Steffen, B.: Property-oriented expansion. In: Cousot, R., Schmidt, D.A. (eds.) Third International Symposium on Static Analysis (SAS ’96). Lecture Notes in Computer Science, vol. 1145, pp. 22–41. Springer, Berlin, Heidelberg (1996)Google Scholar
  56. 56.
    Steffen, B.: Unifying models. In: Reischuk, R., Morvan, M. (eds.) STACS 97. Lecture Notes in Computer Science, vol. 1200, pp. 1–20. Springer, Berlin, Heidelberg (1997)Google Scholar
  57. 57.
    van de Pol, J., Ruys, T.C., te Brinke, S.: Thoughtful brute force attack of the RERS 2012 and 2013 challenges. Softw. Tools Technol. Transf. doi: 10.1007/s10009-014-0324-3 (2014)
  58. 58.
    Yang, X., Chen, Y., Eide, E., Regehr, J.: Finding and understanding bugs in C compilers. In: Proceedings of the 32nd ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI 2011) (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Bernhard Steffen
    • 1
  • Malte Isberner
    • 1
  • Stefan Naujokat
    • 1
    Email author
  • Tiziana Margaria
    • 2
  • Maren Geske
    • 1
  1. 1.Chair for Programming SystemsTechnische Universität DortmundDortmundGermany
  2. 2.Chair Service and Software EngineeringUniversität PotsdamPotsdamGermany

Personalised recommendations