The first reactive synthesis competition (SYNTCOMP 2014)

  • Swen Jacobs
  • Roderick Bloem
  • Romain Brenguier
  • Rüdiger Ehlers
  • Timotheus Hell
  • Robert Könighofer
  • Guillermo A. Pérez
  • Jean-François Raskin
  • Leonid Ryzhyk
  • Ocan Sankur
  • Martina Seidl
  • Leander Tentrup
  • Adam Walker
Regular Paper
First Online:

Abstract

We introduce the reactive synthesis competition (SYNTCOMP), a long-term effort intended to stimulate and guide advances in the design and application of synthesis procedures for reactive systems. The first iteration of SYNTCOMP is based on the controller synthesis problem for finite-state systems and safety specifications. We provide an overview of this problem and existing approaches to solve it, and report on the design and results of the first SYNTCOMP. This includes the definition of the benchmark format, the collection of benchmarks, the rules of the competition, and the five synthesis tools that participated. We present and analyze the results of the competition and draw conclusions on the state of the art. Finally, we give an outlook on future directions of SYNTCOMP.

Keywords

Synthesis Reactive systems Competition Experimental evaluation Benchmarks Safety games 
This is a preview of subscription content, log in to check access

Notes

Acknowledgments

We thank the anonymous reviewers for their detailed and insightful comments on drafts of this article. We thank Armin Biere for his advice on running a competition, and Ayrat Khalimov for supplying the reference implementation Aisy for the competition. The organization of SYNTCOMP 2014 was supported by the Austrian Science Fund (FWF) through projects RiSE (S11406-N23) and QUAINT (I774-N23), by the German Research Foundation (DFG) as part of the Transregional Collaborative Research Center “Automatic Verification and Analysis of Complex Systems” (SFB/TR 14 AVACS) and through project “Automatic Synthesis of Distributed and Parameterized Systems” (JA 2357/2-1), as well as by the Institutional Strategy of the University of Bremen, funded by the German Excellence Initiative. The development of AbsSynthe was supported by an F.R.S.-FNRS fellowship, and the ERC inVEST (279499) project. The development of Basil was supported by the Institutional Strategy of the University of Bremen, funded by the German Excellence Initiative. The development of \(\texttt {Demiurge}\) was supported by the FWF through projects RiSE (S11406-N23, S11408-N23) and QUAINT (I774-N23). The development of Realizer was supported by the DFG as part of SFB/TR 14 AVACS. The development of Simple BDD Solver was supported by a gift from the Intel Corporation, and NICTA is funded by the Australian Government through the Department of Communications and the Australian Research Council through the ICT Centre of Excellence Program.

References

  1. 1.
    Alur, R., Bodík, R., Dallal, E., Fisman, D., Garg, P., Juniwal, G., Kress-Gazit, H., Madhusudan, P., Martin, M.M.K., Raghothaman, M., Saha, S., Seshia, S.A., Singh, R., Solar-Lezama, A., Torlak, E., Udupa, A.: Syntax-guided synthesis. In: Dependable Software Systems Engineering, NATO Science for Peace and Security Series, D: Information and Communication Security, vol. 40, pp. 1–25. IOS Press (2015)Google Scholar
  2. 2.
    Alur, R., Madhusudan, P., Nam, W.: Symbolic computational techniques for solving games. STTT 7(2), 118–128 (2005)CrossRefMATHGoogle Scholar
  3. 3.
    Aziz, A., Tasiran, S., Brayton, R.K.: BDD variable ordering for interacting finite state machines. In: DAC, pp. 283–288 (1994)Google Scholar
  4. 4.
    Balint, A., Diepold, D., Gall, D., Gerber, S., Kapler, G., Retz, R.: EDACC - an advanced platform for the experiment design, administration and analysis of empirical algorithms. In: LION 5. Selected Papers, LNCS, vol. 6683, pp. 586–599. Springer, (2011)Google Scholar
  5. 5.
    Barrett, C.W., de Moura, L.M., Stump, A.: Design and results of the first satisfiability modulo theories competition (SMT-COMP 2005). J. Autom. Reason. 35(4), 373–390 (2005)CrossRefMATHGoogle Scholar
  6. 6.
    Beyer, D.: Competition on software verification - (SV-COMP). In: TACAS, LNCS, vol. 7214, pp. 504–524. Springer (2012)Google Scholar
  7. 7.
    Beyer, D., Löwe, S., Wendler, P.: Benchmarking and resource measurement. In: SPIN 2015, LNCS, vol. 9232, pp. 160–178. Springer (2015)Google Scholar
  8. 8.
    Bloem, R., Cimatti, A., Greimel, K., Hofferek, G., Könighofer, R., Roveri, M., Schuppan, V., Seeber, R.: RATSY - A new requirements analysis tool with synthesis. In: CAV, LNCS, vol. 6174, pp. 425–429. Springer (2010)Google Scholar
  9. 9.
    Bloem, R., Egly, U., Klampfl, P., Könighofer, R., Lonsing, F.: SAT-based methods for circuit synthesis. In: FMCAD’14, pp. 31–34. IEEE (2014)Google Scholar
  10. 10.
    Bloem, R., Galler, S.J., Jobstmann, B., Piterman, N., Pnueli, A., Weiglhofer, M.: Automatic hardware synthesis from specifications: a case study. In: DATE, pp. 1188–1193. ACM (2007)Google Scholar
  11. 11.
    Bloem, R., Galler, S.J., Jobstmann, B., Piterman, N., Pnueli, A., Weiglhofer, M.: Specify, compile, run: hardware from PSL. Electr. Notes Theor. Comput. Sci. 190(4), 3–16 (2007)CrossRefGoogle Scholar
  12. 12.
    Bloem, R., Jobstmann, B., Piterman, N., Pnueli, A., Sa’ar, Y.: Synthesis of reactive(1) designs. J. Comput. Syst. Sci. 78(3), 911–938 (2012)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Bloem, R., Könighofer, R., Seidl, M.: SAT-based synthesis methods for safety specs. In: VMCAI, LNCS, vol. 8318, pp. 1–20. Springer (2014)Google Scholar
  14. 14.
    Bohy, A., Bruyère, V., Filiot, E., Jin, N., Raskin, J.-F.: Acacia+, a tool for LTL synthesis. In: CAV, LNCS, vol. 7358, pp. 652–657. Springer (2012)Google Scholar
  15. 15.
    Bradley, A.R.: SAT-based model checking without unrolling. In: VMCAI, LNCS, vol. 6538, pp. 70–87. Springer (2011)Google Scholar
  16. 16.
    Brayton, R.K., Hachtel, G.D., Sangiovanni-Vincentelli, A.L., Somenzi, F., Aziz, A., Cheng, S., Edwards, S.A., Khatri, S.P., Kukimoto, Y., Pardo, A., Qadeer, S., Ranjan, R.K., Sarwary, S., Shiple, T.R., Swamy, G., Villa, T.: VIS: a system for verification and synthesis. In: CAV, LNCS, vol. 1102, pp. 428–432. Springer (1996)Google Scholar
  17. 17.
    Brayton, R.K., Mishchenko, A.: ABC: an academic industrial-strength verification tool. In: CAV, LNCS, vol. 6174, pp. 24–40. Springer (2010)Google Scholar
  18. 18.
    Brenguier, R., Pérez, G.A., Raskin, J.-F., Sankur, O.: AbsSynthe: abstract synthesis from succinct safety specifications. In: SYNT, EPTCS, vol. 157, pp. 100–116. Open Publishing Association (2014)Google Scholar
  19. 19.
    Bryant, R.E.: Graph-based algorithms for boolean function manipulation. IEEE Trans. Comput. 35(8), 677–691 (1986)CrossRefMATHGoogle Scholar
  20. 20.
    Büchi, J., Landweber, L.: Solving sequential conditions by finite-state strategies. Trans. Am. Math. Soc. 138, 295–311 (1969)CrossRefMATHGoogle Scholar
  21. 21.
    Burch, J.R., Clarke, E.M., Long, D.E.: Symbolic model checking with partitioned transistion relations. In: VLSI, pp. 49–58 (1991)Google Scholar
  22. 22.
    Chiang, T., Jiang. J.R.: Property-directed synthesis of reactive systems from safety specifications. In: ICCAD, pp. 794–801. ACM (2015)Google Scholar
  23. 23.
    Church, A.: Logic, arithmetic and automata. In: Proceedings of the International Congress of Mathematicians, pp. 23–35 (1962)Google Scholar
  24. 24.
    Coudert, O., Madre, J.C.: A unified framework for the formal verification of sequential circuits. In: ICCAD, pp. 126–129 (1990)Google Scholar
  25. 25.
    Cousot, P., Cousot, R.: Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: POPL, pp. 238–252. ACM (1977)Google Scholar
  26. 26.
    de Alfaro, L., Roy, P.: Solving games via three-valued abstraction refinement. In: CONCUR, LNCS, vol. 4703, pp. 74–89. Springer (2007)Google Scholar
  27. 27.
    Ehlers, R.: Experimental aspects of synthesis. In: iWIGP, EPTCS, vol. 50, pp. 1–16 (2011)Google Scholar
  28. 28.
    Ehlers, R.: Unbeast: symbolic bounded synthesis. In: TACAS, LNCS, vol. 6605, pp. 272–275. Springer (2011)Google Scholar
  29. 29.
    Ehlers, R.: Symbolic bounded synthesis. Formal Methods Syst. Des. 40(2), 232–262 (2012)CrossRefMATHGoogle Scholar
  30. 30.
    Ehlers, R., Könighofer, R., Hofferek, G.: Symbolically synthesizing small circuits. In: FMCAD’12, pp. 91–100. IEEE (2012)Google Scholar
  31. 31.
    Emerson, E.A., Clarke, E.M.: Using branching time temporal logic to synthesize synchronization skeletons. Sci. Comput. Program. 2(3), 241–266 (1982)CrossRefMATHGoogle Scholar
  32. 32.
    Filiot, E., Jin, N., Raskin, J.: Exploiting structure in LTL synthesis. STTT 15(5–6), 541–561 (2013)CrossRefGoogle Scholar
  33. 33.
    Filiot, E., Jin, N., Raskin, J.-F.: Antichains and compositional algorithms for LTL synthesis. Formal Methods Syst. Des. 39(3), 261–296 (2011)CrossRefMATHGoogle Scholar
  34. 34.
    Finkbeiner, B., Jacobs, S.: Lazy synthesis. In: VMCAI, LNCS, vol. 7148, pp. 219–234. Springer (2012)Google Scholar
  35. 35.
    Finkbeiner, B., Schewe, S.: Bounded synthesis. STTT 15(5–6), 519–539 (2013)CrossRefMATHGoogle Scholar
  36. 36.
    Graf, S., Saïdi, H.: Construction of abstract state graphs with PVS. In: CAV, LNCS, vol. 1254, pp. 72–83. Springer (1997)Google Scholar
  37. 37.
    Henzinger, T.A., Jhala, R., Majumdar, R.: Counterexample-guided control. In: ICALP, LNCS, vol. 2719, pp. 886–902, Springer (2003)Google Scholar
  38. 38.
    Hong, Y., Beerel, P.A., Burch, J.R., McMillan, K.L.: Sibling-substitution-based BDD minimization using don’t cares. IEEE Trans. CAD of Integr. Circuits Syst. 19(1), 44–55 (2000)CrossRefGoogle Scholar
  39. 39.
    Jacobs, S.: Extended AIGER format for synthesis. CoRR (2014). arXiv:1405.5793. Accessed Feb 2016
  40. 40.
    Jacobs, S., Bloem, R., Brenguier, R., Ehlers, R., Hell, T., Könighofer, R., Pérez, G.A., Raskin, J., Ryzhyk, L., Sankur, O., Seidl, M., Tentrup, L., Walker, A.: The first reactive synthesis competition (SYNTCOMP 2014). CoRR (2015). arXiv:1506.08726. Accessed Feb 2016
  41. 41.
    Jacobs, S., Bloem, R., Brenguier, R., Könighofer, R., Pérez, G.A., Raskin, J.-F., Ryzhyk, L., Sankur, O., Seidl, M., Tentrup, L., Walker, A.: The second reactive synthesis competition (SYNTCOMP 2015). In: SYNT, EPTCS, vol. 202, pp. 27–57. Open Publishing Association (2016)Google Scholar
  42. 42.
    Jacobs, S., Klein, F.: A high-level LTL synthesis format: TLSF v1.0. CoRR (2016). arXiv:1601.05228. Accessed Feb 2016
  43. 43.
    Järvisalo, M., Berre, D.L., Roussel, O., Simon, L.: The international SAT solver competitions. AI Mag 33(1), 89–94 (2012)Google Scholar
  44. 44.
    Jobstmann, B., Bloem, R.: Optimizations for LTL synthesis. In: FMCAD, pp. 117–124. IEEE Computer Society (2006)Google Scholar
  45. 45.
    Jobstmann, B., Galler, S.J., Weiglhofer, M., Bloem, R.: Anzu: A tool for property synthesis. In: CAV, LNCS, vol. 4590, pp. 258–262. Springer (2006)Google Scholar
  46. 46.
    Kupferman, O., Vardi, M.Y.: Safraless decision procedures. In: FOCS, pp. 531–542. IEEE Computer Society (2005)Google Scholar
  47. 47.
    Kurshan, R.P.: Automata-theoretic verification of coordinating processes. In: Analysis and Optimization of Systems: Discrete Event Systems, pp. 16–28. Springer (1994)Google Scholar
  48. 48.
    Lecoutre, C., Roussel, O., van Dongen, M.R.C.: Promoting robust black-box solvers through competitions. Constraints 15(3), 317–326 (2010)CrossRefMATHGoogle Scholar
  49. 49.
    Mishchenko, A., Chatterjee, S., Brayton, R.K.: Dag-aware AIG rewriting a fresh look at combinational logic synthesis. In: DAC, pp. 532–535. ACM (2006)Google Scholar
  50. 50.
    Mishchenko, A., Chatterjee, S., Jiang, R., Brayton, R.: FRAIGs: A unifying representation for logic synthesis and verification. Technical report, EECS Department, U. C. Berkeley (2005)Google Scholar
  51. 51.
    Morgenstern, A., Gesell, M., Schneider, K.: Solving games using incremental induction. In: IFM’13, LNCS 7940, pp. 177–191. Springer (2013)Google Scholar
  52. 52.
    Niemetz, A., Preiner, M., Lonsing, F., Seidl, M., Biere, A.: Resolution-based certificate extraction for QBF. In: SAT’12, LNCS 7317, pp. 430–435. Springer (2012)Google Scholar
  53. 53.
    Pnueli, A., Rosner, R.: On the synthesis of a reactive module. In: POPL, pp. 179–190. ACM Press (1989)Google Scholar
  54. 54.
    Rabin, M.O.: Decidability of second-order theories and automata on infinite trees. Trans. Am. Math. Soc. 141, 1–35 (1969)MATHGoogle Scholar
  55. 55.
    Ranjan, R.K., Aziz, A., Brayton, R.K., Plessier, B., Pixley, C.: Efficient bdd algorithms for fsm synthesis and verification. In: International Workshop on Logic Synthesis (1995)Google Scholar
  56. 56.
    Roussel, O.: Controlling a solver execution with the runsolver tool. JSAT 7(4), 139–144 (2011)MathSciNetMATHGoogle Scholar
  57. 57.
    Rudell, R.: Dynamic variable ordering for ordered binary decision diagrams. In: ICCAD, pp. 42–47. IEEE Computer Society (1993)Google Scholar
  58. 58.
    Seidl, M., Könighofer, R.: Partial witnesses from preprocessed quantified boolean formulas. In: DATE’14, pp. 1–6. IEEE (2014)Google Scholar
  59. 59.
    Sohail, S., Somenzi, F.: Safety first: a two-stage algorithm for the synthesis of reactive systems. STTT 15(5–6), 433–454 (2013)CrossRefGoogle Scholar
  60. 60.
    Somenzi, F.: Binary decision diagrams. In: Calculational System Design, vol. 173, pp. 303. IOS Press (1999)Google Scholar
  61. 61.
    Sutcliffe, G., Suttner, C.B.: Evaluating general purpose automated theorem proving systems. Artif. Intell. 131(1–2), 39–54 (2001)MathSciNetCrossRefMATHGoogle Scholar
  62. 62.
    Sutcliffe, G., Suttner, C.B.: The state of CASC. AI Commun. 19(1), 35–48 (2006)MathSciNetMATHGoogle Scholar
  63. 63.
    Thomas, W.: On the synthesis of strategies in infinite games. In: STACS, pp. 1–13 (1995)Google Scholar
  64. 64.
    van Dijk, T., Laarman, A., van de Pol, J.: Multi-core BDD operations for symbolic reachability. Electron. Notes Theor. Comput. Sci. 296, 127–143 (2013)CrossRefGoogle Scholar
  65. 65.
    van Dijk, T., van de Pol, J.: Sylvan: Multi-core decision diagrams. In: TACAS 2015, LNCS, vol. 9035, pp. 677–691 Springer (2015)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Swen Jacobs
    • 1
    • 2
  • Roderick Bloem
    • 1
  • Romain Brenguier
    • 3
  • Rüdiger Ehlers
    • 4
    • 5
  • Timotheus Hell
    • 1
  • Robert Könighofer
    • 1
  • Guillermo A. Pérez
    • 3
  • Jean-François Raskin
    • 3
  • Leonid Ryzhyk
    • 6
    • 7
  • Ocan Sankur
    • 3
    • 8
  • Martina Seidl
    • 9
  • Leander Tentrup
    • 2
  • Adam Walker
    • 6
  1. 1.Graz University of TechnologyGrazAustria
  2. 2.Saarland UniversitySaarbrückenGermany
  3. 3.Université Libre de BruxellesBrusselsBelgium
  4. 4.University of BremenBremenGermany
  5. 5.DFKI GmbHBremenGermany
  6. 6.NICTASydneyAustralia
  7. 7.Carnegie Mellon UniversityPittsburghUSA
  8. 8.CNRS, IRISARennesFrance
  9. 9.Johannes-Kepler-University LinzLinzAustria

Personalised recommendations