The first reactive synthesis competition (SYNTCOMP 2014)
- 305 Downloads
- 9 Citations
Abstract
We introduce the reactive synthesis competition (SYNTCOMP), a long-term effort intended to stimulate and guide advances in the design and application of synthesis procedures for reactive systems. The first iteration of SYNTCOMP is based on the controller synthesis problem for finite-state systems and safety specifications. We provide an overview of this problem and existing approaches to solve it, and report on the design and results of the first SYNTCOMP. This includes the definition of the benchmark format, the collection of benchmarks, the rules of the competition, and the five synthesis tools that participated. We present and analyze the results of the competition and draw conclusions on the state of the art. Finally, we give an outlook on future directions of SYNTCOMP.
Keywords
Synthesis Reactive systems Competition Experimental evaluation Benchmarks Safety gamesNotes
Acknowledgments
We thank the anonymous reviewers for their detailed and insightful comments on drafts of this article. We thank Armin Biere for his advice on running a competition, and Ayrat Khalimov for supplying the reference implementation Aisy for the competition. The organization of SYNTCOMP 2014 was supported by the Austrian Science Fund (FWF) through projects RiSE (S11406-N23) and QUAINT (I774-N23), by the German Research Foundation (DFG) as part of the Transregional Collaborative Research Center “Automatic Verification and Analysis of Complex Systems” (SFB/TR 14 AVACS) and through project “Automatic Synthesis of Distributed and Parameterized Systems” (JA 2357/2-1), as well as by the Institutional Strategy of the University of Bremen, funded by the German Excellence Initiative. The development of AbsSynthe was supported by an F.R.S.-FNRS fellowship, and the ERC inVEST (279499) project. The development of Basil was supported by the Institutional Strategy of the University of Bremen, funded by the German Excellence Initiative. The development of \(\texttt {Demiurge}\) was supported by the FWF through projects RiSE (S11406-N23, S11408-N23) and QUAINT (I774-N23). The development of Realizer was supported by the DFG as part of SFB/TR 14 AVACS. The development of Simple BDD Solver was supported by a gift from the Intel Corporation, and NICTA is funded by the Australian Government through the Department of Communications and the Australian Research Council through the ICT Centre of Excellence Program.
References
- 1.Alur, R., Bodík, R., Dallal, E., Fisman, D., Garg, P., Juniwal, G., Kress-Gazit, H., Madhusudan, P., Martin, M.M.K., Raghothaman, M., Saha, S., Seshia, S.A., Singh, R., Solar-Lezama, A., Torlak, E., Udupa, A.: Syntax-guided synthesis. In: Dependable Software Systems Engineering, NATO Science for Peace and Security Series, D: Information and Communication Security, vol. 40, pp. 1–25. IOS Press (2015)Google Scholar
- 2.Alur, R., Madhusudan, P., Nam, W.: Symbolic computational techniques for solving games. STTT 7(2), 118–128 (2005)CrossRefzbMATHGoogle Scholar
- 3.Aziz, A., Tasiran, S., Brayton, R.K.: BDD variable ordering for interacting finite state machines. In: DAC, pp. 283–288 (1994)Google Scholar
- 4.Balint, A., Diepold, D., Gall, D., Gerber, S., Kapler, G., Retz, R.: EDACC - an advanced platform for the experiment design, administration and analysis of empirical algorithms. In: LION 5. Selected Papers, LNCS, vol. 6683, pp. 586–599. Springer, (2011)Google Scholar
- 5.Barrett, C.W., de Moura, L.M., Stump, A.: Design and results of the first satisfiability modulo theories competition (SMT-COMP 2005). J. Autom. Reason. 35(4), 373–390 (2005)CrossRefzbMATHGoogle Scholar
- 6.Beyer, D.: Competition on software verification - (SV-COMP). In: TACAS, LNCS, vol. 7214, pp. 504–524. Springer (2012)Google Scholar
- 7.Beyer, D., Löwe, S., Wendler, P.: Benchmarking and resource measurement. In: SPIN 2015, LNCS, vol. 9232, pp. 160–178. Springer (2015)Google Scholar
- 8.Bloem, R., Cimatti, A., Greimel, K., Hofferek, G., Könighofer, R., Roveri, M., Schuppan, V., Seeber, R.: RATSY - A new requirements analysis tool with synthesis. In: CAV, LNCS, vol. 6174, pp. 425–429. Springer (2010)Google Scholar
- 9.Bloem, R., Egly, U., Klampfl, P., Könighofer, R., Lonsing, F.: SAT-based methods for circuit synthesis. In: FMCAD’14, pp. 31–34. IEEE (2014)Google Scholar
- 10.Bloem, R., Galler, S.J., Jobstmann, B., Piterman, N., Pnueli, A., Weiglhofer, M.: Automatic hardware synthesis from specifications: a case study. In: DATE, pp. 1188–1193. ACM (2007)Google Scholar
- 11.Bloem, R., Galler, S.J., Jobstmann, B., Piterman, N., Pnueli, A., Weiglhofer, M.: Specify, compile, run: hardware from PSL. Electr. Notes Theor. Comput. Sci. 190(4), 3–16 (2007)CrossRefGoogle Scholar
- 12.Bloem, R., Jobstmann, B., Piterman, N., Pnueli, A., Sa’ar, Y.: Synthesis of reactive(1) designs. J. Comput. Syst. Sci. 78(3), 911–938 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
- 13.Bloem, R., Könighofer, R., Seidl, M.: SAT-based synthesis methods for safety specs. In: VMCAI, LNCS, vol. 8318, pp. 1–20. Springer (2014)Google Scholar
- 14.Bohy, A., Bruyère, V., Filiot, E., Jin, N., Raskin, J.-F.: Acacia+, a tool for LTL synthesis. In: CAV, LNCS, vol. 7358, pp. 652–657. Springer (2012)Google Scholar
- 15.Bradley, A.R.: SAT-based model checking without unrolling. In: VMCAI, LNCS, vol. 6538, pp. 70–87. Springer (2011)Google Scholar
- 16.Brayton, R.K., Hachtel, G.D., Sangiovanni-Vincentelli, A.L., Somenzi, F., Aziz, A., Cheng, S., Edwards, S.A., Khatri, S.P., Kukimoto, Y., Pardo, A., Qadeer, S., Ranjan, R.K., Sarwary, S., Shiple, T.R., Swamy, G., Villa, T.: VIS: a system for verification and synthesis. In: CAV, LNCS, vol. 1102, pp. 428–432. Springer (1996)Google Scholar
- 17.Brayton, R.K., Mishchenko, A.: ABC: an academic industrial-strength verification tool. In: CAV, LNCS, vol. 6174, pp. 24–40. Springer (2010)Google Scholar
- 18.Brenguier, R., Pérez, G.A., Raskin, J.-F., Sankur, O.: AbsSynthe: abstract synthesis from succinct safety specifications. In: SYNT, EPTCS, vol. 157, pp. 100–116. Open Publishing Association (2014)Google Scholar
- 19.Bryant, R.E.: Graph-based algorithms for boolean function manipulation. IEEE Trans. Comput. 35(8), 677–691 (1986)CrossRefzbMATHGoogle Scholar
- 20.Büchi, J., Landweber, L.: Solving sequential conditions by finite-state strategies. Trans. Am. Math. Soc. 138, 295–311 (1969)CrossRefzbMATHGoogle Scholar
- 21.Burch, J.R., Clarke, E.M., Long, D.E.: Symbolic model checking with partitioned transistion relations. In: VLSI, pp. 49–58 (1991)Google Scholar
- 22.Chiang, T., Jiang. J.R.: Property-directed synthesis of reactive systems from safety specifications. In: ICCAD, pp. 794–801. ACM (2015)Google Scholar
- 23.Church, A.: Logic, arithmetic and automata. In: Proceedings of the International Congress of Mathematicians, pp. 23–35 (1962)Google Scholar
- 24.Coudert, O., Madre, J.C.: A unified framework for the formal verification of sequential circuits. In: ICCAD, pp. 126–129 (1990)Google Scholar
- 25.Cousot, P., Cousot, R.: Abstract interpretation: A unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: POPL, pp. 238–252. ACM (1977)Google Scholar
- 26.de Alfaro, L., Roy, P.: Solving games via three-valued abstraction refinement. In: CONCUR, LNCS, vol. 4703, pp. 74–89. Springer (2007)Google Scholar
- 27.Ehlers, R.: Experimental aspects of synthesis. In: iWIGP, EPTCS, vol. 50, pp. 1–16 (2011)Google Scholar
- 28.Ehlers, R.: Unbeast: symbolic bounded synthesis. In: TACAS, LNCS, vol. 6605, pp. 272–275. Springer (2011)Google Scholar
- 29.Ehlers, R.: Symbolic bounded synthesis. Formal Methods Syst. Des. 40(2), 232–262 (2012)CrossRefzbMATHGoogle Scholar
- 30.Ehlers, R., Könighofer, R., Hofferek, G.: Symbolically synthesizing small circuits. In: FMCAD’12, pp. 91–100. IEEE (2012)Google Scholar
- 31.Emerson, E.A., Clarke, E.M.: Using branching time temporal logic to synthesize synchronization skeletons. Sci. Comput. Program. 2(3), 241–266 (1982)CrossRefzbMATHGoogle Scholar
- 32.Filiot, E., Jin, N., Raskin, J.: Exploiting structure in LTL synthesis. STTT 15(5–6), 541–561 (2013)CrossRefGoogle Scholar
- 33.Filiot, E., Jin, N., Raskin, J.-F.: Antichains and compositional algorithms for LTL synthesis. Formal Methods Syst. Des. 39(3), 261–296 (2011)CrossRefzbMATHGoogle Scholar
- 34.Finkbeiner, B., Jacobs, S.: Lazy synthesis. In: VMCAI, LNCS, vol. 7148, pp. 219–234. Springer (2012)Google Scholar
- 35.Finkbeiner, B., Schewe, S.: Bounded synthesis. STTT 15(5–6), 519–539 (2013)CrossRefzbMATHGoogle Scholar
- 36.Graf, S., Saïdi, H.: Construction of abstract state graphs with PVS. In: CAV, LNCS, vol. 1254, pp. 72–83. Springer (1997)Google Scholar
- 37.Henzinger, T.A., Jhala, R., Majumdar, R.: Counterexample-guided control. In: ICALP, LNCS, vol. 2719, pp. 886–902, Springer (2003)Google Scholar
- 38.Hong, Y., Beerel, P.A., Burch, J.R., McMillan, K.L.: Sibling-substitution-based BDD minimization using don’t cares. IEEE Trans. CAD of Integr. Circuits Syst. 19(1), 44–55 (2000)CrossRefGoogle Scholar
- 39.Jacobs, S.: Extended AIGER format for synthesis. CoRR (2014). arXiv:1405.5793. Accessed Feb 2016
- 40.Jacobs, S., Bloem, R., Brenguier, R., Ehlers, R., Hell, T., Könighofer, R., Pérez, G.A., Raskin, J., Ryzhyk, L., Sankur, O., Seidl, M., Tentrup, L., Walker, A.: The first reactive synthesis competition (SYNTCOMP 2014). CoRR (2015). arXiv:1506.08726. Accessed Feb 2016
- 41.Jacobs, S., Bloem, R., Brenguier, R., Könighofer, R., Pérez, G.A., Raskin, J.-F., Ryzhyk, L., Sankur, O., Seidl, M., Tentrup, L., Walker, A.: The second reactive synthesis competition (SYNTCOMP 2015). In: SYNT, EPTCS, vol. 202, pp. 27–57. Open Publishing Association (2016)Google Scholar
- 42.Jacobs, S., Klein, F.: A high-level LTL synthesis format: TLSF v1.0. CoRR (2016). arXiv:1601.05228. Accessed Feb 2016
- 43.Järvisalo, M., Berre, D.L., Roussel, O., Simon, L.: The international SAT solver competitions. AI Mag 33(1), 89–94 (2012)Google Scholar
- 44.Jobstmann, B., Bloem, R.: Optimizations for LTL synthesis. In: FMCAD, pp. 117–124. IEEE Computer Society (2006)Google Scholar
- 45.Jobstmann, B., Galler, S.J., Weiglhofer, M., Bloem, R.: Anzu: A tool for property synthesis. In: CAV, LNCS, vol. 4590, pp. 258–262. Springer (2006)Google Scholar
- 46.Kupferman, O., Vardi, M.Y.: Safraless decision procedures. In: FOCS, pp. 531–542. IEEE Computer Society (2005)Google Scholar
- 47.Kurshan, R.P.: Automata-theoretic verification of coordinating processes. In: Analysis and Optimization of Systems: Discrete Event Systems, pp. 16–28. Springer (1994)Google Scholar
- 48.Lecoutre, C., Roussel, O., van Dongen, M.R.C.: Promoting robust black-box solvers through competitions. Constraints 15(3), 317–326 (2010)CrossRefzbMATHGoogle Scholar
- 49.Mishchenko, A., Chatterjee, S., Brayton, R.K.: Dag-aware AIG rewriting a fresh look at combinational logic synthesis. In: DAC, pp. 532–535. ACM (2006)Google Scholar
- 50.Mishchenko, A., Chatterjee, S., Jiang, R., Brayton, R.: FRAIGs: A unifying representation for logic synthesis and verification. Technical report, EECS Department, U. C. Berkeley (2005)Google Scholar
- 51.Morgenstern, A., Gesell, M., Schneider, K.: Solving games using incremental induction. In: IFM’13, LNCS 7940, pp. 177–191. Springer (2013)Google Scholar
- 52.Niemetz, A., Preiner, M., Lonsing, F., Seidl, M., Biere, A.: Resolution-based certificate extraction for QBF. In: SAT’12, LNCS 7317, pp. 430–435. Springer (2012)Google Scholar
- 53.Pnueli, A., Rosner, R.: On the synthesis of a reactive module. In: POPL, pp. 179–190. ACM Press (1989)Google Scholar
- 54.Rabin, M.O.: Decidability of second-order theories and automata on infinite trees. Trans. Am. Math. Soc. 141, 1–35 (1969)zbMATHGoogle Scholar
- 55.Ranjan, R.K., Aziz, A., Brayton, R.K., Plessier, B., Pixley, C.: Efficient bdd algorithms for fsm synthesis and verification. In: International Workshop on Logic Synthesis (1995)Google Scholar
- 56.Roussel, O.: Controlling a solver execution with the runsolver tool. JSAT 7(4), 139–144 (2011)MathSciNetzbMATHGoogle Scholar
- 57.Rudell, R.: Dynamic variable ordering for ordered binary decision diagrams. In: ICCAD, pp. 42–47. IEEE Computer Society (1993)Google Scholar
- 58.Seidl, M., Könighofer, R.: Partial witnesses from preprocessed quantified boolean formulas. In: DATE’14, pp. 1–6. IEEE (2014)Google Scholar
- 59.Sohail, S., Somenzi, F.: Safety first: a two-stage algorithm for the synthesis of reactive systems. STTT 15(5–6), 433–454 (2013)CrossRefGoogle Scholar
- 60.Somenzi, F.: Binary decision diagrams. In: Calculational System Design, vol. 173, pp. 303. IOS Press (1999)Google Scholar
- 61.Sutcliffe, G., Suttner, C.B.: Evaluating general purpose automated theorem proving systems. Artif. Intell. 131(1–2), 39–54 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
- 62.Sutcliffe, G., Suttner, C.B.: The state of CASC. AI Commun. 19(1), 35–48 (2006)MathSciNetzbMATHGoogle Scholar
- 63.Thomas, W.: On the synthesis of strategies in infinite games. In: STACS, pp. 1–13 (1995)Google Scholar
- 64.van Dijk, T., Laarman, A., van de Pol, J.: Multi-core BDD operations for symbolic reachability. Electron. Notes Theor. Comput. Sci. 296, 127–143 (2013)CrossRefGoogle Scholar
- 65.van Dijk, T., van de Pol, J.: Sylvan: Multi-core decision diagrams. In: TACAS 2015, LNCS, vol. 9035, pp. 677–691 Springer (2015)Google Scholar