Abstract
We address automated testing and interactive proving of properties involving complex data structures with constraints, like the ones studied in enumerative combinatorics, e.g., permutations and maps. In this paper we show testing techniques to check properties of custom data generators for these structures. We focus on random property-based testing and bounded exhaustive testing, to find counterexamples for false conjectures in the Coq proof assistant. For random testing we rely on the existing Coq plugin QuickChick and its toolbox to write random generators. For bounded exhaustive testing, we use logic programming to generate all the data up to a given size. We also propose an extension of QuickChick with bounded exhaustive testing based on generators developed inside Coq, but also on correct-by-construction generators developed with Why3. These tools are applied to an original Coq formalization of the combinatorial structures of permutations and rooted maps, together with some operations on them and properties about them. Recursive generators are defined for each combinatorial family. They are used for debugging properties which are finally proved in Coq. This large case study is also a contribution in enumerative combinatorics.
Similar content being viewed by others
References
Baril J-L (2007) Gray code for permutations with a fixed number of cycles. Discrete Math 307(13): 1559–1571
Bertot Y, Castéran P (2004) Interactive theorem proving and program development. Coq’Art: the calculus of inductive constructions. Texts in theoretical computer science. Springer, New York
Bowles J, Caminati MB (2017) A verified algorithm enumerating event structures. In: Intelligent Computer Mathematics, volume 10383 of LNCS (LNAI). Springer, pp 239–254
François B, Sylvain C, Evelyne C, Stéphane L (2008) Implementing polymorphism in SMT solvers. In: SMT ’08/BPR ’08: proceedings of the joint workshops of the 6th international workshop on satisfiability modulo theories and 1st international workshop on bit-precise reasoning. ACM, New York, pp 1–5
Brun C, Dufourd J-F, Magaud N (2012) Designing and proving correct a convex hull algorithm with hypermaps in Coq. Comput Geom 45(8): 436–457
Bereg S, Levy A, Sudborough IH (2018) Constructing permutation arrays from groups. Des Codes Cryptogr 86(5): 1095–1111
Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. In: Software engineering and formal methods (SEFM 2004). IEEE Computer Society, pp 230–239
Blanchette JC, Nipkow T (2010) Nitpick: a counterexample generator for higher-order logic based on a relational model finder. In: ITP 2010, volume 6172 of LNCS. Springer, Heidelberg, pp 131–146
Bulwahn L (2012) The new quickcheck for Isabelle—random, exhaustive and symbolic testing under one roof. In: CPP 2012, volume 7679 of LNCS. Springer, Heidelberg, pp 92–108
Baril J-L, Vajnovszki V (2017) A permutation code preserving a double Eulerian bistatistic. Discrete Appl Math 224: 9–15
Cruanes S, Blanchette JC (2016) Extending Nunchaku to dependent type theory. In: Proceedings first international workshop on hammers for type theories, HaTT@IJCAR 2016, Coimbra, Portugal, July 1, 2016, volume 210 of EPTCS, pp 3–12
Carlier M, Dubois C, Gotlieb A (2010) Constraint reasoning in FOCALTEST. In: Proceedings of the 5th International Conference on Software and Data Technologies - Volume 2: ICSOFT. SciTePress, pp 82–91
Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the fifth ACM SIGPLAN international conference on functional programming, volume 35 of SIGPLAN notices. ACM, New York, pp 268–279
Cohen C (2013) Pragmatic quotient types in Coq. In: ITP 2013, volume 7998 of LNCS. Springer, Berlin, pp 213–228
The Coq Development Team (2017) The Coq Proof Assistant Reference Manual. http://coq.inria.fr/. Version 8.7
Cruanes S (2017) Satisfiability modulo bounded checking. In: Automated deduction–CADE 26, volume 10395 of LNCS. Springer, pp 114–129
Dubois C, Giorgetti A, Genestier R (2016) Tests and proofs for enumerative combinatorics. In: Tests and proofs (TAP), volume 6792 of LNCS. Springer, pp 57–75
Dybjer P, Haiyan Q, Takeyama M (2003) Combining testing and proving in dependent type theory. In: TPHOLs 2003, volume 2758 of LNCS. Springer, Heidelberg, pp 188–203
Duregård J, Jansson P, Wang M (2012) Feat: functional enumeration of algebraic types. In: Proceedings of the 2012 Haskell Symposium, volume 47 of SIGPLAN Notices. ACM, New York, pp 61–72
Dubois C, Mota J-M (2007) Geometric modeling with B: formal specification of generalized maps.. J Sci Pract Comput 1(2): 9–24
Dufourd J-F (2007) Design and formal proof of a new optimal image segmentation program with hypermaps. Pattern Recogn 40(11): 2974–2993
Dufourd J-F (2008) Polyhedra genus theorem and Euler formula: a hypermap-formalized intuitionistic proof. Theor Comput Sci 403(2–3): 133–159
Dufourd J-F (2009) An intuitionistic proof of a discrete form of the Jordan curve theorem formalized in Coq with combinatorial hypermaps.. J Autom Reason 43(1): 19–51
Dumont D, Viennot G (1980) A combinatorial interpretation of the Seidel generation of Genocchi numbers. In: Srivastava J (ed) Combinatorial mathematics, optimal designs and their applications, volume 6 of annals of discrete mathematics. Elsevier, pp 77–87
Eynard B (2011) Formal matrix integrals and combinatorics of maps. Springer, New York, pp 415–442
Filliâtre J-C, Paskevich A (2013) Why3—where programs meet provers. In: Proceedings of the 22nd European symposium on programming, volume 7792 of LNCS. Springer, pp 125–128
Filliâtre J-C, Pereira M (2016) A modular way to reason about iteration. In: 8th NASA formal methods symposium, volume 9690 of LNCS. Springer, pp 322–336
Genestier R, Giorgetti A, Petiot G (2015) Sequential generation of structured arrays and its deductive verification. In: Tests and proofs (TAP) 2015, volume 9154 of LNCS. Springer, Heidelberg, pp 109–128
Gonthier G (2005) A computer checked proof of the Four Colour Theorem. http://research.microsoft.com/gonthier/4colproof.pdf
Gonthier G (2008) The four colour theorem: engineering of a formal proof. In: ASCM 2007, volume 5081 of LNCS (LNAI). Springer, Heidelberg, pp 333–333
Giorgetti A, Senni V (2012) Specification and validation of algorithms generating planar Lehman words. GASCom’12. https://hal.inria.fr/hal-00753008
Hriţcu C, Lampropoulos L, Dénès M, Paraskevopoulou Z (2018) QuickChick: randomized property-based testing plugin for Coq. https://github.com/QuickChick/QuickChick
Kitaev S (2011) Patterns in permutations and words. Springer, New York
Lazarus F (2014) Combinatorial graphs and surfaces from the computational and topological viewpoint followed by some notes on the isometric embedding of the square flat torus. http://www.gipsa-lab.grenoble-inp.fr/~francis.lazarus/Documents/hdr-Lazarus.pdf.
Lehmer DH (1960) Teaching combinatorial tricks to a computer. In: Proceedings of symposia in applied mathematics combinatorial analysis. American Mathematical Society, vol 10, pp 179–193
Lampropoulos L, Gallois-Wong D, Hriţcu C, Hughes J, Pierce BC, Xia L (2017) Beginner’s luck: a language for property-based generators. In: Proceedings of the 44th ACM SIGPLAN symposium on principles of programming languages, POPL 2017, Paris, France, January 18–20, 2017. ACM, pp 114–129
Lindblad F (2007) Property directed generation of first-order test data. In: Proceedings of the Eighth Symposium on Trends in Functional Programming, TFP 2007, New York City, New York, USA, April 2–4, 2007, volume 8 of Trends in Functional Programming. Intellect, pp 105–123
Lampropoulos L, Paraskevopoulou Z, Pierce BC (2018) Generating good generators for inductive relations. PACMPL 2(POPL):45:1–45:30
Lando SK, Zvonkin AK (2004) Graphs on surfaces and their applications. Springer
Mathematical Components Team (2018) Mathematical components library. http://math-comp.github.io/math-comp/
Mednykh A, Nedela R (2017) Recent progress in enumeration of hypermaps.. J Math Sci 226(5): 635–654
Mantaci R, Rakotondrajao F (2001) A permutations representation that knows what “Eulerian” means. Discrete Math Theor Comput Sci 4(2): 101–108
The OEIS Foundation Inc. The on-line encyclopedia of integer sequences. https://oeis.org/A000698
Owre S (2006) Random testing in PVS. Workshop on Automated Formal Methods (AFM). http://fm.csl.sri.com/AFM06/papers/5-Owre.pdf
Palka MH, Claessen K, Russo A, Hughes J (2011) Testing an optimising compiler by generating random lambda terms. In: Proceedings of the 6th international workshop on automation of software test, AST 2011, Waikiki, Honolulu, HI, USA, May 23–24, 2011. ACM, pp 91–97
Planat M, Giorgetti A, Holweck F, Saniga M (2015) Quantum contextual finite geometries from dessins d’enfants. Int J Geom Methods Mod Phys 12: 1–17
Paraskevopoulou Z, Hriţcu C, Dénès M, Lampropoulos L, Pierce BC (2015) Foundational property-based testing. In: ITP 2015, volume 9236 of LNCS. Springer, Heidelberg, pp 325–343
Runciman C, Naylor M, Lindblad F (2008) SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. In: Proceedings of the 1st ACM SIGPLAN symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008, pp 37–48
Senni V (2018) Validation library. https://subversion.assembla.com/svn/validation/
Stanley RP (1997) Enumerative combinatorics, vol 1. Cambridge University Press, Cambridge
Seidel EL, Vazou N, Jhala R (2015) Type targeted testing. In: ESOP 2015, volume 9032 of LNCS. Springer, Heidelberg, pp 812–836
SWI (2018) Prolog. http://www.swi-prolog.org/
Tarau P (2015) On type-directed generation of lambda terms. In: Proceedings of the technical communications of the 31st international conference on logic programming (ICLP 2015), Cork, Ireland, August 31–September 4, 2015, volume 1433 of CEUR Workshop Proceedings. CEUR-WS.org
Tutte WT (1973) What is a map? In: New directions in the theory of graphs: proceedings. Academic Press, New York, pp 309–325
Tutte WT (1979) Combinatorial oriented maps.. Can J Math 31(5): 986–1004
Vajnovszki V (2011) A new Euler–Mahonian constructive bijection.. Discrete Appl Math 159(14): 1453–1459
Vajnovszki V (2013) Lehmer code transforms and Mahonian statistics on permutations.. Discrete Math, 313(5): 581–589
Walsh TRS, Lehman AB (1972) Counting rooted maps by genus I.. J Comb Theory Ser B 13: 192–218
Acknowledgements
For this work Alain Giorgetti was supported by the French ``Investissements d'Avenir'' program, project ISITEBFC (contract ANR-15-IDEX-03). The authors warmly thank Nicolas Magaud for help with Coq, Valerio Senni for advice about his validation library, Noam Zeilberger and Cyril Cohen for fruitful discussions. We deeply thank Leonidas Lampropoulos for his support on QuickChick. We also thank anonymous referees for their helpful suggestions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Bernhard Aichernig, Marie-Claude Gaudel, Carlo Furia, and Rob Hierons
Rights and permissions
About this article
Cite this article
Dubois, C., Giorgetti, A. Tests and proofs for custom data generators. Form Asp Comp 30, 659–684 (2018). https://doi.org/10.1007/s00165-018-0459-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00165-018-0459-1