Formal Aspects of Computing

, Volume 30, Issue 6, pp 659–684 | Cite as

Tests and proofs for custom data generators

  • Catherine Dubois
  • Alain GiorgettiEmail author
Original Article


We address automated testing and interactive proving of properties involving complex data structures with constraints, like the ones studied in enumerative combinatorics, e.g., permutations and maps. In this paper we show testing techniques to check properties of custom data generators for these structures. We focus on random property-based testing and bounded exhaustive testing, to find counterexamples for false conjectures in the Coq proof assistant. For random testing we rely on the existing Coq plugin QuickChick and its toolbox to write random generators. For bounded exhaustive testing, we use logic programming to generate all the data up to a given size. We also propose an extension of QuickChick with bounded exhaustive testing based on generators developed inside Coq, but also on correct-by-construction generators developed with Why3. These tools are applied to an original Coq formalization of the combinatorial structures of permutations and rooted maps, together with some operations on them and properties about them. Recursive generators are defined for each combinatorial family. They are used for debugging properties which are finally proved in Coq. This large case study is also a contribution in enumerative combinatorics.


Interactive theorem proving Random testing Bounded-exhaustive testing Logic programming Combinatorial enumeration Permutations Rooted maps 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



For this work Alain Giorgetti was supported by the French ``Investissements d'Avenir'' program, project ISITEBFC (contract ANR-15-IDEX-03). The authors warmly thank Nicolas Magaud for help with Coq, Valerio Senni for advice about his validation library, Noam Zeilberger and Cyril Cohen for fruitful discussions. We deeply thank Leonidas Lampropoulos for his support on QuickChick. We also thank anonymous referees for their helpful suggestions.


  1. Bar07.
    Baril J-L (2007) Gray code for permutations with a fixed number of cycles. Discrete Math 307(13): 1559–1571MathSciNetCrossRefGoogle Scholar
  2. BC04.
    Bertot Y, Castéran P (2004) Interactive theorem proving and program development. Coq’Art: the calculus of inductive constructions. Texts in theoretical computer science. Springer, New YorkGoogle Scholar
  3. BC17.
    Bowles J, Caminati MB (2017) A verified algorithm enumerating event structures. In: Intelligent Computer Mathematics, volume 10383 of LNCS (LNAI). Springer, pp 239–254Google Scholar
  4. BCCL08.
    François B, Sylvain C, Evelyne C, Stéphane L (2008) Implementing polymorphism in SMT solvers. In: SMT ’08/BPR ’08: proceedings of the joint workshops of the 6th international workshop on satisfiability modulo theories and 1st international workshop on bit-precise reasoning. ACM, New York, pp 1–5Google Scholar
  5. BDM12.
    Brun C, Dufourd J-F, Magaud N (2012) Designing and proving correct a convex hull algorithm with hypermaps in Coq. Comput Geom 45(8): 436–457MathSciNetCrossRefGoogle Scholar
  6. BLS18.
    Bereg S, Levy A, Sudborough IH (2018) Constructing permutation arrays from groups. Des Codes Cryptogr 86(5): 1095–1111MathSciNetCrossRefGoogle Scholar
  7. BN04.
    Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. In: Software engineering and formal methods (SEFM 2004). IEEE Computer Society, pp 230–239Google Scholar
  8. BN10.
    Blanchette JC, Nipkow T (2010) Nitpick: a counterexample generator for higher-order logic based on a relational model finder. In: ITP 2010, volume 6172 of LNCS. Springer, Heidelberg, pp 131–146CrossRefGoogle Scholar
  9. Bul12.
    Bulwahn L (2012) The new quickcheck for Isabelle—random, exhaustive and symbolic testing under one roof. In: CPP 2012, volume 7679 of LNCS. Springer, Heidelberg, pp 92–108CrossRefGoogle Scholar
  10. BV17.
    Baril J-L, Vajnovszki V (2017) A permutation code preserving a double Eulerian bistatistic. Discrete Appl Math 224: 9–15MathSciNetCrossRefGoogle Scholar
  11. CB16.
    Cruanes S, Blanchette JC (2016) Extending Nunchaku to dependent type theory. In: Proceedings first international workshop on hammers for type theories, HaTT@IJCAR 2016, Coimbra, Portugal, July 1, 2016, volume 210 of EPTCS, pp 3–12MathSciNetCrossRefGoogle Scholar
  12. CDG10.
    Carlier M, Dubois C, Gotlieb A (2010) Constraint reasoning in FOCALTEST. In: Proceedings of the 5th International Conference on Software and Data Technologies - Volume 2: ICSOFT. SciTePress, pp 82–91Google Scholar
  13. CH00.
    Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the fifth ACM SIGPLAN international conference on functional programming, volume 35 of SIGPLAN notices. ACM, New York, pp 268–279CrossRefGoogle Scholar
  14. Coh13.
    Cohen C (2013) Pragmatic quotient types in Coq. In: ITP 2013, volume 7998 of LNCS. Springer, Berlin, pp 213–228CrossRefGoogle Scholar
  15. Coq17.
    The Coq Development Team (2017) The Coq Proof Assistant Reference Manual. Version 8.7
  16. Cru17.
    Cruanes S (2017) Satisfiability modulo bounded checking. In: Automated deduction–CADE 26, volume 10395 of LNCS. Springer, pp 114–129Google Scholar
  17. DGG16.
    Dubois C, Giorgetti A, Genestier R (2016) Tests and proofs for enumerative combinatorics. In: Tests and proofs (TAP), volume 6792 of LNCS. Springer, pp 57–75Google Scholar
  18. DHT03.
    Dybjer P, Haiyan Q, Takeyama M (2003) Combining testing and proving in dependent type theory. In: TPHOLs 2003, volume 2758 of LNCS. Springer, Heidelberg, pp 188–203Google Scholar
  19. DJW12.
    Duregård J, Jansson P, Wang M (2012) Feat: functional enumeration of algebraic types. In: Proceedings of the 2012 Haskell Symposium, volume 47 of SIGPLAN Notices. ACM, New York, pp 61–72CrossRefGoogle Scholar
  20. DM07.
    Dubois C, Mota J-M (2007) Geometric modeling with B: formal specification of generalized maps.. J Sci Pract Comput 1(2): 9–24Google Scholar
  21. Duf07.
    Dufourd J-F (2007) Design and formal proof of a new optimal image segmentation program with hypermaps. Pattern Recogn 40(11): 2974–2993CrossRefGoogle Scholar
  22. Duf08.
    Dufourd J-F (2008) Polyhedra genus theorem and Euler formula: a hypermap-formalized intuitionistic proof. Theor Comput Sci 403(2–3): 133–159MathSciNetCrossRefGoogle Scholar
  23. Duf09.
    Dufourd J-F (2009) An intuitionistic proof of a discrete form of the Jordan curve theorem formalized in Coq with combinatorial hypermaps.. J Autom Reason 43(1): 19–51MathSciNetCrossRefGoogle Scholar
  24. DV80.
    Dumont D, Viennot G (1980) A combinatorial interpretation of the Seidel generation of Genocchi numbers. In: Srivastava J (ed) Combinatorial mathematics, optimal designs and their applications, volume 6 of annals of discrete mathematics. Elsevier, pp 77–87Google Scholar
  25. Eyn11.
    Eynard B (2011) Formal matrix integrals and combinatorics of maps. Springer, New York, pp 415–442CrossRefGoogle Scholar
  26. FP13.
    Filliâtre J-C, Paskevich A (2013) Why3—where programs meet provers. In: Proceedings of the 22nd European symposium on programming, volume 7792 of LNCS. Springer, pp 125–128Google Scholar
  27. FP16.
    Filliâtre J-C, Pereira M (2016) A modular way to reason about iteration. In: 8th NASA formal methods symposium, volume 9690 of LNCS. Springer, pp 322–336Google Scholar
  28. GGP15.
    Genestier R, Giorgetti A, Petiot G (2015) Sequential generation of structured arrays and its deductive verification. In: Tests and proofs (TAP) 2015, volume 9154 of LNCS. Springer, Heidelberg, pp 109–128CrossRefGoogle Scholar
  29. Gon05.
    Gonthier G (2005) A computer checked proof of the Four Colour Theorem.
  30. Gon08.
    Gonthier G (2008) The four colour theorem: engineering of a formal proof. In: ASCM 2007, volume 5081 of LNCS (LNAI). Springer, Heidelberg, pp 333–333Google Scholar
  31. GS12.
    Giorgetti A, Senni V (2012) Specification and validation of algorithms generating planar Lehman words. GASCom’12.
  32. HLDP18.
    Hriţcu C, Lampropoulos L, Dénès M, Paraskevopoulou Z (2018) QuickChick: randomized property-based testing plugin for Coq.
  33. Kit11.
    Kitaev S (2011) Patterns in permutations and words. Springer, New YorkCrossRefGoogle Scholar
  34. Laz14.
    Lazarus F (2014) Combinatorial graphs and surfaces from the computational and topological viewpoint followed by some notes on the isometric embedding of the square flat torus.
  35. Leh60.
    Lehmer DH (1960) Teaching combinatorial tricks to a computer. In: Proceedings of symposia in applied mathematics combinatorial analysis. American Mathematical Society, vol 10, pp 179–193Google Scholar
  36. LGH+17.
    Lampropoulos L, Gallois-Wong D, Hriţcu C, Hughes J, Pierce BC, Xia L (2017) Beginner’s luck: a language for property-based generators. In: Proceedings of the 44th ACM SIGPLAN symposium on principles of programming languages, POPL 2017, Paris, France, January 18–20, 2017. ACM, pp 114–129Google Scholar
  37. Lin07.
    Lindblad F (2007) Property directed generation of first-order test data. In: Proceedings of the Eighth Symposium on Trends in Functional Programming, TFP 2007, New York City, New York, USA, April 2–4, 2007, volume 8 of Trends in Functional Programming. Intellect, pp 105–123Google Scholar
  38. LPP18.
    Lampropoulos L, Paraskevopoulou Z, Pierce BC (2018) Generating good generators for inductive relations. PACMPL 2(POPL):45:1–45:30CrossRefGoogle Scholar
  39. LZ04.
    Lando SK, Zvonkin AK (2004) Graphs on surfaces and their applications. SpringerGoogle Scholar
  40. Mat18.
    Mathematical Components Team (2018) Mathematical components library.
  41. MN17.
    Mednykh A, Nedela R (2017) Recent progress in enumeration of hypermaps.. J Math Sci 226(5): 635–654CrossRefGoogle Scholar
  42. MR01.
    Mantaci R, Rakotondrajao F (2001) A permutations representation that knows what “Eulerian” means. Discrete Math Theor Comput Sci 4(2): 101–108MathSciNetzbMATHGoogle Scholar
  43. OEIS.
    The OEIS Foundation Inc. The on-line encyclopedia of integer sequences.
  44. Owr06.
    Owre S (2006) Random testing in PVS. Workshop on Automated Formal Methods (AFM).
  45. PCRH11.
    Palka MH, Claessen K, Russo A, Hughes J (2011) Testing an optimising compiler by generating random lambda terms. In: Proceedings of the 6th international workshop on automation of software test, AST 2011, Waikiki, Honolulu, HI, USA, May 23–24, 2011. ACM, pp 91–97Google Scholar
  46. PGHS15.
    Planat M, Giorgetti A, Holweck F, Saniga M (2015) Quantum contextual finite geometries from dessins d’enfants. Int J Geom Methods Mod Phys 12: 1–17MathSciNetCrossRefGoogle Scholar
  47. PHD+15.
    Paraskevopoulou Z, Hriţcu C, Dénès M, Lampropoulos L, Pierce BC (2015) Foundational property-based testing. In: ITP 2015, volume 9236 of LNCS. Springer, Heidelberg, pp 325–343Google Scholar
  48. RNL08.
    Runciman C, Naylor M, Lindblad F (2008) SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. In: Proceedings of the 1st ACM SIGPLAN symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008, pp 37–48Google Scholar
  49. Sen18.
    Senni V (2018) Validation library. Scholar
  50. Sta97.
    Stanley RP (1997) Enumerative combinatorics, vol 1. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  51. SVJ15.
    Seidel EL, Vazou N, Jhala R (2015) Type targeted testing. In: ESOP 2015, volume 9032 of LNCS. Springer, Heidelberg, pp 812–836CrossRefGoogle Scholar
  52. SWI18.
    SWI (2018) Prolog.
  53. Tar15.
    Tarau P (2015) On type-directed generation of lambda terms. In: Proceedings of the technical communications of the 31st international conference on logic programming (ICLP 2015), Cork, Ireland, August 31–September 4, 2015, volume 1433 of CEUR Workshop Proceedings. CEUR-WS.orgGoogle Scholar
  54. Tut73.
    Tutte WT (1973) What is a map? In: New directions in the theory of graphs: proceedings. Academic Press, New York, pp 309–325Google Scholar
  55. Tut79.
    Tutte WT (1979) Combinatorial oriented maps.. Can J Math 31(5): 986–1004MathSciNetCrossRefGoogle Scholar
  56. Vaj11.
    Vajnovszki V (2011) A new Euler–Mahonian constructive bijection.. Discrete Appl Math 159(14): 1453–1459MathSciNetCrossRefGoogle Scholar
  57. Vaj13.
    Vajnovszki V (2013) Lehmer code transforms and Mahonian statistics on permutations.. Discrete Math, 313(5): 581–589MathSciNetCrossRefGoogle Scholar
  58. WL72.
    Walsh TRS, Lehman AB (1972) Counting rooted maps by genus I.. J Comb Theory Ser B 13: 192–218MathSciNetCrossRefGoogle Scholar

Copyright information

© British Computer Society 2018

Authors and Affiliations

  1. 1.Samovar, ENSIIE, CNRSÉvryFrance
  2. 2.FEMTO-ST InstituteUniversity of Bourgogne Franche-Comté, CNRSBesançonFrance

Personalised recommendations