Skip to main content
SpringerLink
Log in

Tests and proofs for custom data generators

  • Original Article
  • Published:
Formal Aspects of Computing

Abstract

We address automated testing and interactive proving of properties involving complex data structures with constraints, like the ones studied in enumerative combinatorics, e.g., permutations and maps. In this paper we show testing techniques to check properties of custom data generators for these structures. We focus on random property-based testing and bounded exhaustive testing, to find counterexamples for false conjectures in the Coq proof assistant. For random testing we rely on the existing Coq plugin QuickChick and its toolbox to write random generators. For bounded exhaustive testing, we use logic programming to generate all the data up to a given size. We also propose an extension of QuickChick with bounded exhaustive testing based on generators developed inside Coq, but also on correct-by-construction generators developed with Why3. These tools are applied to an original Coq formalization of the combinatorial structures of permutations and rooted maps, together with some operations on them and properties about them. Recursive generators are defined for each combinatorial family. They are used for debugging properties which are finally proved in Coq. This large case study is also a contribution in enumerative combinatorics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baril J-L (2007) Gray code for permutations with a fixed number of cycles. Discrete Math 307(13): 1559–1571

    Article  MathSciNet  Google Scholar 

  2. Bertot Y, Castéran P (2004) Interactive theorem proving and program development. Coq’Art: the calculus of inductive constructions. Texts in theoretical computer science. Springer, New York

  3. Bowles J, Caminati MB (2017) A verified algorithm enumerating event structures. In: Intelligent Computer Mathematics, volume 10383 of LNCS (LNAI). Springer, pp 239–254

  4. François B, Sylvain C, Evelyne C, Stéphane L (2008) Implementing polymorphism in SMT solvers. In: SMT ’08/BPR ’08: proceedings of the joint workshops of the 6th international workshop on satisfiability modulo theories and 1st international workshop on bit-precise reasoning. ACM, New York, pp 1–5

  5. Brun C, Dufourd J-F, Magaud N (2012) Designing and proving correct a convex hull algorithm with hypermaps in Coq. Comput Geom 45(8): 436–457

    Article  MathSciNet  Google Scholar 

  6. Bereg S, Levy A, Sudborough IH (2018) Constructing permutation arrays from groups. Des Codes Cryptogr 86(5): 1095–1111

    Article  MathSciNet  Google Scholar 

  7. Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. In: Software engineering and formal methods (SEFM 2004). IEEE Computer Society, pp 230–239

  8. Blanchette JC, Nipkow T (2010) Nitpick: a counterexample generator for higher-order logic based on a relational model finder. In: ITP 2010, volume 6172 of LNCS. Springer, Heidelberg, pp 131–146

    Chapter  Google Scholar 

  9. Bulwahn L (2012) The new quickcheck for Isabelle—random, exhaustive and symbolic testing under one roof. In: CPP 2012, volume 7679 of LNCS. Springer, Heidelberg, pp 92–108

    Chapter  Google Scholar 

  10. Baril J-L, Vajnovszki V (2017) A permutation code preserving a double Eulerian bistatistic. Discrete Appl Math 224: 9–15

    Article  MathSciNet  Google Scholar 

  11. Cruanes S, Blanchette JC (2016) Extending Nunchaku to dependent type theory. In: Proceedings first international workshop on hammers for type theories, HaTT@IJCAR 2016, Coimbra, Portugal, July 1, 2016, volume 210 of EPTCS, pp 3–12

    Article  MathSciNet  Google Scholar 

  12. Carlier M, Dubois C, Gotlieb A (2010) Constraint reasoning in FOCALTEST. In: Proceedings of the 5th International Conference on Software and Data Technologies - Volume 2: ICSOFT. SciTePress, pp 82–91

  13. Claessen K, Hughes J (2000) QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of the fifth ACM SIGPLAN international conference on functional programming, volume 35 of SIGPLAN notices. ACM, New York, pp 268–279

    Article  Google Scholar 

  14. Cohen C (2013) Pragmatic quotient types in Coq. In: ITP 2013, volume 7998 of LNCS. Springer, Berlin, pp 213–228

    Chapter  Google Scholar 

  15. The Coq Development Team (2017) The Coq Proof Assistant Reference Manual. http://coq.inria.fr/. Version 8.7

  16. Cruanes S (2017) Satisfiability modulo bounded checking. In: Automated deduction–CADE 26, volume 10395 of LNCS. Springer, pp 114–129

  17. Dubois C, Giorgetti A, Genestier R (2016) Tests and proofs for enumerative combinatorics. In: Tests and proofs (TAP), volume 6792 of LNCS. Springer, pp 57–75

  18. Dybjer P, Haiyan Q, Takeyama M (2003) Combining testing and proving in dependent type theory. In: TPHOLs 2003, volume 2758 of LNCS. Springer, Heidelberg, pp 188–203

    Google Scholar 

  19. Duregård J, Jansson P, Wang M (2012) Feat: functional enumeration of algebraic types. In: Proceedings of the 2012 Haskell Symposium, volume 47 of SIGPLAN Notices. ACM, New York, pp 61–72

    Article  Google Scholar 

  20. Dubois C, Mota J-M (2007) Geometric modeling with B: formal specification of generalized maps.. J Sci Pract Comput 1(2): 9–24

    Google Scholar 

  21. Dufourd J-F (2007) Design and formal proof of a new optimal image segmentation program with hypermaps. Pattern Recogn 40(11): 2974–2993

    Article  Google Scholar 

  22. Dufourd J-F (2008) Polyhedra genus theorem and Euler formula: a hypermap-formalized intuitionistic proof. Theor Comput Sci 403(2–3): 133–159

    Article  MathSciNet  Google Scholar 

  23. Dufourd J-F (2009) An intuitionistic proof of a discrete form of the Jordan curve theorem formalized in Coq with combinatorial hypermaps.. J Autom Reason 43(1): 19–51

    Article  MathSciNet  Google Scholar 

  24. Dumont D, Viennot G (1980) A combinatorial interpretation of the Seidel generation of Genocchi numbers. In: Srivastava J (ed) Combinatorial mathematics, optimal designs and their applications, volume 6 of annals of discrete mathematics. Elsevier, pp 77–87

  25. Eynard B (2011) Formal matrix integrals and combinatorics of maps. Springer, New York, pp 415–442

    Chapter  Google Scholar 

  26. Filliâtre J-C, Paskevich A (2013) Why3—where programs meet provers. In: Proceedings of the 22nd European symposium on programming, volume 7792 of LNCS. Springer, pp 125–128

  27. Filliâtre J-C, Pereira M (2016) A modular way to reason about iteration. In: 8th NASA formal methods symposium, volume 9690 of LNCS. Springer, pp 322–336

  28. Genestier R, Giorgetti A, Petiot G (2015) Sequential generation of structured arrays and its deductive verification. In: Tests and proofs (TAP) 2015, volume 9154 of LNCS. Springer, Heidelberg, pp 109–128

    Chapter  Google Scholar 

  29. Gonthier G (2005) A computer checked proof of the Four Colour Theorem. http://research.microsoft.com/gonthier/4colproof.pdf

  30. Gonthier G (2008) The four colour theorem: engineering of a formal proof. In: ASCM 2007, volume 5081 of LNCS (LNAI). Springer, Heidelberg, pp 333–333

  31. Giorgetti A, Senni V (2012) Specification and validation of algorithms generating planar Lehman words. GASCom’12. https://hal.inria.fr/hal-00753008

  32. Hriţcu C, Lampropoulos L, Dénès M, Paraskevopoulou Z (2018) QuickChick: randomized property-based testing plugin for Coq. https://github.com/QuickChick/QuickChick

  33. Kitaev S (2011) Patterns in permutations and words. Springer, New York

    Book  Google Scholar 

  34. Lazarus F (2014) Combinatorial graphs and surfaces from the computational and topological viewpoint followed by some notes on the isometric embedding of the square flat torus. http://www.gipsa-lab.grenoble-inp.fr/~francis.lazarus/Documents/hdr-Lazarus.pdf.

  35. Lehmer DH (1960) Teaching combinatorial tricks to a computer. In: Proceedings of symposia in applied mathematics combinatorial analysis. American Mathematical Society, vol 10, pp 179–193

  36. Lampropoulos L, Gallois-Wong D, Hriţcu C, Hughes J, Pierce BC, Xia L (2017) Beginner’s luck: a language for property-based generators. In: Proceedings of the 44th ACM SIGPLAN symposium on principles of programming languages, POPL 2017, Paris, France, January 18–20, 2017. ACM, pp 114–129

  37. Lindblad F (2007) Property directed generation of first-order test data. In: Proceedings of the Eighth Symposium on Trends in Functional Programming, TFP 2007, New York City, New York, USA, April 2–4, 2007, volume 8 of Trends in Functional Programming. Intellect, pp 105–123

  38. Lampropoulos L, Paraskevopoulou Z, Pierce BC (2018) Generating good generators for inductive relations. PACMPL 2(POPL):45:1–45:30

    Article  Google Scholar 

  39. Lando SK, Zvonkin AK (2004) Graphs on surfaces and their applications. Springer

  40. Mathematical Components Team (2018) Mathematical components library. http://math-comp.github.io/math-comp/

  41. Mednykh A, Nedela R (2017) Recent progress in enumeration of hypermaps.. J Math Sci 226(5): 635–654

    Article  Google Scholar 

  42. Mantaci R, Rakotondrajao F (2001) A permutations representation that knows what “Eulerian” means. Discrete Math Theor Comput Sci 4(2): 101–108

    MathSciNet  MATH  Google Scholar 

  43. The OEIS Foundation Inc. The on-line encyclopedia of integer sequences. https://oeis.org/A000698

  44. Owre S (2006) Random testing in PVS. Workshop on Automated Formal Methods (AFM). http://fm.csl.sri.com/AFM06/papers/5-Owre.pdf

  45. Palka MH, Claessen K, Russo A, Hughes J (2011) Testing an optimising compiler by generating random lambda terms. In: Proceedings of the 6th international workshop on automation of software test, AST 2011, Waikiki, Honolulu, HI, USA, May 23–24, 2011. ACM, pp 91–97

  46. Planat M, Giorgetti A, Holweck F, Saniga M (2015) Quantum contextual finite geometries from dessins d’enfants. Int J Geom Methods Mod Phys 12: 1–17

    Article  MathSciNet  Google Scholar 

  47. Paraskevopoulou Z, Hriţcu C, Dénès M, Lampropoulos L, Pierce BC (2015) Foundational property-based testing. In: ITP 2015, volume 9236 of LNCS. Springer, Heidelberg, pp 325–343

    Google Scholar 

  48. Runciman C, Naylor M, Lindblad F (2008) SmallCheck and Lazy SmallCheck: automatic exhaustive testing for small values. In: Proceedings of the 1st ACM SIGPLAN symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008, pp 37–48

  49. Senni V (2018) Validation library. https://subversion.assembla.com/svn/validation/

  50. Stanley RP (1997) Enumerative combinatorics, vol 1. Cambridge University Press, Cambridge

    Book  Google Scholar 

  51. Seidel EL, Vazou N, Jhala R (2015) Type targeted testing. In: ESOP 2015, volume 9032 of LNCS. Springer, Heidelberg, pp 812–836

    Chapter  Google Scholar 

  52. SWI (2018) Prolog. http://www.swi-prolog.org/

  53. Tarau P (2015) On type-directed generation of lambda terms. In: Proceedings of the technical communications of the 31st international conference on logic programming (ICLP 2015), Cork, Ireland, August 31–September 4, 2015, volume 1433 of CEUR Workshop Proceedings. CEUR-WS.org

  54. Tutte WT (1973) What is a map? In: New directions in the theory of graphs: proceedings. Academic Press, New York, pp 309–325

  55. Tutte WT (1979) Combinatorial oriented maps.. Can J Math 31(5): 986–1004

    Article  MathSciNet  Google Scholar 

  56. Vajnovszki V (2011) A new Euler–Mahonian constructive bijection.. Discrete Appl Math 159(14): 1453–1459

    Article  MathSciNet  Google Scholar 

  57. Vajnovszki V (2013) Lehmer code transforms and Mahonian statistics on permutations.. Discrete Math, 313(5): 581–589

    Article  MathSciNet  Google Scholar 

  58. Walsh TRS, Lehman AB (1972) Counting rooted maps by genus I.. J Comb Theory Ser B 13: 192–218

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

For this work Alain Giorgetti was supported by the French ``Investissements d'Avenir'' program, project ISITEBFC (contract ANR-15-IDEX-03). The authors warmly thank Nicolas Magaud for help with Coq, Valerio Senni for advice about his validation library, Noam Zeilberger and Cyril Cohen for fruitful discussions. We deeply thank Leonidas Lampropoulos for his support on QuickChick. We also thank anonymous referees for their helpful suggestions.

Author information

Authors and Affiliations

  1. Samovar, ENSIIE, CNRS, Évry, France

    Catherine Dubois

  2. FEMTO-ST Institute, University of Bourgogne Franche-Comté, CNRS, Besançon, France

    Alain Giorgetti

Authors
  1. Catherine Dubois
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alain Giorgetti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alain Giorgetti.

Additional information

Bernhard Aichernig, Marie-Claude Gaudel, Carlo Furia, and Rob Hierons

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dubois, C., Giorgetti, A. Tests and proofs for custom data generators. Form Asp Comp 30, 659–684 (2018). https://doi.org/10.1007/s00165-018-0459-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00165-018-0459-1

Keywords

Search

Navigation