Skip to main content
SpringerLink
Log in

How testing helps to diagnose proof failures

  • Original Article
  • Published:
Formal Aspects of Computing

Abstract

Applying deductive verification to formally prove that a program respects its formal specification is a very complex and time-consuming task due in particular to the lack of feedback in case of proof failures. Along with a non-compliance between the code and its specification (due to an error in at least one of them), possible reasons of a proof failure include a missing or too weak specification for a called function or a loop, and lack of time or simply incapacity of the prover to finish a particular proof. This work proposes a methodology where test generation helps to identify the reason of a proof failure and to exhibit a counterexample clearly illustrating the issue. We define the categories of proof failures, introduce two subcategories of contract weaknesses (single and global ones), and examine their properties. We describe how to transform a C program formally specified in an executable specification language into C code suitable for testing, and illustrate the benefits of the method on comprehensive examples. The method has been implemented in StaDy, a plugin of the software analysis platform Frama-C. Initial experiments show that detecting non-compliances and contract weaknesses allows to precisely diagnose most proof failures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Arlt S, Arenis SF, Podelski A, Wehrle M (2015) System testing and program verification. Softw Eng Manag, vol 239 of LNI. GI, pp 71–72

  2. Ahn KY, Denney E (2010) Testing first-order logic axioms in program verification. TAP, vol 6143 of LNCS. Springer, pp 22–37

  3. Bertot Y., Castéran P. (2004) Interactive theorem proving and program development; Coq'Art: the calculus of inductive constructions Texts in theoretical computer science. An EATCS series.Springer, Berlin.

    Book  Google Scholar 

  4. Baudin P, Cuoq P, Filliâtre J-C, Marché C., Monate B., Moy Y., Prevosto V (2017) ACSL: ANSI/ISO C specification language. http://frama-c.com/acsl.html

  5. Botella B, Delahaye M, Hong Tuan Ha S, Kosmatov N,Mouy P, Roger M,Williams N (2009) Automating structural testing of C programs: experience with Path Crawler. AST. IEEE Computer Society, pp 70–78

  6. Burghardt J, Gerlach J (2017) ACSL by example. https://github.com/fraunhoferfokus/acsl-by-example

  7. Beckert B,Hähnle R, Schmitt PH (eds) (2007) Verification of object-oriented software: the key approach.LNCS 4334. Springer, Heidelberg

  8. Blatter L. Kosmatov N. Le Gall P. Prevosto V. Petiot G. (2018) Static and dynamic verification of relational properties on self-composed C code. TAP, LNCS. Springer To appear

  9. Berghofer S, Nipkow T (2004) Random testing in Isabelle/HOL. SEFM. IEEE Computer Society, pp 230–239

  10. Cousot P, Cousot R, Fähndrich M, Logozzo F (2013) Automatic inference of necessary preconditions. VMCAI, vol 7737 of LNCS. Springer, pp 128–148

  11. Chamarthi HR, Dillinger PC, Kaufmann M,Manolios P (2011) Integrating testing and interactive theorem proving. ACL2, vol 70 of EPTCS, pp 4–19

    Article  Google Scholar 

  12. Christakis M, Emmisberger P, Müller P (2014) Dynamic 1075 test generation with static fields and initializers. RV, vol 8734 of LNCS. Springer, pp 269–284

  13. Christ J, Ermis E, Schäf M,Wies T (2013) Flow-sensitive fault localization. VMCAI, vol 7737 of LNCS. Springer, pp 189–208

  14. Clarke E., Grumberg O., Jha S., Lu Y., Veith H. (2003) Counterexample-guided abstraction refinement for symbolic model checking. J ACM 50(5): 752–794

    Article  MathSciNet  Google Scholar 

  15. Chebaro O, Kosmatov N, Giorgetti A, Julliand J (2012) Program slicing enhances a verification technique combining static and dynamic analysis. SAC. ACM, pp 1284–1291

  16. Christakis M, Leino KRM, Müller P, Wüstholz V. Integrated environment for diagnosing verification errors. TACAS, vol 9636 of LNCS. Springer, pp 424–441

  17. Christakis M, Mü ller P, Wüstholz V (2012) Collaborative verification and testing with explicit assumptions. FM, vol 7436 of LNCS. Springer, pp 132–146

  18. Coq Development Team. The Coq Proof Assistant Reference Manual , 2018. http://coq.inria.fr/.

  19. Claessen K, Svensson H (2008) Finding counter examples in induction proofs. TAP, vol 4966 of LNCS. Springer, pp 48–65

  20. Chen TY., Tse TH., Zhou Z. (2011) Semi-proving: an integrated method for program proving, testing, and debugging. IEEE Trans Softw Eng 37(1): 109–125

    Article  Google Scholar 

  21. Dimitrova R, Finkbeiner B (2012). Counterexample-guided synthesis of observation predicates. FORMATS, vol 7595 of LNCS. Springer, pp 107–122

  22. de Gouw S, Rot J, de Boer FS, Bubel R, Hähnle R (2015) Open JDK’s Java.utils.Collection.sort() is broken: the good, the bad and the worst case. CAV, vol 9206 of LNCS. Springer, pp 273–289

  23. Dybjer P, Haiyan Q, Takeyama M (2003) Combining testing and proving in dependent type theory. TPHOLs, vol 2758 of LNCS. Springer, pp 188–203

  24. Dijkstra EW. (1976) A discipline of programming Series in automatic computation. Prentice Hall, Englewood Cliffs

    Google Scholar 

  25. Delahaye M, Kosmatov N, Signoles J (2013) Common specification language for static and dynamic analysis of C programs. SAC. ACM, pp 1230–1235

  26. Engel C, Hähnle R (2007) Generating unit tests from formal proofs. TAP, vol 4454 of LNCS. Springer, pp 169–188

  27. Genestier R, Giorgetti A, Petiot G (2015) Sequential generation of structured arrays and its deductive verification. TAP, vol 9154 of LNCS. Springer, pp 109–128

  28. Gulavani BS, Henzinger TA, Kannan Y, Nori AV, Rajamani SK (2006) SYNERGY: a new algorithm for property checking. FSE. ACM, pp 117–127

  29. Groce A, Kroening D, Lerda F (2004) Understanding counterexamples with explain. CAV, vol 3114 of LNCS. Springer, pp 453–456

  30. Guo S, Kusano M, Wang C, Yang Z, Gupta A (2015) Assertion guided symbolic execution of multithreaded programs. ESEC/FSE.ACM, pp 854–865

  31. Gladisch C (2009) Could we have chosen a better loop invariant or method contract?. TAP, vol 5668 of LNCS. Springer, pp 74–89

  32. Godefroid P, Nori AV, Rajamani SK, Tetali SD (2010) Compositional may-must program analysis: unleashing the power of alternation. POPL. ACM, pp 43–56

  33. Hauzar D, Marché C, Moy Y (2016) Counterexamples from proof failures in SPARK. SEFM vol 9763 of LNCS. Springer, pp 215–233

  34. Jakobsson A, Kosmatov N, Signoles J (2015) Fast as a shadow, expressive as a tree: hybrid memory monitoring for C. SAC. ACM,pp 1765–1772

  35. Kirchner F., Kosmatov N., Prevosto V., Signoles J., Yakobowski B. (2015) Frama-C: a software analysis perspective. Formal Asp Comput 27(3): 573–609

    Article  MathSciNet  Google Scholar 

  36. Kosmatov N (2010–2015). Online version of PathCrawler.http://pathcrawler-online.com/

  37. Kosmatov N. Petiot G. Signoles J. (2013) An optimized memory monitoring for runtime assertion checking of C programs. RV, vol 8174 of LNCS. Springer, pp 328–333

  38. Kovács L, Voronkov A (2009) Finding loop invariants for programs over arrays using a theorem prover. FASE, vol 5503 of LNCS. Springer, pp 470–485

  39. Müller P, Ruskiewicz JN (2011) Using debuggers to understand failed verification attempts. FM, vol 6664 of LNCS. Springer, pp 73–87

  40. Mansour T., Vajnovszki V. (2013) Efficient generation of restricted growth words. Inf Process Lett 113(17): 613–616

    Article  MathSciNet  Google Scholar 

  41. Owre S (2006) Random testing in PVS. Workshop on automated formal methods (AFM)

  42. Petiot G, Botella B, Julliand J, Kosmatov N, Signoles J (2014) Instrumentation of annotated C programs for test generation. SCAM. IEEE Computer Society, pp 105–114

  43. Petiot G, Kosmatov N, Botella B, Giorgetti A, Julliand J (2016) Your proof fails? Testing helps to find the reason. TAP, vol 9762 of LNCS. Springer, pp 130–150

  44. Petiot G, Kosmatov N, Giorgetti A, Julliand J (2014) Howtest generation helps software specification and deductive verification in Frama-C. TAP, vol 8570 of LNCS. Springer, pp 53–60

  45. Podelski A, Wies T (2010) Counterexample-guided focus. POPL. ACM, pp 249–260

  46. Signoles J (2012). E-ACSL: executable ANSI/ISO C specification language. http://frama-c.com/download/e-acsl/e-acsl.pdf.

  47. Tschannen J, Furia CA, Nordio M, Meyer B(2013) Program checking with less hassle. VSTTE, vol 8164 of LNCS. Springer, pp 149–169

  48. Williams N, Marre B, Mouy P, Roger M (2005) PathCrawler: automatic generation of path tests by combining static and dynamic analysis. EDCC, vol 3463 LNCS. Springer, pp 281–292

Download references

Author information

Authors and Affiliations

  1. Software Reliability Laboratory, CEA, List, PC 174, 91191, Gif-sur-Yvette, France

    Guillaume Petiot, Nikolai Kosmatov & Bernard Botella

  2. FEMTO-ST Institute, University of Bourgogne Franche-Comté, CNRS, 25030, Besançon Cedex, France

    Guillaume Petiot, Alain Giorgetti & Jacques Julliand

Authors
  1. Guillaume Petiot
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nikolai Kosmatov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernard Botella
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alain Giorgetti
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jacques Julliand
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nikolai Kosmatov.

Additional information

Bernhard Aichernig, Marie-Claude Gaudel, Carlo A. Furia, and Robert M. Hierons

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Petiot, G., Kosmatov, N., Botella, B. et al. How testing helps to diagnose proof failures. Form Asp Comp 30, 629–657 (2018). https://doi.org/10.1007/s00165-018-0456-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00165-018-0456-4

Keywords

Search

Navigation