Foundational Property-Based Testing

  • Zoe Paraskevopoulou
  • Cătălin HriţcuEmail author
  • Maxime Dénès
  • Leonidas Lampropoulos
  • Benjamin C. Pierce
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9236)


Integrating property-based testing with a proof assistant creates an interesting opportunity: reusable or tricky testing code can be formally verified using the proof assistant itself. In this work we introduce a novel methodology for formally verified property-based testing and implement it as a foundational verification framework for QuickChick, a port of QuickCheck to Coq. Our framework enables one to verify that the executable testing code is testing the right Coq property. To make verification tractable, we provide a systematic way for reasoning about the set of outcomes a random data generator can produce with non-zero probability, while abstracting away from the actual probabilities. Our framework is firmly grounded in a fully verified implementation of QuickChick itself, using the same underlying verification methodology. We also apply this methodology to a complex case study on testing an information-flow control abstract machine, demonstrating that our verification methodology is modular and scalable and that it requires minimal changes to existing code.



We thank John Hughes for insightful discussions and the anonymous reviewers for their helpful comments. This work was supported by NSF award 1421243, Random Testing for Language Design.


  1. 1.
    Appel, A.W.: Efficient verified red-black trees, Manuscript (2011)Google Scholar
  2. 2.
    Berghofer, S., Bulwahn, L., Haftmann, F.: Turning inductive into equational specifications. In: Berghofer, S., Nipkow, T., Urban, C., Wenzel, M. (eds.) TPHOLs 2009. LNCS, vol. 5674, pp. 131–146. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  3. 3.
    Brucker, A.D., Brügger, L., Wolff, B.: Formal firewall conformance testing: an application of test and proof techniques. Softw. Test. Verification Reliab. 25(1), 34–71 (2015)CrossRefGoogle Scholar
  4. 4.
    Brucker, A.D., Wolff, B.: On theorem prover-based testing. Formal Aspects Comput. 25(5), 683–721 (2013)zbMATHMathSciNetCrossRefGoogle Scholar
  5. 5.
    Bulwahn, L.: The new quickcheck for Isabelle. In: Hawblitzel, C., Miller, D. (eds.) CPP 2012. LNCS, vol. 7679, pp. 92–108. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  6. 6.
    Bulwahn, L.: Smart testing of functional programs in Isabelle. In: Bjørner, N., Voronkov, A. (eds.) LPAR-18 2012. LNCS, vol. 7180, pp. 153–167. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  7. 7.
    Carlier, M., Dubois, C., Gotlieb, A.: A first step in the design of a formally verified constraint-based testing tool: focaltest. In: Brucker, A.D., Julliand, J. (eds.) TAP 2012. LNCS, vol. 7305, pp. 35–50. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  8. 8.
    Chamarthi, H.R., Dillinger, P.C., Kaufmann, M., Manolios, P.: Integrating testing and interactive theorem proving. In: 10th International Workshop on the ACL2 Theorem Prover and its Applications. EPTCS, vol. 70, pp. 4–19 (2011)Google Scholar
  9. 9.
    Christiansen, J., Fischer, S.: EasyCheck — test data for free. In: Garrigue, J., Hermenegildo, M.V. (eds.) FLOPS 2008. LNCS, vol. 4989, pp. 322–336. Springer, Heidelberg (2008) CrossRefGoogle Scholar
  10. 10.
    Claessen, K.: Shrinking and showing functions: (functional pearl). In: 5th ACM SIGPLAN Symposium on Haskell, pp. 73–80. ACM (2012)Google Scholar
  11. 11.
    Claessen, K., Duregård, J., Pałka, M.H.: Generating constrained random data with uniform distribution. In: Codish, M., Sumii, E. (eds.) FLOPS 2014. LNCS, vol. 8475, pp. 18–34. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  12. 12.
    Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: 5th ACM SIGPLAN International Conference on Functional Programming (ICFP), pp. 268–279. ACM (2000)Google Scholar
  13. 13.
    Dybjer, P., Haiyan, Q., Takeyama, M.: Combining testing and proving in dependent type theory. In: Basin, D., Wolff, B. (eds.) TPHOLs 2003. LNCS, vol. 2758, pp. 188–203. Springer, Heidelberg (2003) CrossRefGoogle Scholar
  14. 14.
    Dybjer, P., Haiyan, Q., Takeyama, M.: Random generators for dependent types. In: Liu, Z., Araki, K. (eds.) ICTAC 2004. LNCS, vol. 3407, pp. 341–355. Springer, Heidelberg (2005) CrossRefGoogle Scholar
  15. 15.
    Fetscher, B., Claessen, K., Pałka, M., Hughes, J., Findler, R.B.: Making random judgments: automatically generating well-typed terms from the definition of a type-system. In: Vitek, J. (ed.) ESOP 2015. LNCS, vol. 9032, pp. 383–405. Springer, Heidelberg (2015) CrossRefGoogle Scholar
  16. 16.
    Fischer, S., Kuchen, H.: Systematic generation of glass-box test cases for functional logic programs. In: 9th International ACM SIGPLAN Conference on Principles and Practice of Declarative Programming (PPDP), pp. 63–74. ACM (2007)Google Scholar
  17. 17.
    Gonthier, G., Mahboubi, A.: An introduction to small scale reflection in Coq. J. Formalized Reasoning 3(2), 95–152 (2010)zbMATHMathSciNetGoogle Scholar
  18. 18.
    Haiyan, Q.: Testing and Proving in Dependent Type Theory. Ph.D. thesis, Chalmers (2003)Google Scholar
  19. 19.
    Hriţcu, C., Hughes, J., Pierce, B.C., Spector-Zabusky, A., Vytiniotis, D., de Amorim, A.A., Lampropoulos, L.: Testing noninterference, quickly. In: 18th ACM SIGPLAN International Conference on Functional Programming (ICFP), pp. 455–468. ACM (2013)Google Scholar
  20. 20.
    Hughes, J.: QuickCheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  21. 21.
    Owre, S.: Random testing in PVS. In: Workshop on Automated Formal Methods (2006)Google Scholar
  22. 22.
    Sozeau, M.: A new look at generalized rewriting in type theory. J. Formalized Reasoning 2(1), 41–62 (2009)zbMATHMathSciNetGoogle Scholar
  23. 23.
    Tollitte, P.-N., Delahaye, D., Dubois, C.: Producing certified functional code from inductive specifications. In: Hawblitzel, C., Miller, D. (eds.) CPP 2012. LNCS, vol. 7679, pp. 76–91. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  24. 24.
    Wilson, S.: Supporting dependently typed functional programming with proof automation and testing. Ph.D. thesis, The University of Edinburgh, June 2011Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Zoe Paraskevopoulou
    • 1
    • 2
  • Cătălin Hriţcu
    • 1
    Email author
  • Maxime Dénès
    • 1
  • Leonidas Lampropoulos
    • 3
  • Benjamin C. Pierce
    • 3
  1. 1.Inria Paris-RocquencourtRocquencourtFrance
  2. 2.ENS CachanCachanFrance
  3. 3.University of PennsylvaniaPhiladelphiaUSA

Personalised recommendations