Advertisement

How to Specify It!

A Guide to Writing Properties of Pure Functions
Conference paper
First Online:
  • 154 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12053)

Abstract

Property-based testing tools test software against a specification, rather than a set of examples. This tutorial paper presents five generic approaches to writing such specifications (for purely functional code). We discuss the costs, benefits, and bug-finding power of each approach, with reference to a simple example with eight buggy variants. The lessons learned should help the reader to develop effective property-based tests in the future.

This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

I’m grateful to the anonymous referees for many useful suggested improvements, and to Vetenskapsrådet for funding this work under the SyTeC grant.

Supplementary material

Open image in new window

References

  1. 1.
    Amaral, C., Florido, M., Santos Costa, V.: PrologCheck – property-based testing in prolog. In: Codish, M., Sumii, E. (eds.) FLOPS 2014. LNCS, vol. 8475, pp. 1–17. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-07151-0_1CrossRefGoogle Scholar
  2. 2.
    Arts, T., Hughes, J., Johansson, J., Wiger, U.T.: Testing telecoms software with Quviq QuickCheck. In: Feeley, M., Trinder, P.W. (eds.) Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang, Portland, Oregon, USA, 16 September 2006, pp. 2–10. ACM (2006)Google Scholar
  3. 3.
    Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S.: The oracle problem in software testing: a survey. IEEE Trans. Soft. Eng. 41(5), 507–525 (2015)CrossRefGoogle Scholar
  4. 4.
    Braquehais, R.M.: Tools for discovery, refinement and generalization of functional properties by enumerative testing. Ph.D. thesis, University of York, UK (2017)Google Scholar
  5. 5.
    Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., Pretschner, A. (eds.): Model-Based Testing of Reactive Systems. LNCS, vol. 3472. Springer, Heidelberg (2005).  https://doi.org/10.1007/b137241CrossRefzbMATHGoogle Scholar
  6. 6.
    Chen, T.Y., Cheung, S.C., Yiu, S.M.: Metamorphic testing: a new approach for generating next test cases. Technical report, HKUST-CS98-01, Department of Computer Science, Hong Kong (1998)Google Scholar
  7. 7.
    Chen, T.Y., et al.: Metamorphic testing: a review of challenges and opportunities. ACM Comput. Surv. 51(1), 4:1–4:27 (2018)CrossRefGoogle Scholar
  8. 8.
  9. 9.
    Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: Proceedings of 5th ACM SIGPLAN International Conference on Functional Programming, ICFP 2000 (2000)Google Scholar
  10. 10.
    Lindley, S., McBride, C., Trinder, P., Sannella, D. (eds.): A List of Successes That Can Change the World. LNCS, vol. 9600. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-30936-1CrossRefGoogle Scholar
  11. 11.
    Floyd, R.W.: Assigning meanings to programs. In: Colburn, T.R., Fetzer, J.H., Rankin, T.L. (eds.) Program Verification, vol. 14, pp. 65–81. Springer, Dordrecht (1993).  https://doi.org/10.1007/978-94-011-1793-7_4CrossRefGoogle Scholar
  12. 12.
    Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. ACM SIGPLAN Not. 40, 213–223 (2005)CrossRefGoogle Scholar
  13. 13.
    Guttag, J.V., Horning, J.J.: The algebraic specification of abstract data types. Acta Inform. 10(1), 27–52 (1978)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hoare, C.A.: Proof of correctness of data representations. Acta Inform. 1(4), 271–281 (1972)CrossRefGoogle Scholar
  15. 15.
    Hoare, C.A.R.: An axiomatic basis for computer programming. Commun. ACM 12(10), 576–580 (1969)CrossRefGoogle Scholar
  16. 16.
    Hughes, J.: Experiences with QuickCheck: testing the hard stuff and staying sane. In: Lindley et al. [10], pp. 169–186 (2016)Google Scholar
  17. 17.
    Löscher, A., Sagonas, K.: Targeted property-based testing. In: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 46–56. ACM (2017)Google Scholar
  18. 18.
    Meyer, B.: Applying ‘design by contract’. Computer 25(10), 40–51 (1992)CrossRefGoogle Scholar
  19. 19.
    Nilsson, R.: ScalaCheck: The Definitive Guide. Artima Press, Mountain View (2014)Google Scholar
  20. 20.
    Papadakis, M., Sagonas, K.: A proper integration of types and function specifications with property-based testing. In: Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, pp. 39–50. ACM (2011)Google Scholar
  21. 21.
    Pike, L.: SmartCheck: automatic and efficient counterexample reduction and generalization. In: Swierstra, W. (ed.) Proceedings of the 2014 ACM SIGPLAN Symposium on Haskell, Gothenburg, Sweden, 4–5 September 2014, pp. 53–64. ACM (2014)Google Scholar
  22. 22.
    Polya, G.: How To Solve It! A System of Thinking Which Can Help You Solve Any Problem. Princeton University Press, Princeton (1945)Google Scholar
  23. 23.
    Runciman, C., Naylor, M., Lindblad, F.: SmallCheck and lazy SmallCheck: automatic exhaustive testing for small values. In: Gill, A. (ed.) Proceedings of the 1st ACM SIGPLAN Symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008, pp. 37–48. ACM (2008)Google Scholar
  24. 24.
    Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. ACM SIGSOFT Softw. Eng. Notes 30, 263–272 (2005)CrossRefGoogle Scholar
  25. 25.
    Spivey, J.M.: Understanding Z: A Specification Language and Its Formal Semantics, vol. 3. Cambridge University Press, Cambridge (1988)zbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Chalmers University of Technology and Quviq ABGöteborgSweden

Personalised recommendations

Cite paper

Buy options