Advertisement

Property-Based Test Case Generators for Free

  • Emanuele De AngelisEmail author
  • Fabio FioravantiEmail author
  • Adrián PalaciosEmail author
  • Alberto PettorossiEmail author
  • Maurizio ProiettiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11823)

Abstract

Property-Based Testing requires the programmer to write suitable generators, i.e., programs that generate (possibly in a random way) input values for which the program under test should be run. However, the process of writing generators is quite a costly, error-prone activity. In the context of Property-Based Testing of Erlang programs, we propose an approach to relieve the programmer from the task of writing generators. Our approach allows the automatic, efficient generation of input test values that satisfy a given specification. In particular, we have considered the case when the input values are data structures satisfying complex constraints. That generation is performed via the symbolic execution of the specification using constraint logic programming.

Notes

Acknowledgements

We would like to thank the anonymous reviewers for their very helpful and constructive comments.

E. De Angelis, F. Fioravanti, A. Pettorossi, and M. Proietti are members of the INdAM Research group GNCS. E. De Angelis, F. Fioravanti, and A. Pettorossi are research associates at CNR-IASI, Rome, Italy.

A. Palacios was partially supported by the EU (FEDER) and the Spanish Ayudas para contratos predoctorales para la formación de doctores and Ayudas a la movilidad predoctoral para la realización de estancias breves en centros de I+D (MICINN) under FPI grants BES-2014-069749 and EEBB-I-17-12101.

References

  1. 1.
    Amaral, C., Florido, M., Santos Costa, V.: PrologCheck – property-based testing in Prolog. In: Codish, M., Sumii, E. (eds.) FLOPS 2014. LNCS, vol. 8475, pp. 1–17. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-07151-0_1CrossRefGoogle Scholar
  2. 2.
    Arts, Th., Hughes, J., Johansson, J., Wiger, U.T.: Testing telecoms software with Quviq QuickCheck. In: Feeley, M., Trinder, Ph.W., (eds.) Proceedings of the 2006 ACM SIGPLAN Workshop on Erlang, Portland, Oregon, USA, 16 September 2006, pp. 2–10. ACM (2006)Google Scholar
  3. 3.
    Bouquet, F., Dadeau, F., Legeard, B.: Automated boundary test generation from JML specifications. In: Misra, J., Nipkow, T., Sekerinski, E. (eds.) FM 2006. LNCS, vol. 4085, pp. 428–443. Springer, Heidelberg (2006).  https://doi.org/10.1007/11813040_29CrossRefGoogle Scholar
  4. 4.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on Java predicates. In: Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 123–133. ACM, New York (2002)CrossRefGoogle Scholar
  5. 5.
    Carlier, M., Dubois, C., Gotlieb, A.: A first step in the design of a formally verified constraint-based testing tool: FocalTest. In: Brucker, A.D., Julliand, J. (eds.) TAP 2012. LNCS, vol. 7305, pp. 35–50. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-30473-6_5CrossRefGoogle Scholar
  6. 6.
    Carlier, M., Dubois, C., Gotlieb, A.: FocalTest: a constraint programming approach for property-based testing. In: Cordeiro, J., Virvou, M., Shishkov, B. (eds.) ICSOFT 2010. CCIS, vol. 170, pp. 140–155. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-29578-2_9CrossRefGoogle Scholar
  7. 7.
    Cheney, J., Momigliano, A.: \(\alpha \)Check: a mechanized metatheory model checker. Theory Pract. Log. Program. 17(3), 311–352 (2017)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Claessen, K., Hughes, J.: QuickCheck: a lightweight tool for random testing of Haskell programs. In: Odersky, M., Wadler, Ph. (eds.) Proceedings of the 5th ACM SIGPLAN International Conference on Functional Programming (ICFP 2000), Montreal, Canada, 18–21 September 2000, pp. 268–279. ACM (2000)Google Scholar
  9. 9.
    Coppit, D., Le, W., Sullivan, K.J., Khurshid, S., Yang, J.: Software assurance by bounded exhaustive testing. IEEE Trans. Softw. Eng. 31(4), 328–339 (2005)CrossRefGoogle Scholar
  10. 10.
    Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms, 3rd edn. MIT Press, Cambridge (2009)zbMATHGoogle Scholar
  11. 11.
    Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: Woodcock, J.C.P., Larsen, P.G. (eds.) FME 1993. LNCS, vol. 670, pp. 268–284. Springer, Heidelberg (1993).  https://doi.org/10.1007/BFb0024651CrossRefGoogle Scholar
  12. 12.
    Carlsson, R., et al.: Core Erlang 1.0.3. language specification. Technical Report, Department of Information Technology, Uppsala University, Uppsala, Sweden (2004). https://www.it.uu.se/research/group/hipe/cerl/doc/core_erlang-1.0.3.pdf
  13. 13.
    Fioravanti, F., Proietti, M., Senni, V.: Efficient generation of test data structures using constraint logic programming and program transformation. J. Log. Comput. 25(6), 1263–1283 (2015)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Gligoric, M., Gvero, T., Jagannath, V., Khurshid, S., Kuncak, V., Marinov, D.: Test generation through programming in UDITA. In: Kramer, J., Bishop, J., Devanbu, P.T., Uchitel, S. (eds.) Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, Cape Town, South Africa, 2–8 May 2010, pp. 225–234. ACM (2010)Google Scholar
  15. 15.
    Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. In: Sarkar, V., Hall, M.W. (eds.) Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation, Chicago, IL, USA, 12–15 June 2005, pp. 213–223. ACM (2005)Google Scholar
  16. 16.
    Gómez-Zamalloa, M., Albert, E., Puebla, G.: Test case generation for object-oriented imperative languages in CLP. Theory Pract. Log. Program. 10(4–6), 659–674 (2010)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Gotlieb, A.: Euclide: a constraint-based testing framework for critical C programs. In: 2nd International Conference on Software Testing, Verification and Validation, ICST 2009, Denver, Colorado, USA, 1–4 April 2009, pp. 151–160. IEEE Computer Society (2009)Google Scholar
  18. 18.
    Gotlieb, A., Botella, B., Rueher, M.: A CLP framework for computing structural test data. In: Lloyd, J. (ed.) CL 2000. LNCS (LNAI), vol. 1861, pp. 399–413. Springer, Heidelberg (2000).  https://doi.org/10.1007/3-540-44957-4_27CrossRefGoogle Scholar
  19. 19.
    Hewitt, C., Bishop, P., Steiger, R.: A universal modular ACTOR formalism for artificial intelligence. In: Proceedings of the 3rd International Joint Conference on Artificial Intelligence, IJCAI 1973, San Francisco, CA, USA, pp. 235–245. Morgan Kaufmann Publishers Inc. (1973)Google Scholar
  20. 20.
    Jaffar, J., Maher, M.: Constraint logic programming: a survey. J. Log. Program. 19(20), 503–581 (1994)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Jones, N.D., Gomard, C.K., Sestoft, P.: Partial Evaluation and Automatic Program Generation. Prentice Hall, Upper Saddle River (1993)zbMATHGoogle Scholar
  22. 22.
    junit-quickcheck: Property-based testing, JUnit-style. https://github.com/pholser/junit-quickcheck
  23. 23.
    Kowalski, R.A.: Logic for Problem Solving. North Holland, Amsterdam (1979)zbMATHGoogle Scholar
  24. 24.
    Lampropoulos, L., Paraskevopoulou, Z., Pierce, B.C.: Generating good generators for inductive relations. In: Proceedings of the ACM on Programming Languages, vol. 2, pp. 45:1–45:30 (2017)CrossRefGoogle Scholar
  25. 25.
    Marre, B.: Toward automatic test data set selection using algebraic specifications and logic programming. In: Furukawa, K. (eds.) Logic Programming, Proceedings of the 8th International Conference, Paris, France, 24–28 June 1991, pp. 202–219. MIT Press (1991)Google Scholar
  26. 26.
    Marre, B., Arnould, A.: Test sequences generation from LUSTRE descriptions: GATeL. In: Proceedings of the 15th IEEE International Conference on Automated Software Engineering, ASE 2000, Grenoble, France, 11–15 September 2000, p. 229. IEEE Computer Society (2000)Google Scholar
  27. 27.
    Papadakis, M., Sagonas, K.: A PropEr integration of types and function specifications with property-based testing. In: Rikitake, K., Stenman, E. (eds.) Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, Tokyo, Japan, 23 September 2011, pp. 39–50. ACM (2011)Google Scholar
  28. 28.
    Paraskevopoulou, Z., Hriţcu, C., Dénès, M., Lampropoulos, L., Pierce, B.C.: Foundational property-based testing. In: Urban, C., Zhang, X. (eds.) ITP 2015. LNCS, vol. 9236, pp. 325–343. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-22102-1_22CrossRefGoogle Scholar
  29. 29.
    Pretschner, A., Lötzbeyer, H.: Model based testing with constraint logic programming: first results and challenges. In: Proceedings of the 2nd ICSE Workshop on Automated Program Analysis, Testing and Verification (WAPATV), pp. 1–9 (2001)Google Scholar
  30. 30.
    PropEr: Property-Based Testing for Erlang. http://proper.softlab.ntua.gr/
  31. 31.
    Runciman, C., Naylor, M., Lindblad, F.: SmallCheck and lazy SmallCheck: automatic exhaustive testing for small values. In: Gill, A. (ed.) Proceedings of the 1st ACM SIGPLAN Symposium on Haskell, Haskell 2008, Victoria, BC, Canada, 25 September 2008, pp. 37–48. ACM (2008)Google Scholar
  32. 32.
    Rushby, J.: Automated test generation and verified software. In: Meyer, B., Woodcock, J. (eds.) VSTTE 2005. LNCS, vol. 4171, pp. 161–172. Springer, Heidelberg (2008).  https://doi.org/10.1007/978-3-540-69149-5_18CrossRefGoogle Scholar
  33. 33.
    ScalaCheck: Property-Based Testing for Scala. http://www.scalacheck.org/
  34. 34.
    Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: Wermelinger, M., Gall, H.C. (eds.) Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2005, Lisbon, Portugal, 5–9 September 2005, pp. 263–272. ACM (2005)Google Scholar
  35. 35.
    The SWI Prolog Logic Programming System. http://www.swi-prolog.org/
  36. 36.
    Virding, R., Wikström, C., Williams, M.: Concurrent programming in Erlang, 2nd edn. In: Armstrong, J. (ed.) Prentice Hall International Ltd, Hertfordshire (1996)Google Scholar
  37. 37.
    Williams, N., Marre, B., Mouy, P., Roger, M.: PathCrawler: automatic generation of path tests by combining static and dynamic analysis. In: Dal Cin, M., Kaâniche, M., Pataricza, A. (eds.) EDCC 2005. LNCS, vol. 3463, pp. 281–292. Springer, Heidelberg (2005).  https://doi.org/10.1007/11408901_21CrossRefGoogle Scholar
  38. 38.
    Xu, G., Yang, Z.: JMLAutoTest: a novel automated testing framework based on JML and JUnit. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 70–85. Springer, Heidelberg (2004).  https://doi.org/10.1007/978-3-540-24617-6_6CrossRefGoogle Scholar
  39. 39.
    Yatoh, K., Sakamoto, K., Ishikawa, F., Honiden, S.: ArbitCheck: a highly automated property-based testing tool for Java. In: Proceedings of the 7th IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2014, Cleveland, Ohio, USA, 31 March–April 4 2014, pp. 405–412. IEEE Computer Society (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.DEC, University “G. d’Annunzio” of Chieti-Pescara PescaraItaly
  2. 2.MiST, DSIC, Polytechnic University of ValenciaValenciaSpain
  3. 3.University of Rome Tor VergataRomeItaly
  4. 4.CNR-IASIRomeItaly

Personalised recommendations