Skip to main content
SpringerLink
Log in

Generating Tests by Example

  • Conference paper
  • First Online:
Verification, Model Checking, and Abstract Interpretation (VMCAI 2018)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10747))

Abstract

Property-based testing is a technique combining parametric tests with value generators, to create an efficient and maintainable way to test general specifications. To test the program, property-based testing randomly generates a large number of inputs defined by the generator to check whether the test-assertions hold.

We present a novel framework that synthesizes property-based tests from existing unit tests. Projects often have a suite of unit tests that have been collected over time, some of them checking specific and subtle cases. Our approach leverages existing unit tests to learn property-based tests that can be used to increase value coverage by orders of magnitude. Further, we show that our approach: (i) preserves the subtleties of the original test suite; and (ii) produces properties that cover a greater range of inputs than those in the example set.

The main idea is to use abstractions to over-approximate the concrete values of tests with similar structure. These abstractions are then used to produce appropriate value generators that can drive the synthesized property-based test.

We present JARVIS, a tool that synthesizes property-based tests from unit tests, while preserving the subtleties of the original unit tests. We evaluate JARVIS on tests from Apache projects, and show that it preserves these interesting tests while increasing value coverage by orders of magnitude.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Jsverify: Write powerful and concise tests

    Google Scholar 

  2. Junit. http://junit.org/

  3. Nunit. http://www.nunit.org/

  4. Scalacheck: Property-based testing for scala. http://www.scalacheck.org/

  5. Scalagen: Java to scala conversion

    Google Scholar 

  6. Alipour, M.A., Groce, A., Gopinath, R., Christi, A.: Generating focused random tests using directed swarm testing. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 70–81. ACM (2016)

    Google Scholar 

  7. Angeletti, D., Giunchiglia, E., Narizzano, M., Puddu, A., Sabina, S.: Automatic test generation for coverage analysis using CBMC. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2009. LNCS, vol. 5717, pp. 287–294. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04772-5_38

    Chapter  Google Scholar 

  8. Ball, T.: A Theory of predicate-complete test coverage and generation. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2004. LNCS, vol. 3657, pp. 1–22. Springer, Heidelberg (2005). https://doi.org/10.1007/11561163_1

    Chapter  Google Scholar 

  9. Brummayer, R., Biere, A.: Fuzzing and delta-debugging SMT solvers. In: Proceedings of the 7th International Workshop on Satisfiability Modulo Theories, pp. 1–5. ACM (2009)

    Google Scholar 

  10. Calamé, J.R., Ioustinova, N., van de Pol, J.: Automatic model-based generation of parameterized test cases using data abstraction. Electronic Notes in Theoretical Computer Science 191, 25–48 (2007)

    Article  Google Scholar 

  11. Carlier, M., Dubois, C., Gotlieb, A.: Constraint reasoning in FocalTest. In: ICSOFT (2010)

    Google Scholar 

  12. Chen, Y., Groce, A., Zhang, C., Wong, W.-K., Fern, X., Eide, E., Regehr, J.: Taming compiler fuzzers. In: Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2013, pp. 197–208. ACM, New York (2013)

    Google Scholar 

  13. Claessen, K., Hughes, J.: Quickcheck: a lightweight tool for random testing of haskell programs. ACM Sigplan Notices 46(4), 53–64 (2011)

    Article  Google Scholar 

  14. Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: POPL, pp. 238–252 (1977)

    Google Scholar 

  15. de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: Ramakrishnan, C.R., Rehof, J. (eds.) TACAS 2008. LNCS, vol. 4963, pp. 337–340. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  16. Derrick, J., et al.: Property-based testing - the protest project. In: de Boer, F.S., Bonsangue, M.M., Hallerstede, S., Leuschel, M. (eds.) FMCO 2009. LNCS, vol. 6286, pp. 250–271. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-17071-3_13

    Chapter  Google Scholar 

  17. Ernst, M.D., Cockrell, J., Griswold, W.G., Notkin, D.: Dynamically discovering likely program invariants to support program evolution. IEEE Transactions on Software Engineering 27(2), 99–123 (2001)

    Article  Google Scholar 

  18. Ferri-Ramírez, C., Hernández-Orallo, J., Ramírez-Quintana, M.J.: Incremental learning of functional logic programs. In: Kuchen, H., Ueda, K. (eds.) FLOPS 2001. LNCS, vol. 2024, pp. 233–247. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44716-4_15

    Chapter  Google Scholar 

  19. Fink, G., Bishop, M.: Property-based testing: a new approach to testing for assurance. ACM SIGSOFT Software Engineering Notes 22(4), 74–80 (1997)

    Article  Google Scholar 

  20. Flanagan, C., Leino, K.R.M.: Houdini, an annotation assistant for ESC/Java. In: Oliveira, J.N., Zave, P. (eds.) FME 2001. LNCS, vol. 2021, pp. 500–517. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-45251-6_29

    Chapter  Google Scholar 

  21. Francisco, M.A., López, M., Ferreiro, H., Castro, L.M.: Turning web services descriptions into quickcheck models for automatic testing. In: Proceedings of the Twelfth ACM SIGPLAN Workshop on Erlang, pp. 79–86. ACM (2013)

    Google Scholar 

  22. Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. In: Proceedings of the 19th International Symposium on Software Testing and Analysis, ISSTA 2010, pp. 147–158. ACM, New York (2010)

    Google Scholar 

  23. Fraser, G., Zeller, A.: Generating parameterized unit tests. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis, pp. 364–374. ACM (2011)

    Google Scholar 

  24. Gay, G., Staats, M., Whalen, M., Heimdahl, M.P.: Automated oracle data selection support. IEEE Transactions on Software Engineering 41(11), 1119–1137 (2015)

    Article  Google Scholar 

  25. Gerdes, A., Hughes, J., Smallbone, N., Wang, M.: Linking unit tests and properties. In: Proceedings of the 14th ACM SIGPLAN Workshop on Erlang, pp. 19–26. ACM (2015)

    Google Scholar 

  26. Godefroid, P., Kiezun, A., Levin, M.Y.: Grammar-based whitebox fuzzing. In: ACM Sigplan Notices, vol. 43, pp. 206–215. ACM (2008)

    Google Scholar 

  27. Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: Proceedings of the 38th Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, POPL 2011, pp. 317–330. ACM, New York (2011)

    Google Scholar 

  28. Gulwani, S., Harris, W.R., Singh, R.: Spreadsheet data manipulation using examples. Commun. ACM 55(8), 97–105 (2012)

    Article  Google Scholar 

  29. Hoffman, D., Strooper, P., White, L.: Boundary values and automated component testing. Software Testing, Verification and Reliability 9(1), 3–26 (1999)

    Article  Google Scholar 

  30. Hughes, J.: Quickcheck testing for fun and profit. In: Hanus, M. (ed.) PADL 2007. LNCS, vol. 4354, pp. 1–32. Springer, Heidelberg (2006). https://doi.org/10.1007/978-3-540-69611-7_1

    Chapter  Google Scholar 

  31. Le, V., Gulwani, S.: FlashExtract: a framework for data extraction by examples. In: O’Boyle, M.F.P., Pingali, K. (eds.) Proceedings of the 35th Conference on Programming Language Design and Implementation, pp. 55. ACM (2014)

    Google Scholar 

  32. Li, H., Thompson, S., Lamela Seijas, P., Francisco, M.A.: Automating property-based testing of evolving web services. In: Proceedings of the ACM SIGPLAN 2014 Workshop on Partial Evaluation and Program Manipulation, pp. 169–180. ACM (2014)

    Google Scholar 

  33. Lopes, N.P., Monteiro, J.: Weakest precondition synthesis for compiler optimizations. In: McMillan, K.L., Rival, X. (eds.) VMCAI 2014. LNCS, vol. 8318, pp. 203–221. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54013-4_12

    Chapter  Google Scholar 

  34. Löscher, A., Sagonas, K.: Targeted property-based testing. In: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 46–56. ACM (2017)

    Google Scholar 

  35. Majumdar, R., Xu, R.-G.: Directed test generation using symbolic grammars. In: Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering, pp. 134–143. ACM (2007)

    Google Scholar 

  36. Murray, K.S.: Multiple convergence: an approach to disjunctive concept acquisition. In: IJCAI, pp. 297–300. Citeseer (1987)

    Google Scholar 

  37. Nystrom, N., Clarkson, M.R., Myers, A.C.: Polyglot: an extensible compiler framework for java. In: Hedin, G. (ed.) CC 2003. LNCS, vol. 2622, pp. 138–152. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36579-6_11

    Chapter  Google Scholar 

  38. Odersky, M., Spoon, L., Venners, B.: Scala (2011). http://blog.typesafe.com/why-scala. Last Accessed: August 28, 2012

  39. Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: Proceedings of the 29th International Conference on Software Engineering, ICSE 2007, pp. 75–84. IEEE Computer Society, Washington, DC (2007)

    Google Scholar 

  40. Pandita, R., Xie, T., Tillmann, N., De Halleux, J.: Guided test generation for coverage criteria. In: 2010 IEEE International Conference on Software Maintenance (ICSM), pp. 1–10. IEEE (2010)

    Google Scholar 

  41. Papadakis, M., Sagonas, K.: A proper integration of types and function specifications with property-based testing. In: Proceedings of the 10th ACM SIGPLAN Workshop on Erlang, pp. 39–50. ACM (2011)

    Google Scholar 

  42. Peleg, H., Shoham, S., Yahav, E.: \(D^3\): data-driven disjunctive abstraction. In: Jobstmann, B., Leino, K.R.M. (eds.) VMCAI 2016. LNCS, vol. 9583, pp. 185–205. Springer, Heidelberg (2016). https://doi.org/10.1007/978-3-662-49122-5_9

    Chapter  Google Scholar 

  43. Pike, L.: Smartcheck: automatic and efficient counterexample reduction and generalization. In: ACM SIGPLAN Notices, vol. 49, pp. 53–64. ACM (2014)

    Google Scholar 

  44. Rojas, J.M., Fraser, G., Arcuri, A.: Automated unit test generation during software development: a controlled experiment and think-aloud observations. In: Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, pp. 338–349. ACM, New York (2015)

    Google Scholar 

  45. Saff, D., Boshernitsan, M., Ernst, M.D.: Theories in practice: easy-to-write specifications that catch bugs (2008)

    Google Scholar 

  46. Shamshiri, S., Just, R., Rojas, J.M., Fraser, G., McMinn, P., Arcuri, A.: Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges (t). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 201–211. IEEE (2015)

    Google Scholar 

  47. Sharma, R., Aiken, A.: From invariant checking to invariant inference using randomized search. In: Biere, A., Bloem, R. (eds.) CAV 2014. LNCS, vol. 8559, pp. 88–105. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-08867-9_6

    Google Scholar 

  48. Singh, R., Gulwani, S.: Predicting a correct program in programming by example. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 398–414. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_23

    Chapter  Google Scholar 

  49. Srivastava, S., Gulwani, S.: Program verification using templates over predicate abstraction. ACM Sigplan Notices 44, 223–234 (2009). ACM

    Article  Google Scholar 

  50. Thummalapenta, S., Marri, M.R., Xie, T., Tillmann, N., de Halleux, J.: Retrofitting unit tests for parameterized unit testing. In: Giannakopoulou, D., Orejas, F. (eds.) FASE 2011. LNCS, vol. 6603, pp. 294–309. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19811-3_21

    Chapter  Google Scholar 

  51. de Halleux, J., Tillmann, N.: Parameterized unit testing with pex (Tutorial). In: Beckert, B., Hähnle, R. (eds.) TAP 2008. LNCS, vol. 4966, pp. 171–181. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-79124-9_12

    Chapter  Google Scholar 

  52. Tillmann, N., Schulte, W.: Parameterized unit tests. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 253–262. ACM (2005)

    Google Scholar 

  53. Tillmann, N., Schulte, W.: Parameterized unit tests with unit meister. In: ACM SIGSOFT Software Engineering Notes, vol. 30, pp. 241–244. ACM (2005)

    Google Scholar 

  54. Wang, C., Pastore, F., Goknil, A., Briand, L., Iqbal, Z.: Automatic generation of system test cases from use case specifications. In: Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, pp. 385–396. ACM, New York (2015)

    Google Scholar 

  55. Xie, T., Tillmann, N., de Halleux, J., Schulte,W.: Mutation analysis of parameterized unit tests. In: 2009 International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2009, pp. 177–181. IEEE (2009)

    Google Scholar 

  56. Yatoh, K., Sakamoto, K., Ishikawa, F., Honiden, S.: Arbitcheck: a highly automated property-based testing tool for Java. In: Proceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops, ICSTW 2014, pp. 405–412. IEEE Computer Society, Washington, DC (2014)

    Google Scholar 

  57. Yatoh, K., Sakamoto, K., Ishikawa, F., Honiden, S.: Feedback-controlled random test generation. In: Proceedings of the 2015 International Symposium on Software Testing and Analysis, ISSTA 2015, Baltimore, MD, USA, July 12–17, 2015, pp. 316–326 (2015)

    Google Scholar 

  58. Yoshida, H., Tokumoto, S., Prasad, M.R., Ghosh, I., Uehara, T.: FSX: fine-grained incremental unit test generation for C/C++ programs. In: Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 106–117. ACM (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Technion, Haifa, Israel

    Hila Peleg, Dan Rasin & Eran Yahav

Authors
  1. Hila Peleg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dan Rasin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eran Yahav
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hila Peleg .

Editor information

Editors and Affiliations

  1. University of Texas , Austin, Texas, USA

    Isil Dillig

  2. University of California , Los Angeles, California, USA

    Jens Palsberg

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Peleg, H., Rasin, D., Yahav, E. (2018). Generating Tests by Example. In: Dillig, I., Palsberg, J. (eds) Verification, Model Checking, and Abstract Interpretation. VMCAI 2018. Lecture Notes in Computer Science(), vol 10747. Springer, Cham. https://doi.org/10.1007/978-3-319-73721-8_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73721-8_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73720-1

  • Online ISBN: 978-3-319-73721-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation