Advertisement

Survey on test data generation tools

An evaluation of white- and gray-box testing tools for C#, C++, Eiffel, and Java
  • Stefan J. Galler
  • Bernhard K. Aichernig
Regular Paper

Abstract

Automating the process of software testing is a very popular research topic and of real interest to industry. Test automation can take part on different levels, e.g., test execution, test case generation, test data generation. This survey gives an overview of state-of-the art test data generation tools, either academic or commercial. The survey focuses on white- and gray-box techniques. The list of existing tools was filtered with respect to their public availability, their maturity, and activity. The remaining seven tools, i.e., AgitarOne, CodePro AnalytiX, AutoTest, C++test, Jtest, RANDOOP, and PEX, are briefly introduced and their evaluation results are summarized. For the evaluation we defined 31 benchmark tests, which check the tools capabilities to generate test data that satisfies a given specification: 24 primitive type benchmarks and 7 non-primitive type and more complex with respect to the specification benchmarks. Most of the commercial tools implement a test data strategy that uses constant values found in the method under test or values that are slightly modified by means of mathematical operations. This strategy turns out to be very effective. In general, all tools that combine multiple techniques perform very well. For example PEX uses constraint solving techniques, but in cases where the constraint solver reaches its limitations it uses random based techniques to overcome those limitations. Especially, the two commercial tools AgitarOne and PEX that combine multiple approaches to test data generation are able to pass all 31 tests. This survey reflects the status in 2011.

Keywords

Test data generation Tool evaluation Java C++ C# Eiffel 

Notes

Acknowledgments

The authors wish to thank the anonymous referees for their detailed and constructive feedback in order to improve the paper. The research herein is partially conducted within the competence network Softnet Austria (http://www.soft-net.at) and funded by the Austrian Federal Ministry of Economics (bm:wa), the province of Styria, the Steirische Wirtschaftsförderungsgesellschaft mbH (SFG), and the city of Vienna in terms of the center for innovation and technology (ZIT).

References

  1. 1.
    Ammann, P., Offutt, J.: Introduction to Software Testing. Cambridge University Press, Cambridge (2008)CrossRefzbMATHGoogle Scholar
  2. 2.
    Thomas A., John, H., Joakim, J., Ulf, W.: Testing Telecoms Software With Quviq QuickCheck. In: 2006 ACM SIGPLAN workshop on Erlang, Portland, Oregon, USA, 2006. ACM, New YorkGoogle Scholar
  3. 3.
    Barnett, M., Fähndrich, M., de Halleux, J., Logozzo, F., Tillmann, N.: Exploiting the synergy between automated-test-generation and programming-by-contract. In: Proceedings of the 31st International Conference on Software Engineering (ICSE’2009). IEEE (2009)Google Scholar
  4. 4.
    Mike, B., Leino, K.R.M., Wolfram, Schulte.: The Spec# programming system: an overview. In: Construction and Analysis of Safe, Secure, and Interoperable Smart Devices (CASSIS 2004), vol. 3362 of Lecture Notes in Computer Science, pp. 49–69. Springer, Berlin (2005)Google Scholar
  5. 5.
    Boshernitsan, M., Doong, R., Savoia, A.: From Daikon to agitator: lessons and challenges in building a commercial tool for developer testing. In: 2006 International Symposium on Software Testing and Analysis, pp. 169–180. ACM Press, New York (2006)Google Scholar
  6. 6.
    Bourdonov, I.B., Kossatchev, A., Kuliamin, V.V., Petrenko, A.: UniTesK test suite architecture. In: FME 2002: Formal Methods Getting IT Right, vol. 2391 of Lecture Notes in Computer Science, pp. 121–152. Springer (2002)Google Scholar
  7. 7.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: automated testing based on Java predicates. In: International Symposium on Software Testing and Analysis, pp. 123–133. ACM Press, New York (2002)Google Scholar
  8. 8.
    Brat, G., Drusinsky, D., Giannakopoulou, D., Goldberg, A., Havelund, K., Lowry, M., Pasareanu, C., Venet, A., Visser, W., Washington, R.: Experimental evaluation of verification and validation tools on Martian Rover software. Formal Methods Syst. Design 25(2/3), 167–198 (2004)CrossRefzbMATHGoogle Scholar
  9. 9.
    Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: USENIX Symposium on Operating Systems Design and Implementation, pp. 209–224. USENIX (2008)Google Scholar
  10. 10.
    Cadar, C., Ganesh, V., Pawlowski, P.M., Dill, D.L., Engler, D.R.: EXE: automatically generating inputs of death. ACM Trans. Inf. Syst. Secur. 12(2), 322–335 (2008)CrossRefGoogle Scholar
  11. 11.
    Campbell, C., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., Veanes, M.: Model-based testing of object-oriented reactive systems with Spec explorer. Technical report, Microsoft Research, Redmond (2005)Google Scholar
  12. 12.
    Ciupa, I., Leitner, A.: Automatic testing based on design by contract. In: Proceedings of Net. ObjectDays 2005, pp. 545–557 (2005)Google Scholar
  13. 13.
    Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: adaptive random testing for object-oriented software. In: 30th International Conference on Software Engineering, pp. 71–80. ACM, New York (2008)Google Scholar
  14. 14.
    Csallner, C., Smaragdakis, Y.: JCrasher: an automatic Robustness tester for Java. Softw.: Pract. Exp. 34(11), 1025–1050 (2004)Google Scholar
  15. 15.
    Csallner, C., Smaragdakis, Y.: Check ’n’ crash: combining static checking and testing. In: 27th ACM/IEEE International Conference on Software Engineering, pp. 422–431. ACM, New York (2005)Google Scholar
  16. 16.
    de Halleux, J., Tillmann, N.: Moles: tool-assisted environment isolation with closures. Vol. 6141 of Lecture Notes in Computer Science, pp. 253–270. Springer, Berlin/Heidelberg (2010)Google Scholar
  17. 17.
    Ernst, M.D., CockrelI, J., Griswold, W.G., Notkin, D.: Dynamically discovering likely program invariants to support program evolution. In: 21st International Conference on Software Engineering, pp. 213–222, Los Alamitos, CA, USA, 1999. IEEE (1999)Google Scholar
  18. 18.
    Cormac, F., Rustan, K., Leino, M., Lillibridge, M., Nelson, G., Saxe, J.B., Stata, R.: Extended static checking for Java. ACM SIGPLAN Not. Confer. Program. Lang. Design Implement. 37(5), 234–245 (2002)Google Scholar
  19. 19.
    Freese, T.: EasyMock: dynamic mock objects for JUnit. In: 3nd International Conference on Extreme Programming and Flexible Processes in Software Engineering (XP 2002), pp. 2–5 (2002)Google Scholar
  20. 20.
    Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 213–223 (2005)Google Scholar
  21. 21.
  22. 22.
    Grieskamp, W.: Multi-paradigmatic model-based testing. In: Formal Approaches to Software Testing and Runtime Verification, vol. 4262 of Lecture Notes in Computer Science, pp. 1–19. Springer, Berlin (2006)Google Scholar
  23. 23.
    Grieskamp, W., Qu, X., Wei, X., Kicillof, N., Cohen, M.: Interaction coverage meets path coverage by SMT constraint solving. In: Testing of Software and Communication Systems, vol. 58262 of Lecture Notes in Computer Science, pp. 97–112. Springer, Berlin (2009)Google Scholar
  24. 24.
    Huima, A.: Implementing conformiq Qtronic. In: Testing of Software and Communicating Systems, vol. 4581 of Lecture Notes in Computer Science, pp. 1–12. Springer, Berlin/Heidelberg (2007)Google Scholar
  25. 25.
    Google Inc. Codepro analytix user guide. http://developers.google.com/java-dev-tools/codepro/doc/. Accessed Nov 2011
  26. 26.
    Jaffuel, E., Legeard, B.: LEIRIOS test generator: automated test generation from B models. In: B 2007: Formal Specification and Development in B, vol. 4355 of Lecture Notes in Computer Science, pp. 277–280. Springer, Berlin/Heidelberg (2006)Google Scholar
  27. 27.
    Kuliamin, V.V., Petrenko, A.K., Kossatchev, A.S., Bourdonov, I.B.: UniTesK: model based testing in industrial practice. In: 1st European Conference on Model Driven, Software Engineering, pp. 55–63 (2003)Google Scholar
  28. 28.
    Legeard, B., Peureux, F., Utting, M.: Automated boundary testing from Z and B. In: Eriksson L.-H., Lindsay, P. (eds.) Formal Methods, vol. 2391 of Lecture Notes in Computer Science, pp 221–236. Springer (2002)Google Scholar
  29. 29.
    Meyer, B., Ciupa, I., Leitner, A., Liu, L.: Automatic testing of object-oriented software. In: SOFSEM 2007: Theory and Practice of Computer Science, vol. 4362 of Lecture Notes in Computer Science, pp. 114–129. Springer, Heidelberg (2007)Google Scholar
  30. 30.
    Meyers, S.: Effective C++: 55 specific ways to improve your programs and design, 3rd edn. Addison-Wesley Professional, Boston (2005)Google Scholar
  31. 31.
    De Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: TACAS’08: Tools and Algorithms for the Construction and Analysis of Systems, vol. 4963 of Lecture Notes in Computer Science, pp. 337–340. Springer, Heidelberg (2008)Google Scholar
  32. 32.
    Myers, G.J., Sandler, C., Badgett, T., Thomas, T.M.: The Art of Software Testing, 2nd edn. Wiley, New York (2004)Google Scholar
  33. 33.
    Pacheco, C., Ernst, M.D.: Eclat: automatic generation and classification of test inputs. In: ECOOP 2005: Object-Oriented Programming, vol. 3568 of Lecture Notes in Computer Science, pp. 504–527. Springer (2005)Google Scholar
  34. 34.
    Pacheco, C., Ernst, M.D.: Randoop: feedback-directed random testing for Java. In: OOPSLA 2007: Conference on Object Oriented Programming Systems Languages and Applications, pp. 815–816. ACM, New York (2007)Google Scholar
  35. 35.
    Pacheco, C., Lahiri, S.K., Ball, T.: Finding errors in.NET with feedback-directed random testing. ISSTA’08: International Symposium on Software Testing and, Analysis, pp. 87–96 (2008)Google Scholar
  36. 36.
    Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: Proceedings of the 29th International Conference on Software Engineering, pp. 75–84, Minneapolis, MN, USA, 2007. IEEE (2007)Google Scholar
  37. 37.
    Parasoft. Parasoft C++test user’s guide (2010)Google Scholar
  38. 38.
    Parasoft. http://www.parasoft.com. Accessed Nov 2011
  39. 39.
    Parasoft. Using design by contract to automate Java software and component testing. http://www.parasoft.com/jsp/products/article.jsp?articleId=579&product=Jcontract. Accessed Nov 2011
  40. 40.
    Penix, J., Visser, W., Park, S., Pasareanu, C., Engstrom, E., Larson, A., Weininger, N.: Verifying time partitioning in the DEOS scheduling Kernel. Formal Methods Syst. Design 26(2), 103–135 (2005) Google Scholar
  41. 41.
    Microsoft Research. Advanced concepts: parameterized unit testing with Microsoft Pex. http://research.microsoft.com/en-us/projects/pex/pexconcepts.pdf (2010)
  42. 42.
    Sen, K., Agha, G.: CUTE and jCUTE: concolic unit testing and explicit path model-checking tools. In: 18th International Conference on Computer Aided Verification, vol. 4144 of Lecture Notes in Computer Science, pp. 419–423, Seattle, Washington, USA, 2006. Springer, New York (2006)Google Scholar
  43. 43.
    Sen, K., Marinov, D., Agha, G.: CUTE: a concolic unit testing engine for C. In: 10th European Software Engineering Conference, vol. 30 of ACM SIGSOFT Software Engineering Notes, pp. 263–272, Lisbon, Portugal, 2005. ACM, New York (2005)Google Scholar
  44. 44.
    SMTCOMP. Call for Entrants. http://www.smtcomp.org/2010/call10.txt (2010)
  45. 45.
    Tillmann N., De Halleux, J.: Pex–White Box Test Generation for.NET. In: Proceedings of the 2nd international conference on tests and proofs (TAP 2008), vol. 4966 of Lecture Notes in Computer Science, pp. 134–153. Springer (2008)Google Scholar
  46. 46.
    Tillmann, N., Grieskamp, W., Schulte, W.: Parameterized unit tests. SIGSOFT Softw. Eng. Notes 30(5), 253–262 (2005)CrossRefGoogle Scholar
  47. 47.
    Veanes, M., de Halleux P., Tillmann, N.: Rex: Symbolic Regular Expression Explorer. In: 2010 Third International Conference on Software Testing, Verification and Validation, pp. 498–507. IEEE (2010)Google Scholar
  48. 48.
    Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with Spec explorer. In: Formal Methods and Testing, vol. 4949 of Lecture Notes in Computer Science, pp. 39–76. Springer (2008)Google Scholar
  49. 49.
    Visser, W., Havelund, K., Brat, G.: Model checking programs. Autom. Softw. Eng. 10(2), 203–232 (2003)CrossRefGoogle Scholar
  50. 50.
    Visser, W., Psreanu, C.S.: Test input generation with Java PathFinder. ACM SIGSOFT Softw. Eng. Notes 29(4), 97–107 (2004)CrossRefGoogle Scholar
  51. 51.
    Wei, Y., Gebhardt, S., Oriol, M., Meyer, B.: Satisfying test preconditions through guided object selection. In: 3rd International Conference on Software Testing, Verification and Validation, pp. 1–10, Paris, France, 2010. IEEE (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Graz University of TechnologyGrazAustria

Personalised recommendations