Advertisement

KeYGenU: combining verification-based and capture and replay techniques for regression unit testing

  • Bernhard Beckert
  • Christoph Gladisch
  • Shmuel Tyszberowicz
  • Amiram Yehudai
Original Article

Abstract

Unit testing plays a major role in the software development process. Two essential criteria to achieve effective unit testing are: (1) testing each unit in isolation from other parts of the program and (2) achieving high code coverage. The former requires a lot of extra work such as writing drivers and stubs, whereas the latter is difficult to achieve when manually writing the tests. When changing existing code it is advocated to run the unit tests to avoid regression bugs. However, in many cases legacy software has no unit tests. Writing those tests from scratch is a hard and tedious process, which might not be cost-effective. This paper presents a tool chain approach that combines verification-based testing (VBT) and capture and replay (CaR) test generation methods. We have built a concrete tool chain, KeYGenU, which consists of two existing tools—KeY and GenUTest. The KeY system is a deductive verification and test-generation tool. GenUTest automatically generates JUnit tests for a correctly working software. This combination provides isolated unit test suites with high code-coverage. The generated tests can also be used for regression testing.

Keywords

Unit testing Verification-based testing Regression tests Isolated tests Test coverage 

Notes

Acknowledgments

We are grateful to Benny Pasternak for modifying GenUTest as needed to combine it with KeY. We also thank Jean-Louis Lanet for providing the banking application that served as our case study.

References

  1. AspectJ. http://www.eclipse.org/aspectj. Visited May 2011
  2. Beckert B, Gladisch C (2007) White-box testing by combining deduction-based specification extraction and black-box testing. In: Gurevich Y, Meyer B (eds) Proceedings, tests and proofs, first international conference, TAP 2007, Zurich, Switzerland, volume 4454 of LNCS. Springer, pp 207–216Google Scholar
  3. Beckert B, Hähnle R, Schmitt PH (eds) (2007) Verification of object-oriented software: the KeY approach. LNCS 4334. SpringerGoogle Scholar
  4. Burdy L, Requet A, Lanet J-L (2003) Java applet correctness: a developer-oriented approach. In: Araki K, Gnesi S, Mandrioli D (eds) Formal methods, international symposium of formal methods Europe, FME 2003, Pisa, Italy, volume 2805 of LNCS. Springer, pp 422–439Google Scholar
  5. Burstall RM (1974) Program proving as hand simulation with a little induction. In: Information processing ’74. Elsevier/North-Holland, pp 308–312Google Scholar
  6. Clifton M (2004) Advanced unit test, part V—unit test patterns, January (2004). http://www.codeproject.com/gen/design/autp5.asp. Visited May 2011
  7. Cok DR, Kiniry J (2004) ESC/Java2: uniting ESC/Java and JML. In: Barthe G, Burdy L, Huisman M, Lanet J-L, Muntean T (eds) Proceedings, construction and analysis of safe, secure, and interoperable smart devices, international workshop, CASSIS 2004, Marseille, France, volume 3362 of LNCS. Springer, pp 108–128Google Scholar
  8. Deng X, Robby, Hatcliff J (2006) Kiasan: a verification and test-case generation framework for Java based on symbolic execution. In: Proceedings, leveraging applications of formal methods, second international symposium, ISoLA 2006, Paphos, Cyprus. IEEE Computer Society, pp 137–137Google Scholar
  9. Deng X, Robby, Hatcliff J (2007) Kiasan/KUnit: automatic test case generation and analysis feedback for open object-oriented systems. In: TAICPART-MUTATION ’07: Proceedings of the testing: academic and industrial conference practice and research techniques—MUTATION. Washington, DC, USA. IEEE Computer Society, pp 3–12Google Scholar
  10. du Bousquet L, Ledru Y, Maury O, Oriat C, Lanet J-L (2004) Case study in JML-based software validation. In: Proceedings, 19th IEEE international conference on automated software engineering, ASE 2004, Linz, Austria. IEEE Computer Society, pp 294–297Google Scholar
  11. EclEmma. http://www.eclemma.org. Visited May 2011
  12. Elbaum S, Chin H, Dwyer M, Dokulil J (2006) Carving differential unit test cases from system test cases. In: Young M, Devanbu PT (eds.) Proceedings of the 14th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2006, Portland, OR, USA. ACM, pp 253–264Google Scholar
  13. Elbaum SG, Chin HN, Dwyer MB, Jorde M (2009) Carving and replaying differential unit test cases from system test cases. IEEE Trans Softw Eng 35(1):29–45Google Scholar
  14. Elssamadisy A (2009) Agile adoption patterns: a roadmap to organizational success. Pearson educationGoogle Scholar
  15. Engel C, Hähnle R (2007) Generating unit tests from formal proofs In: Gurevich Y, Meyer B (eds) Proceedings, tests and proofs, first international conference, TAP 2007, Zurich, Switzerland, volume. 4454 of LNCS. Springer, pp 169–188Google Scholar
  16. Engel C, Gladisch C, Klebanov V, Rümmer P (2008) Integrating verification and testing of object-oriented software. In: Beckert B, Hähnle R (eds) Proceedings, tests and proofs, second international conference, TAP 2008, Prato, Italy, volume 4966 of LNCS. Springer, pp 182–191Google Scholar
  17. Ernst MD (2004) Static and dynamic analysis: synergy and duality. In: Flanagan C, Zeller A (eds.) Proceedings of the 2004 ACM SIGPLAN-SIGSOFT workshop on program analysis for software tools and engineering, PASTE’04. Washington, DC, USA. ACM, p 35Google Scholar
  18. Extreme Programming. http://www.extremeprogramming.org. Visited May 2011
  19. Fowler M (2000) Refactoring: Improving the Design of Existing Code. Addison-WesleyGoogle Scholar
  20. Gladisch C (2008) Verification-based testing for full feasible branch coverage. In: Cerone A, Gruner S (eds) Proceedings, sixth IEEE international conference on software engineering and formal methods, SEFM 2008, Cape Town, South Africa. IEEE Computer Society, pp 159–168Google Scholar
  21. Gladisch C (2011) Verification-based software-fault detection. PhD thesis, Karlsruhe Institute of Technology (KIT). http://digbib.ubka.uni-karlsruhe.de/volltexte/1000023056
  22. Gladisch C, Tyszberowicz SS, Beckert B, Yehudai A (2010) Generating regression unit tests using a combination of verification and capture & replay. In: Proceedings, tests and proofs, 4th international conference, TAP2010, Málaga, Spain, volume 6143 of LNCS. Springer, pp 61–76Google Scholar
  23. Graves TL, Harrold MJ, Kim J-M, Porter A, Rothermel G (2001) An empirical study of regression test selection techniques. TOSEM 10(2):184–208CrossRefzbMATHGoogle Scholar
  24. Hamill P (2004) Unit test frameworks. O’ReillyGoogle Scholar
  25. Harel D, Kozen D, Tiuryn J (2000) Dynamic logic. MIT PressGoogle Scholar
  26. Harrold MJ, Jones JA, Li T, Liang D, Orso A, Pennings M, Sinha S, Spoon SA, Gujarathi A (2001) Regression test selection for Java software. SIGPLAN Not 36(11):312–326CrossRefGoogle Scholar
  27. Heisel M, Reif W, Stephan W (1987) Program verification by symbolic execution and induction. In: Morik K (ed) Proceedings, 11th German workshop on artificial intelligence, GWAI 87, volume 152 of Informatik Fachberichte. Springer, pp 201–210Google Scholar
  28. Hoare CAR (1969) An axiomatic basis for computer programming. Commun ACM 12(10):576–580CrossRefzbMATHGoogle Scholar
  29. HP WinRunner software. http://www.cbueche.de/WinRunner%20User%20Guide.pdf. Visited May 2011
  30. Husted T, Massol V (2003) JUnit in Action. Manning Publications Co.Google Scholar
  31. Intel VTune Performance Analyzer http://www.intel.com/cd/software/products/asmo-na/eng/239144.htm. Visited May 2011
  32. JUnit. http://www.junit.org. Visited May 2011
  33. Laddad R (2003) Aspect J in action: practical aspect-oriented programming. ManningGoogle Scholar
  34. Larsson D, Mostowski W (2004) Specifying Java Card API in OCL. In: Schmitt PH (ed) OCL 2.0 workshop at UML 2003, volume 102 of ENTCS. Elsevier, pp. 3–19, November 2004Google Scholar
  35. Mackinnon T, Freeman S, Craig P(2001) Endo-testing: unit testing with mock objects. In: Extreme programming examined. Addison-Wesley, pp 287–301Google Scholar
  36. Meyer B (1997) Design by contract: making object-oriented programs that work. In: Proceedings, TOOLS 1997: 25th international conference on technology of object-oriented languages and systems, Melbourne, Australia. IEEE Computer Society, p 360Google Scholar
  37. Orso A, Kennedy B (2005) Selective capture and replay of program executions. In: Proceedings of the 2005 workshop on dynamic analysis. pp 1–7Google Scholar
  38. Pasternak B, Tyszberowicz S, Yehudai A (2009) GenUTest: a unit test and mock aspect generation tool. J Softw Tools Technol Transfer 11(4):273–290CrossRefGoogle Scholar
  39. Saff D, Artzi S, Perkins JH, Ernst MD (2005) Automatic test factoring for Java. In: Redmiles DF, Ellman T, Zisman A (eds) Proceedings, 20th IEEE/ACM international conference on automated software engineering, ASE 2005, Long Beach, CA, USA. ACM, pp 114–123Google Scholar
  40. Smaragdakis Y, Csallner C (2007) Combining static and dynamic reasoning for bug detection. In: Gurevich Y, Meyer B (eds) Proceedings, tests and proofs, first international conference, TAP 2007, Zurich, Switzerland, volume 4454 of LNCS. Springer, pp 1–16Google Scholar
  41. Steven J, Chandra P, Fleck B, Podgurski A (2000) jRapture: a Capture/Replay tool for observation-based testing. In: Proceedings of the international symposium on software testing and analysis, pp 158–167Google Scholar
  42. Strichman O (2009) Regression verification: proving the equivalence of similar programs. In: Bouajjani A, Maler O (eds) Proceedings, computer aided verification, 21st international conference, CAV 2009, Grenoble, France, volume 5643 of LNCS. Springer, pp 63–68Google Scholar
  43. Taneja K, Xie T (2008) DiffGen: automated regression unit-test generation. In: Proceedings, 23rd IEEE/ACM international conference on automated software engineering, ASE 2008, L’Aquila, Italy. IEEE Computer SocietyGoogle Scholar
  44. Tillmann N, de Halleux J (2008) Pex-white box test generation for NET. In: Beckert B, Hähnle R (eds) Proceedings, tests and proofs, second international conference, TAP 2008, Prato, Italy, volume 4966 of LNCS. Springer, pp 134–153Google Scholar
  45. Tonin I (2007) Verifying the Mondex Case Study—the KeY approach. Technical Report ISSN: 1432-7864, Fakultät für Informatik (Fak. f. Informatik) Institut für Theoretische Informatik (ITI)Google Scholar
  46. The Verisoft Project. http://www.verisoft.de. Visited May 2011
  47. Visser W, Pasareanu CS, Khurshid S (2004) Test input generation with Java PathFinder. In: Avrunin GS, Rothermel G (eds) Proceedings of the ACM/SIGSOFT international symposium on software testing and analysis, ISSTA 2004, Boston, MA, USA. ACM, pp 97–107Google Scholar
  48. Wenzel M, Paulson LC, Nipkow T (2008) The Isabelle framework. In: Mohamed OA, Muñoz C, Tahar S (eds) Proceedings, theorem proving in higher order logics, 21st international conference, TPHOLs 2008, Montreal, Canada, volume 5170 of LNCS. Springer, pp 33–38Google Scholar
  49. Xie T (2006) Augmenting automatically generated unit-test suites with regression oracle checking. In: Thomas D (ed) Proceedings, European conference object-oriented programming, ECOOP, Nantes, France, volume 4067 of LNCS. Springer, pp 380–403Google Scholar
  50. Yuan H, Xie T (2006) Substra: a framework for automatic generation of integration tests. In: Zhu H, Horgan JR, Cheung S-C, Li JJ (eds) Proceedings of the 2006 international workshop on automation of software test, AST 2006, Shanghai, China. ACM, pp 64–70Google Scholar

Copyright information

© The Society for Reliability Engineering, Quality and Operations Management (SREQOM), India and The Division of Operation and Maintenance, Lulea University of Technology, Sweden 2011

Authors and Affiliations

  • Bernhard Beckert
    • 1
  • Christoph Gladisch
    • 1
  • Shmuel Tyszberowicz
    • 2
  • Amiram Yehudai
    • 3
  1. 1.Department of InformaticsKarlsruhe Institute of TechnologyKarlsruheGermany
  2. 2.School of Computer ScienceThe Academic College of Tel Aviv YaffoTel Aviv YaffoIsrael
  3. 3.School of Computer ScienceTel Aviv UniversityTel AvivIsrael

Personalised recommendations