Using Contracts and Boolean Queries to Improve the Quality of Automatic Test Generation

  • Lisa (Ling) Liu
  • Bertrand Meyer
  • Bernd Schoeller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4454)

Abstract

Since test cases cannot be exhaustive, any effective test case generation strategy must identify the execution states most likely to uncover bugs. The key issue is to define criteria for selecting such interesting states.

If the units being tested are classes in object-oriented programming, it seems attractive to rely on the boolean queries present in each class, which indeed define criteria on the states of the corresponding objects, and — in contract-equipped O-O software — figure prominently in preconditions, postconditions and invariants. As these queries are part of the class specification and hence relevant to its clients, one may conjecture that the resulting partition of the state space is also relevant for tests.

We explore this conjecture by examining whether relying on the boolean queries of a class to extract abstract states improves the results of black-box testing. The approach uses constraint-solving and proof techniques to generate objects that satisfy the class invariants, then performs testing by relying on postconditions as test oracles. The resulting strategy, in our experiments on library classes used in production software, finds significantly more bugs than random testing.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Barnettl, M., Rustan, K., Leinol, M., Schultel, W.: The Spec# programming system: An overview. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, Springer, Heidelberg (2005)Google Scholar
  2. 2.
    Ball, T.: A theory of predicate-complete test coverage and generation. In: 3rd International Symposium on Formal Methods for Components and Objects, pp. 1–22 (2004)Google Scholar
  3. 3.
    Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on Java predicates. In: ISSTA 2002. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 123–133. ACM Press, New York (2002)CrossRefGoogle Scholar
  4. 4.
    Csallner, C., Smaragdakis, Y.: Check ’n’ crash: combining static checking and testing. In: Inverardi, P., Jazayeri, M. (eds.) ICSE 2005. LNCS, vol. 4309, pp. 422–431. Springer, Heidelberg (2006)Google Scholar
  5. 5.
    Ciupa, I., Leitner, A.: Automatic testing based on design by contract. In: Proceedings of Net.ObjectDays 2005 (6th Annual International Conference on Object-Oriented and Internet-based Technologies, Concepts and Applications for a Networked World), pp. 545–557 (2005)Google Scholar
  6. 6.
    Detlefs, D., Nelson, G., Saxe, J.B.: Simplify: A theorem prover for program checking. Technical Report HPL-2003-148, HP Labs (2003), http://research.compaq.com/SRC/esc/Simplify.html
  7. 7.
    Duran, J., Ntafos, S.: An evaluation of random testing. IEEE Transactions on Software Engineering SE-10, 438–444 (1984)CrossRefGoogle Scholar
  8. 8.
    Hamie, A.: Towards verifying Java realization of OCL-constrained design models using JML. In: Proceedings of 6th IASTED International Conference on Software Engineering and Applications, ACTA Press, MIT, Cambridge, MA, USA (2002)Google Scholar
  9. 9.
    Hamlet, D., Taylor, R.: Partition testing does not inspire confidence. IEEE Transactions on Software Engineering 16(12), 1402–1411 (1990)CrossRefMathSciNetGoogle Scholar
  10. 10.
    Hamlet, R.: Random testing. In: Marciniak, J. (ed.) Encyclopedia of Software Engineering, pp. 970–978. Wiley, Chichester (1994)Google Scholar
  11. 11.
    Jackson, D.: Alloy: Alightweight object modeling notation. ACM Trans. Soft. Eng. Methodology 11(2), 256–290 (2002)CrossRefGoogle Scholar
  12. 12.
    Kramer, R.: iContract - the JavaTM design by contractTM tool. In: Proceedings of Object-Oriented Language and Systems, pp. 295–307. IEEE Computer Society, Washington, DC, USA (1998)Google Scholar
  13. 13.
    Leavens, G.T., Baker, A.L.: Enhancing the pre- and postcondition technique for more expressive specifications. In: World Congress on Formal Methods, pp. 1087–1106 (1999)Google Scholar
  14. 14.
    Leavens, G.T., Cheon, Y., Clifton, C., Ruby, C., Cok, D.R.: How the design of jml accommodates both runtime assertion checking and formal verification. In: de Boer, F.S., Bonsangue, M.M., Graf, S., de Roever, W.-P. (eds.) FMCO 2002. LNCS, vol. 2852, pp. 262–284. Springer, Heidelberg (2003)Google Scholar
  15. 15.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - A survey. In: Proc. The IEEE, pp. 1090–1123. IEEE Computer Society Press, Los Alamitos (1996)Google Scholar
  16. 16.
    Liu, L., Leitner, A., Offutt, J.: Using contracts to automate forward class testing. Journal of System and Software (submitted)Google Scholar
  17. 17.
    Meyer, B.: Reusable Software: The Base Object-Oriented Libraries. Prentice Hall, Englewood Cliffs (1994)Google Scholar
  18. 18.
    Meyer, B.: Object-Oriented Software Construction, 2nd edn. Prentice-Hall, Englewood Cliffs (1997)MATHGoogle Scholar
  19. 19.
    Meyer, B.: Eiffel: The Language, Prentice Hall, 1991, revised edn. in progress at (2006), http://se.ethz.ch/~meyer/ongoing/etl/
  20. 20.
    Marinov, D., Khurshid, S.: TestEra: A novel framework for automated testing of Java programs. In: ASE. Proc. 16th IEEE International Conference on Automated Software Engineering, pp. 22–34. IEEE Computer Society Press, Los Alamitos (2001)Google Scholar
  21. 21.
    Nimmer, J.W., Ernst, M.D.: Invariant inference for static checking: An empirical evaluation. In: Daemen, J., Rijmen, V. (eds.) FSE 2002. LNCS, vol. 2365, pp. 11–20. Springer, Heidelberg (2002)Google Scholar
  22. 22.
    Ostrand, T.J., Balcer, M.J.: The Category-Partition method for specifying and generating functional test. Comm. ACM 31(6), 676–686 (1988)CrossRefGoogle Scholar
  23. 23.
    Richtersl, M., Gogolla, M.: On formalizing the UML object constraint language OCL. In: Ling, T.-W., Ram, S., Lee, M.L. (eds.) Conceptual Modeling – ER 1998. LNCS, vol. 1507, Springer, Heidelberg, Singapore (1998)Google Scholar
  24. 24.
  25. 25.
    Whaley, J., Martin, M.C., Lam, M.S.: Automatic extraction of object-oriented component interface. In: ISSTA 2002, pp. 218–228 (2002)Google Scholar
  26. 26.
    Xie, T., Notkin, D.: Automatic extraction of object-oriented observer abstractions from unit-test executions. In: Davies, J., Schulte, W., Barnett, M. (eds.) ICFEM 2004. LNCS, vol. 3308, pp. 290–305. Springer, Heidelberg (2004)Google Scholar
  27. 27.
    Yorsh, G., Ball, T., Sagiv, M.: Testing, abstraction, theorem proving: better together! In: ISSTA 2006, pp. 145-156 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Lisa (Ling) Liu
    • 1
  • Bertrand Meyer
    • 1
  • Bernd Schoeller
    • 1
  1. 1.Chair of Software Engineering, ETH ZurichSwitzerland

Personalised recommendations