Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer

  • Margus Veanes
  • Colin Campbell
  • Wolfgang Grieskamp
  • Wolfram Schulte
  • Nikolai Tillmann
  • Lev Nachmanson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4949)

Abstract

Testing is one of the costliest aspects of commercial software development. Model-based testing is a promising approach addressing these deficits. At Microsoft, model-based testing technology developed by the Foundations of Software Engineering group in Microsoft Research has been used since 2003. The second generation of this tool set, Spec Explorer, deployed in 2004, is now used on a daily basis by Microsoft product groups for testing operating system components, .NET framework components and other areas. This chapter provides a comprehensive survey of the concepts of the tool and their foundations.

Keywords

Eter Harness Metaphor Verse Wolfram 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Spec Explorer tool. public release January 2005, updated release October 2006, http://research.microsoft.com/specexplorer
  2. 2.
    Spec# tool (public release, March 2005) http://research.microsoft.com/specsharp
  3. 3.
    Alur, R., Courcoubetis, C., Yannakakis, M.: Distinguishing tests for nondeterministic and probabilistic machines. In: Proc. 27th Ann. ACM Symp. Theory of Computing, pp. 363–372 (1995)Google Scholar
  4. 4.
    Alur, R., Henzinger, T., Kupferman, O., Vardi, M.: Alternating refinement relations. In: Sangiorgi, D., de Simone, R. (eds.) CONCUR 1998. LNCS, vol. 1466, pp. 163–178. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  5. 5.
    Artho, C., Drusinsky, D., Goldberg, A., Havelund, K., Lowry, M., Pasareanu, C., Rosu, G., Visser, W.: Experiments with test case generation and runtime analysis. In: Börger, E., Gargantini, A., Riccobene, E. (eds.) ASM 2003. LNCS, vol. 2589, pp. 87–107. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  6. 6.
    Barnett, M., Grieskamp, W., Nachmanson, L., Schulte, W., Tillmann, N., Veanes, M.: Towards a tool environment for model-based testing with AsmL. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 264–280. Springer, Heidelberg (2004)Google Scholar
  7. 7.
    Barnett, M., Leino, R., Schulte, W.: The Spec# programming system: An overview. In: Barthe, G., Burdy, L., Huisman, M., Lanet, J.-L., Muntean, T. (eds.) CASSIS 2004. LNCS, vol. 3362, pp. 49–69. Springer, Heidelberg (2005)Google Scholar
  8. 8.
    Blass, A., Gurevich, Y.: Background, reserve, and Gandy machines. In: Clote, P.G., Schwichtenberg, H. (eds.) CSL 2000. LNCS, vol. 1862, pp. 1–17. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  9. 9.
    Blass, A., Gurevich, Y., Nachmanson, L., Veanes, M.: Play to test. In: Grieskamp, W., Weise, C. (eds.) FATES 2005. LNCS, vol. 3997, pp. 32–46. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  10. 10.
    Börger, E., Stärk, R.: Abstract State Machines: A Method for High-Level System Design and Analysis. Springer, Heidelberg (2003)MATHGoogle Scholar
  11. 11.
    Brinksma, E., Tretmans, J.: Testing Transition Systems: An Annotated Bibliography. In: Cassez, F., Jard, C., Rozoy, B., Dermot, M. (eds.) MOVEP 2000. LNCS, vol. 2067, pp. 187–193. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  12. 12.
    Campbell, C., Veanes, M.: State exploration with multiple state groupings. In: Beauquier, D., Börger, E., Slissenko, A. (eds.) 12th International Workshop on Abstract State Machines, ASM 2005, March 8–11, 2005, Laboratory of Algorithms, Complexity and Logic, University Paris 12 – Val de Marne, Créteil, France, pp. 119–130 (2005)Google Scholar
  13. 13.
    Campbell, C., Veanes, M., Huo, J., Petrenko, A.: Multiplexing of partially ordered events. In: Khendek, F., Dssouli, R. (eds.) TestCom 2005. LNCS, vol. 3502, pp. 97–110. Springer, Heidelberg (2005)Google Scholar
  14. 14.
    de Alfaro, L.: Computing minimum and maximum reachability times in probabilistic systems. In: Baeten, J.C.M., Mauw, S. (eds.) CONCUR 1999. LNCS, vol. 1664, pp. 66–81. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  15. 15.
    de Alfaro, L.: Game models for open systems. In: Dershowitz, N. (ed.) Verification: Theory and Practice. LNCS, vol. 2772, pp. 269–289. Springer, Heidelberg (2004)Google Scholar
  16. 16.
    de Alfaro, L., Henzinger, T.A.: Interface automata. In: Proceedings of the 8th European Software Engineering Conference and the 9th ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 109–120. ACM, New York (2001)CrossRefGoogle Scholar
  17. 17.
    Franzen, L., Tretmans, J., Willemse, T.A.C.: Test generation based on symbolic specifications. In: Grabowski, J., Nielsen, B. (eds.) FATES 2004. LNCS, vol. 3395, pp. 1–15. Springer, Heidelberg (2005)Google Scholar
  18. 18.
    Glässer, U., Gurevich, Y., Veanes, M.: Abstract communication model for distributed systems. IEEE Transactions on Software Engineering 30(7), 458–472 (2004)CrossRefGoogle Scholar
  19. 19.
    Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating finite state machines from abstract state machines. In: ISSTA 2002. Software Engineering Notes, vol. 27, pp. 112–122. ACM, New York (2002)CrossRefGoogle Scholar
  20. 20.
    Grieskamp, W., Kicillof, N., Tillmann, N.: Action machines: a framework for encoding and composing partial behaviors. International Journal of Software Engineering and Knowledge Engineering 16(5), 705–726 (2006)CrossRefGoogle Scholar
  21. 21.
    Grieskamp, W., Tillmann, N., Veanes, M.: Instrumenting scenarios in a model-driven development environment. Information and Software Technology 46(15), 1027–1036 (2004)CrossRefGoogle Scholar
  22. 22.
    Gujiwara, S., Bochman, G.V.: Testing non-deterministic state machines with fault-coverage. In: Kroon, J., Heijunk, R.J., Brinksma, E. (eds.) Protocol Test Systems, pp. 267–280 (1992)Google Scholar
  23. 23.
    Gurevich, Y.: Evolving Algebras 1993: Lipari Guide. In: Börger, E. (ed.) Specification and Validation Methods, pp. 9–36. Oxford University Press, Oxford (1995)Google Scholar
  24. 24.
    Gurevich, Y., Kutter, P.W., Odersky, M., Thiele, L. (eds.): ASM 2000. LNCS, vol. 1912. Springer, Heidelberg (2000)MATHGoogle Scholar
  25. 25.
    Gurevich, Y., Rossman, B., Schulte, W.: Semantic essence of AsmL. Theoretical Computer Science 343(3), 370–412 (2005)CrossRefMathSciNetMATHGoogle Scholar
  26. 26.
    Hartman, A., Nagin, K.: Model driven testing - AGEDIS architecture interfaces and tools. In: 1st European Conference on Model Driven Software Engineering, Nuremberg, Germany, December 2003, pp. 1–11 (2003)Google Scholar
  27. 27.
    Jard, C., Jéron, T.: TGV: theory, principles and algorithms. In: The Sixth World Conference on Integrated Design and Process Technology, IDPT 2002, Pasadena, California (June 2002)Google Scholar
  28. 28.
    Kuliamin, V.V., Petrenko, A.K., Kossatchev, A.S., Bourdonov, I.B.: UniTesK: Model based testing in industrial practice. In: 1st European Conference on Model Driven Software Engineering, Nuremberg, Germany, December 2003, pp. 55–63 (2003)Google Scholar
  29. 29.
    Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines – a survey. In: Proceedings of the IEEE, Berlin, August 1996, vol. 84, pp. 1090–1123. IEEE Computer Society Press, Los Alamitos (1996)Google Scholar
  30. 30.
    Nachmanson, L., Veanes, M., Schulte, W., Tillmann, N., Grieskamp, W.: Optimal strategies for testing nondeterministic systems. In: ISSTA 2004, July 2004. Software Engineering Notes, vol. 29, pp. 55–64. ACM, New York (2004)CrossRefGoogle Scholar
  31. 31.
    Petrenko, A., Yevtushenko, N., Bochmann, G.v.: Testing deterministic implementations from nondeterministic FSM specifications. In: Baumgarten, B., Burkhardt, H.-J., Giessler, A. (eds.) IFIP TC6 9th International Workshop on Testing of Communicating Systems, pp. 125–140. Chapman & Hall, Boca Raton (1996)Google Scholar
  32. 32.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley-Interscience, New York (1994)MATHGoogle Scholar
  33. 33.
    Tretmans, J., Belinfante, A.: Automatic testing with formal methods. In: EuroSTAR 1999: 7th European Int. Conference on Software Testing, Analysis & Review. EuroStar Conferences, Barcelona, Spain, Galway, Ireland, November 8–12 (1999)Google Scholar
  34. 34.
    Tretmans, J., Brinksma, E.: TorX: Automated model based testing. In: 1st European Conference on Model Driven Software Engineering, Nuremberg, Germany, December 2003, pp. 31–43 (2003)Google Scholar
  35. 35.
    van der Bij, M., Rensink, A., Tretmans, J.: Compositional testing with ioco. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 86–100. Springer, Heidelberg (2004)Google Scholar
  36. 36.
    Veanes, M., Campbell, C., Schulte, W., Kohli, P.: On-the-fly testing of reactive systems. Technical Report MSR-TR-2005-03, Microsoft Research (January 2005)Google Scholar
  37. 37.
    Veanes, M., Roy, P., Campbell, C.: Online testing with reinforcement learning. In: Havelund, K., Núñez, M., Roşu, G., Wolff, B. (eds.) FATES 2006 and RV 2006. LNCS, vol. 4262, pp. 240–253. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  38. 38.
    Veanes, M., Campbell, C., Schulte, W., Tillmann, N.: Online testing with model programs. In: ESEC/FSE-13: Proceedings of the 10th European Software Engineering Conference held jointly with 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 273–282. ACM, New York (2005)CrossRefGoogle Scholar
  39. 39.
    Yannakakis, M.: Testing, optimization, and games. In: Proceedings of the Nineteenth Annual IEEE Symposium on Logic In Computer Science, LICS 2004, pp. 78–88. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  40. 40.
    Yi, W., Larsen, K.G.: Testing probabilistic and nondeterministic processes. In: Testing and Verification XII, pp. 347–361. North-Holland, Amsterdam (1992)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Margus Veanes
    • 1
  • Colin Campbell
    • 1
  • Wolfgang Grieskamp
    • 1
  • Wolfram Schulte
    • 1
  • Nikolai Tillmann
    • 1
  • Lev Nachmanson
    • 1
  1. 1.Microsoft ResearchRedmondUSA

Personalised recommendations