Advertisement

Coverage-biased random exploration of large models and application to testing

  • Alain Denise
  • Marie-Claude Gaudel
  • Sandrine-Dominique Gouraud
  • Richard Lassaigne
  • Johan Oudinet
  • Sylvain Peyronnet
Regular Paper

Abstract

This paper presents several randomised algorithms for generating paths in large models according to a given coverage criterion. Using methods for counting combinatorial structures, these algorithms can efficiently explore very large models, based on a graphical representation by an automaton or by a product of several automata. This new approach can be applied to random exploration in order to optimise path coverage and can be generalised to take into account other coverage criteria, via the definition of a notion of randomised coverage satisfaction. Our main contributions are a method for drawing paths uniformly at random in composed models, i.e. models that are given as products of automata, first without and then with synchronisation; a new efficient approach to draw paths at random taking into account some other coverage criterion. Experimental results show promising agreement with theoretical predictions and significant improvement over previous randomised approaches. This work opens new perspectives for future studies of statistical testing and model checking, mainly to fight the combinatorial explosion problem.

Keywords

Model based testing Random testing Uniform exploration 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abed, N., Tripakis, S., Vincent, J.-M.: Resource-aware verification using randomized exploration of large state spaces. In: Model Checking Software. 15th International SPIN Workshop. Lecture Notes in Computer Science, vol. 5156, pp. 214–231 (2008)Google Scholar
  2. 2.
    Aldous D.: An introduction to covering problems for random walks on graphs. J. Theor. Probab. 4, 197–211 (1991)MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Arnold A.: Finite Transition Systems. Prentice-Hall, Englewood Cliffs (1994)MATHGoogle Scholar
  4. 4.
    Barrett, C., Ranise, S., Stump, A., Tinelli, C.: The Satisfiability Modulo Theories Library (SMT-LIB). http://www.SMT-LIB.org (2008)
  5. 5.
    Bobot, F., Conchon, S., Contejean, E., Lescuyer, S.: Implementing polymorphism in SMT solvers. In: SMT ’08/BPR ’08: Proceedings of the Joint Workshops of the 6th International Workshop on Satisfiability Modulo Theories and 1st International Workshop on Bit-Precise Reasoning, pp. 1–5. ACM, New York (2008)Google Scholar
  6. 6.
    Cavalli, A., Lee, D., Rinderknecht, C., Zaïdi, F.: Hit-or-Jump: an algorithm for embedded testing with applications to in services. In: Wu, J., et al. (eds.) Proceeding of IFIP International Conference FORTE/PSTV’99, pp. 41–56. Beijing, China October 1999Google Scholar
  7. 7.
    Chen T.Y., Yu Y.T.: On the relationship between partition and random testing. IEEE Trans. Softw. Eng. 20(12), 977–980 (1994)CrossRefGoogle Scholar
  8. 8.
    Creutzig C., Oevel W.: MuPAD Tutorial, 2nd edn. Springer, New York (2004)CrossRefGoogle Scholar
  9. 9.
    Crouzet, Y., Waeselynck, H., Lussier, B., Powell, D.: The SESAME experience: from assembly languages to declarative models. In: MUTATION ’06: Proceedings of the Second Workshop on Mutation Analysis, p. 7. IEEE Computer Society, Washington DC (2006)Google Scholar
  10. 10.
    Mendonça de Moura, L., Bjørner, N.: Z3: an efficient SMT solver. In: TACAS. Lecture Notes in Computer Science, vol. 4963, pp. 337–340 (2008)Google Scholar
  11. 11.
    DeMillo, R.A.: Mutation analysis as a tool for software quality assurance. In: Proceedings COMPSAC’80, pp. 390–393 (1980)Google Scholar
  12. 12.
    Denise, A., Gaudel, M.-C., Gouraud S.-D.: A generic method for statistical testing. In: ISSRE, pp. 25–34. IEEE Computer Society (2004)Google Scholar
  13. 13.
    Duran, J.W., Ntafos, S.C.: A report on random testing. In: 5th IEEE International Conference on Software Engineering, pp. 179–183 (1981)Google Scholar
  14. 14.
    Duran J.W., Ntafos S.C.: An evaluation of random testing. IEEE Trans. Softw. Eng. SE-10, 438–444 (1984)CrossRefGoogle Scholar
  15. 15.
    Dwyer, M.B., Elbaum, S.G., Person, S., Purandare, R.: Parallel randomized state-space search. In: 29th International Conference on Software Engineering (ICSE 2007), pp. 3–12 (2007)Google Scholar
  16. 16.
    Ferguson R., Korel B.: The chaining approach for software test data generation. ACM Trans. Softw. Eng. Methodol. 5(1), 63–86 (1996)CrossRefGoogle Scholar
  17. 17.
    Flajolet, P., Sedgewick, R.: Analytic Combinatorics. Cambridge University Press (2008)Google Scholar
  18. 18.
    Flajolet P., Zimmermann P., Van Cutsem B.: A calculus for the random generation of Labelled Combinatorial Structures. Theor. Comput. Sci. 132, 1–35 (1994)MathSciNetMATHCrossRefGoogle Scholar
  19. 19.
    Garavel, H., Descoubes, N.: Very Large Transition Systems. http://tinyurl.com/yuroxx. July 2003
  20. 20.
    Garavel, H., Lang, F., Mateescu, R.: An overview of CADP 2001. Technical Report 0254, INRIA (2001)Google Scholar
  21. 21.
    Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. In: PLDI ’05: Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation, pp. 213–223. ACM, New York (2005)Google Scholar
  22. 22.
    Gouraud, S.-D.: Utilisation des Structures Combinatoires pour le Test Statistique. PhD thesis, Université Paris-Sud 11, LRI. June 2004Google Scholar
  23. 23.
    Gouraud S.-D.: Auguste: a tool for statistical testing. LRI research report RR-1400, Université de Paris-Sud 11 (2005)Google Scholar
  24. 24.
    Gouraud, S.-D., Denise, A., Gaudel M.-C., Marre B.: A new way of automating statistical testing methods. In: ASE, pp. 5–12. IEEE Computer Society (2001)Google Scholar
  25. 25.
    Torbjörn Granlund: GNU MP: The GNU Multiple Precision Arithmetic Library. http://gmplib.org/manual/
  26. 26.
    Grosu, R., Smolka, S.A.: Monte Carlo model checking. In: TACAS. Lecture Notes in Computer Science, vol. 3440, pp. 271–286. Springer, New York (2005)Google Scholar
  27. 27.
    Gutjahr W.J.: Partition testing vs. random testing: the influence of uncertainty. IEEE Trans. Softw. Eng. 25(5), 661–674 (1999)CrossRefGoogle Scholar
  28. 28.
    Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. In: Steffen, B., Levi, G. (eds.) VMCAI. Lecture Notes in Computer Science, vol. 2937, pp. 73–84. Springer, New York (2004)Google Scholar
  29. 29.
    Hogrefe, D.: OSI Formal Specification Case Study: The INRES Protocol and Service (revised). Technical Report IAM-91-012, Institut fur Informatik, Universitat Bern. May 1991Google Scholar
  30. 30.
    Hopcroft J.E., Motwani R., Ullman J.D.: Introduction to automata theory, languages, and computation, 2nd edn. SIGACT News 32(1), 60–65 (2001)CrossRefGoogle Scholar
  31. 31.
    Kalaji, A.S., Hierons, R.M., Swift, S.: Generating feasible transition paths for testing from an extended finite state machine (EFSM). In: ICST ’09: Proceedings of the 2009 International Conference on Software Testing Verification and Validation, pp. 230–239. IEEE Computer Society, Washington DC (2009)Google Scholar
  32. 32.
    Marre, B., Blanc, B.: Test selection strategies for Lustre descriptions in GATeL. In: Model-Based Testing Workshop, volume 111, 1 of Electronic Notes in Theoretical Computer Science, pp. 93–111 (2005)Google Scholar
  33. 33.
    Maurer, J., Abrahams, D., Dawes, B., Rivera, R.: Boost random number library. http://www.boost.org/libs/random/. June 2000
  34. 34.
    Mihail M., Papadimitriou C.H.: On the random walk method for protocol testing. In: Dill, D.L. (eds) CAV. Lecture Notes in Computer Science, vol. 818, pp. 132–141. Springer, New York (1994)Google Scholar
  35. 35.
    Musa J., Fuoco G., Irving N., Krofl D., Juhli B.: The operational profile. In: Lyu, M.R. (eds) Handbook on Software Reliability Engineering, pp. 167–218. IEEE Computer Society Press, McGraw-Hill, New York (1996)Google Scholar
  36. 36.
    Ntafos S.C.: On comparisons of random, partition, and proportional partition testing. IEEE Trans. Softw. Eng. 27(10), 949–960 (2001)CrossRefGoogle Scholar
  37. 37.
    Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: 29th International Conference on Software Engineering (ICSE 2007), pp. 75–84 (2007)Google Scholar
  38. 38.
    Prowell S., Poore J.: Computing system reliability using Markov chain usage models. J. Syst. Softw. 73(2), 215–225 (2004)CrossRefGoogle Scholar
  39. 39.
    Reid, S.: An empirical analysis of equivalence partitioning, boundary value analysis and random testing. In: IEEE METRICS Conference, pp. 64–73 (1997)Google Scholar
  40. 40.
    Rungta, N., Mercer, E.G.: Generating counter-examples through randomized guided search. In: SPIN Workshop. Lecture Notes in Computer Science, vol. 4595, pp. 39–57. Springer, New York (2007)Google Scholar
  41. 41.
    Sen, K.: Effective random testing of concurrent programs. In: ASE ’07: Proceedings of the twenty-second IEEE/ACM International Conference on Automated Software Engineering, pp. 323–332. ACM, New York (2007)Google Scholar
  42. 42.
    Seneta, E.: Non-negative Matrices and Markov Chains, 2nd edn. Springer-Verlag (2006). (The 1st edn. appeared in 1981)Google Scholar
  43. 43.
    Thévenod-Fosse, P.: Software validation by means of statistical testing: Retrospect and future direction. In: International Working Conference on Dependable Computing for Critical Applications, pp. 15–22. Rapport L.A.A.S. No 89043 (1989)Google Scholar
  44. 44.
    Thévenod-Fosse P., Waeselynck H.: An investigation of software statistical testing. J. Softw. Test. Verif. Reliab. 1(2), 5–26 (1991)Google Scholar
  45. 45.
    Thévenod-Fosse, P., Waeselynck, H., Crouzet, Y.: Software statistical testing. In: Randell, B., Laprie, J.C., Kopetz, H., Littlewood, B. (eds.) Predictably Dependable Computing Systems, pp. 253–272. Springer, New York (1995). ISBN:3-540-59334-9Google Scholar
  46. 46.
    Visser W., Havelund K., Brat G.P., Park S., Lerda F.: Model checking programs. Autom. Softw. Eng. 10(2), 203–232 (2003)CrossRefGoogle Scholar
  47. 47.
    West, C.H.: Protocol validation in complex systems. In: SIGCOMM, pp. 303–312 (1989)Google Scholar
  48. 48.
    Weyuker E.J., Jeng B.: Analyzing partition testing strategies. IEEE Trans. Softw. Eng. 17(7), 703–711 (1991)CrossRefGoogle Scholar
  49. 49.
    Wilf H.S.: A unified setting for sequencing, ranking, and selection algorithms for combinatorial objects. Adv. Math. 24, 281–291 (1977)MathSciNetMATHGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Alain Denise
    • 1
    • 2
    • 3
  • Marie-Claude Gaudel
    • 1
    • 2
  • Sandrine-Dominique Gouraud
    • 1
    • 2
  • Richard Lassaigne
    • 4
    • 5
  • Johan Oudinet
    • 1
    • 2
  • Sylvain Peyronnet
    • 1
    • 2
    • 3
  1. 1.LRIUniv Paris-SudOrsayFrance
  2. 2.CNRSOrsayFrance
  3. 3.INRIA Saclay-Ile-de-FranceOrsay cedexFrance
  4. 4.Logique Mathématique (IMJ)Univ Paris VIIParisFrance
  5. 5.CNRSParisFrance

Personalised recommendations