Dynamic Testing Via Automata Learning

  • Harald Raffelt
  • Bernhard Steffen
  • Tiziana Margaria
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4899)


This paper presents dynamic testing, a method that exploits automata learning to systematically test (black box) systems almost without prerequisites. Based on interface descriptions, our method successively explores the system under test (SUT), while it at the same time extrapolates a behavioral model. This is in turn used to steer the further exploration process. Due to the applied learning technique, our method is optimal in the sense that the extrapolated models are most concise in consistently representing all the information gathered during the exploration. Using the LearnLib, our framework for automata learning, our method can elegantly be combined with numerous optimizations of the learning procedure, various choices of model structures, and last but not least, with the option to dynamically/interactively enlarge the alphabet underlying the learning process. All these features will be illustrated using as a case study the web application Mantis, a bug tracking system widely used in practice. We will show how the dynamic testing procedure proceeds and how the behavioral models arise that concisely summarize the current testing effort. It has turned out that these models, besides steering the automatic exploration process, are ideal for user guidance and to support analyzes to improve the system understanding.


Dynamic Testing System Under Test Input Symbol Automaton Learning Conformance Testing 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In: Cerioli, M. (ed.) FASE 2005. LNCS, vol. 3442, pp. 175–189. Springer, Heidelberg (2005)Google Scholar
  2. 2.
    Niese, O., Margaria, T., Hagerer, A., Steffen, B., Brune, G., Goerigk, W., Ide, H.D.: Automated regression testing of cti-systems. In: ETW’01. Proc. of the IEEE European Test Workshop, Washington, DC, USA, p. 51. IEEE Computer Society, Los Alamitos (2001)Google Scholar
  3. 3.
    Margaria, T., Niese, O., Steffen, B., Erochok, A.: System level testing of virtual switch (re-)configuration over ip. In: ETW 2002. Proc. of the IEEE European Test Workshop, pp. 67–74. IEEE Computer Society Press, Los Alamitos (2002)Google Scholar
  4. 4.
    Hagerer, A., Margaria, T., Niese, O., Steffen, B., Brune, G., Ide, H.D.: Efficient regression testing of cti-systems: Testing a complex call-center solution. Annual Review of Communication, Int. Engineering Consortium (IEC), Chicago (USA) vol. 55, pp. 1033–1040 (2001)Google Scholar
  5. 5.
    Raffelt, H., Steffen, B., Berg, T.: Learnlib: A library for automata learning and experimentation. In: FMICS 2005. Proc.of the 10th International Workshop on Formal Methods for Industrial Critical Systems, Lisbon, Portugal, pp. 62–71. ACM Press, New York (2005)CrossRefGoogle Scholar
  6. 6.
    Jörges, S., Kubczak, C., Nagel, R., Margaria, T., Steffen, B.: Model-driven development with the jabc. In: Bin, E., Ziv, A., Ur, S. (eds.) HVC 2006. LNCS, vol. 4383, Springer, Heidelberg (2007)Google Scholar
  7. 7.
    Margaria, T., Steffen, B., Reitenspieß, M.: Service-oriented design: The jabc approach. In: SOC 2005. Proc. of Service Oriented Computing, Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany (2005)Google Scholar
  8. 8.
    Karusseit, M., Margaria, T.: Feature-based modelling of a complex, online-reconfigurable decision support service. Electr. Notes Theor. Comput. Sci. 157(2), 101–118 (2006)CrossRefGoogle Scholar
  9. 9.
    Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. In: ITC 2003. Proc. of 2003 International Test Conference, Charlotte, NC, pp. 971–980. IEEE Computer Society, Los Alamitos (September 2003)CrossRefGoogle Scholar
  10. 10.
    Hörmann, M., Margaria, T., Mender, T., Nagel, R., Schuster, M., Steffen, B., Trinh, H.: The jabc approach to collaborative development of embedded applications. In: CCE 2006. Int. Workshop on Challenges in Collaborative Engineering - State of the Art and Future Challenges on Collaborative Design, April 2006 (Industry Day), Prag (CZ) (2006)Google Scholar
  11. 11.
    Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: A library for automata learning and experimentation. International Journal on Software Tools Technology Transfer (Submitted, 2005)Google Scholar
  12. 12.
    Ernst, M.D., Czeisler, A., Griswold, W.G., Notkin, D.: Quickly detecting relevant program invariants. In: ICSE 2000. Proc. of 22nd Int. Conf. on Software Engineering, pp. 449–458 (June 2000)Google Scholar
  13. 13.
    Nimmer, J.W., Ernst, M.D.: Automatic generation of program specifications. In: ISSTA 2002. Proc. of the 2002 Int. Symposium on Software Testing and Analysis, Rome, Italy, pp. 229–239 (July 22–24, 2002)Google Scholar
  14. 14.
    Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: ICSE 2004. Proc. of the 26th Int. Conf. on Software Engineering, pp. 480–490 (May 2004)Google Scholar
  15. 15.
    Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In: Kutsche, R.-D., Weber, H. (eds.) ETAPS 2002 and FASE 2002. LNCS, vol. 2306, pp. 80–95. Springer, Heidelberg (2002)Google Scholar
  16. 16.
    de la Higuera, C.: A bibliographical study of grammatical inference. Pattern Recognition 38, 1332–1348 (2005)CrossRefGoogle Scholar
  17. 17.
    Cook, J.E., Wolf, A.L.: Discovering models of software processes from event-based data (TOSEM) ACM Transactions on Software Engineering and Methodology 7(3), 215–249 (1998)CrossRefGoogle Scholar
  18. 18.
    Mariani, L., Pezzè, M.: A technique for verifying component-based software. In: TACoS 2004. Proc. of Int. Workshop on Test and Analysis of Component Based Systems, pp. 17–30 (March 2004)Google Scholar
  19. 19.
    Xie, T., Notkin, D.: Mutually enhancing test generation and specification inference. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 60–69. Springer, Heidelberg (2004)Google Scholar
  20. 20.
    Peled, D., Vardi, M.Y., Yannakakis, M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds.) FORTE/PSTV 1999. Proc. of the Joint Int. Conference on Formal Description Techniques for Distributed System and Communication/Protocols and Protocol Specification, Testing and Verification, pp. 225–240. Kluwer Academic Publishers, Boston, MA (1999)Google Scholar
  21. 21.
    Cook, J.E., Du, Z., Liu, C., Wolf, A.L.: Discovering models of behavior for concurrent systems. Technical report, New Mexico State University, Deppartment of Computer Science, NMSU-CS-2002-010. (August 2002)Google Scholar
  22. 22.
    Steffen, B., Margaria, T., Raffelt, H., Niese, O.: Efficient test-based model generation of legacy systems. In: HLDVT 2004. Proc. of the 9th IEEE Int. Workshop on High Level Design Validation and Test (HLDVT’04), Sonoma (CA), USA, pp. 95–100. IEEE Computer Society Press, Los Alamitos (2004)Google Scholar
  23. 23.
    Margaria, T., Hinchey, M.G., Raffelt, H., Rash, J., Rouff, C.A., Steffen, B.: Completing and adapting models of biological processes. In: BiCC 2006. Proc. of IFIP Conf. on Biologically Inspired Cooperative Computing, Santiago (Chile) (2006)Google Scholar
  24. 24.
    Steffen, B., Hungar, H.: Behavior-based model construction. In: Zuck, L.D., Attie, P.C., Cortesi, A., Mukhopadhyay, S. (eds.) VMCAI 2003. LNCS, vol. 2575, pp. 5–19. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  25. 25.
    Angluin, D.: Learning regular sets from queries and counterexamples. Information and Computation 2(75), 87–106 (1987)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Margaria, T., Raffelt, H., Steffen, B.: Knowledge-based relevance filtering for efficient system-level test-based model generation. Innovations in Systems and Software Engineering 1(2), 147–156 (2005)CrossRefGoogle Scholar
  27. 27.
    Fujiwara, S., von Bochmann, G., Khendek, F., Amalou, M., Ghedamsi, A.: Test selection based on finite state models. IEEE Transactions on Software Engineering 17(6), 591–603 (1991)CrossRefGoogle Scholar
  28. 28.
    Rational unified process (seen June 2007),
  29. 29.
    Mantis bug tracker. (seen June 2007),

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Harald Raffelt
    • 1
  • Bernhard Steffen
    • 1
  • Tiziana Margaria
    • 2
  1. 1.Chair of Programming SystemsUniversity of DortmundDortmundGermany
  2. 2.Chair of Services and Software EngineeringUniversität PotsdamPotsdamGermany

Personalised recommendations