Dynamic testing via automata learning

  • Harald Raffelt
  • Maik Merten
  • Bernhard Steffen
  • Tiziana MargariaEmail author
Regular Paper


This paper presents dynamic testing, a method that exploits automata learning to systematically test (black box) systems almost without prerequisites. Based on interface descriptions and optional sample test cases, our method successively explores the system under test (SUT), in order to extrapolate a behavioural model. This is in turn used to steer the further exploration process. Due to the applied learning technique, our method is optimal in the sense that the extrapolated models are most concise (i.e. state minimal) in consistently representing all the information gathered during the exploration. Using the LearnLib, our framework for automata learning, our method can be elegantly combined with numerous optimisations of the learning procedure, with various choices of model structures, and with the option of dynamically/interactively enlarging the alphabet underlying the learning process. The latter is important in the Web context, where totally new situations may arise when following links. All these features are illustrated using as a case study the web application Mantis, a bug tracking system widely used in practice. In addition, we present another case study that demonstrates the scalability of the approach. We show how the dynamic testing procedure works and how behavioural models arise that concisely summarize the current testing effort. It turns out that these models reveal the system structure from a user perspective. Besides steering the automatic exploration process, they are ideal for user guidance and to support analyses to improve the system understanding, as they reveal the system structure from a user perspective.


Test Suite System Under Test Automaton Learning Membership Query Message Queue 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Raffelt, H., Steffen, B., Margaria, T.: Dynamic Testing Via Automata Learning. In: Proceedings of Haifa Verification Conference 2007, LNCS 4899, pp.136–152 (2007)Google Scholar
  2. 2.
    Raffelt, H., Margaria, T., Steffen, B., Merten, M.: Hybrid test of web applications with webtest. In: Proceedings of the 2008 Workshop on Testing, analysis, and verification of web services and applications (TAV-WEB ’08), pp. 1–7Google Scholar
  3. 3.
    Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In Cerioli, M. (ed.) Proceedings of 8th International Conference on Fundamental Approaches to Software Engineering (FASE’05). vol. 3442 of LNCS, pp. 175–189. Springer, 4–8 April 2005Google Scholar
  4. 4.
    Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Tech. Transf. (2009). doi: 10.1007/s10009-009-0111-8
  5. 5.
    Niese, O., Margaria, T., Hagerer, A., Steffen, B., Brune, G., Goerigk, W., Ide, H.D.: Automated regression testing of cti-systems. In: Proceedings of the IEEE European Test Workshop (ETW’01), p. 51. IEEE Computer Society, Washington, DC, USA (2001)Google Scholar
  6. 6.
    Margaria, T., Niese, O., Steffen, B., Erochok, A.: System level testing of virtual switch (re-)configuration over ip. In: Proceedings of the IEEE European Test Workshop (ETW’02), IEEE Computer Society Press, pp. 67–74 (2002)Google Scholar
  7. 7.
    Hagerer A., Margaria T., Niese O., Steffen B., Brune G., Ide H.D.: Efficient regression testing of cti-systems: testing a complex call-center solution. Ann. Rev. Commun. Int. Eng. Consortium (IEC), Chicago (USA) 55, 1033–1040 (2001)Google Scholar
  8. 8.
    Jörges, S., Kubczak, C., Nagel, R., Margaria, T., Steffen, B.: Model-driven development with the jabc. In: Proceedings of Haifa verification conference 2006 (HVC 2006). LNCS, Springer.
  9. 9.
    Margaria, T., Steffen, B., Reitenspieß, M.: Service-oriented design: The jabc approach. In: Proceedings of Service Oriented Computing (SOC 2005), Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany (2005).
  10. 10.
    Karusseit M., Margaria T.: Feature-based modelling of a complex, online-reconfigurable decision support service. Electr. Notes Theor. Comput. Sci. 157(2), 101–118 (2006)CrossRefGoogle Scholar
  11. 11.
    Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. In: Proceedings of 2003 International Test Conference (ITC 2003), pp. 971–980. IEEE Computer Society, Charlotte, NC (2003)Google Scholar
  12. 12.
    Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece, October 13-15, 2008. Serie “Communications in Computer and Information Science” no. 17, pp. 490–502Google Scholar
  13. 13.
    Rational unified process. (seen June 2007)
  14. 14.
    Hörmann, M., Margaria, T., Mender, T., Nagel, R., Schuster, M., Steffen, B., Trinh, H.: The jABC approach to rigorous collaborative development of SCM applications. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece. Serie “Communications in Computer and Information Science” no. 17, pp. 724–737, 13–15 October 2008Google Scholar
  15. 15.
    Steffen, B., Narayan, P.: Full Life-Cycle Support for Endto- End Processes Computer. IEEE Computer Society Press, Los Alamitos (2007).
  16. 16.
    Raffelt, H., Steffen, B., Berg, T.: Learnlib: A library for automata learning and experimentation. In: Proc. of the 10th International Workshop on Formal Methods for Industrial Critical Systems (FMICS’05), pp. 62–71. ACM Press, Lisbon, Portugal (2005)Google Scholar
  17. 17.
    Mariani, L., Pezze, M.: Behaviour capture and test for controlling the quality of component-based integrated systems. In: Proceedings of ESEC/FSE Workshop on Tool Integration in System Development, Helsinki (2003)Google Scholar
  18. 18.
    Mercury TestDirector. (seen April 2008)
  19. 19.
    Ernst, M.D., Czeisler, A., Griswold, W.G., Notkin, D.: Quickly detecting relevant program invariants. In: Proceedings of 22th International Conference on Software Engineering (ICSE’00), pp. 449–458 (2000)Google Scholar
  20. 20.
    Nimmer, J.W., Ernst, M.D.: Automatic generation of program specifications. In: Proceedings of the 2002 International Symposium on Software Testing and Analysis (ISSTA’02), Rome, Italy, pp. 229–239, 22–24 July 2002Google Scholar
  21. 21.
    Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: Proceedings of the 26th International Conference on Software Engineering (ICSE’04), Edinburgh, Scotland, pp. 480–490 (2004)Google Scholar
  22. 22.
    Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In: Kutsche, H.W.R., (ed.) Proceedings of the 5th International Conference on Fundamental Approaches to Software Engineering (FASE’02), vol. 2306 of LNCS., Heidelberg, Germany, Springer, pp. 80–95 (2002)Google Scholar
  23. 23.
    de la Higuera C.: A bibliographical study of grammatical inference. Pattern Recognit 38, 1332–1348 (2005)CrossRefGoogle Scholar
  24. 24.
    Cook J.E., Wolf A.L.:: Discovering models of software processes from event-based data. (TOSEM) ACM Trans Softw Eng Methodol 7(3), 215–249 (1998)CrossRefGoogle Scholar
  25. 25.
    Xie, T., Notkin, D.: Mutually enhancing test generation and specification inference. In: Petrenk, A., Ulrich, A. (eds.) Proceedings of 3rd International Workshop on Formal Approaches to Testing of Software (FATES’03). vol. 2931 of LNCS, pp. 60–69. Springer (2004)Google Scholar
  26. 26.
    Peled D., Vardi M.Y., Yannakakis M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds) Proceedings of the Joint International Conference on Formal Description Techniques for Distributed System and Communication/Protocols and Protocol Specification, Testing and Verification FORTE/PSTV’99, pp. 225–240. Kluwer, Dordrecht (1999)Google Scholar
  27. 27.
    Cook, J.E., Du, Z., Liu, C., Wolf, A.L.: Discovering models of behavior for concurrent systems. Technical report, New Mexico State University, Department of Computer Science (2002) NMSU-CS-2002-010Google Scholar
  28. 28.
    Mariani, L., Pezzè, M.: A technique for verifying component-based software. In: Proceedings of International Workshop on Test and Analysis of Component Based Systems (TACoS’04), pp. 17–30 (2004)Google Scholar
  29. 29.
    Steffen, B., Margaria, T., Raffelt, H., Niese, O.: Efficient test-based model generation of legacy systems. In: Proceedings of the 9th IEEE International Workshop on High Level Design Validation and Test (HLDVT’04), Sonoma (CA), USA, IEEE Computer Society Press, pp. 95–100 (2004)Google Scholar
  30. 30.
    Margaria, T., Hinchey, M.G., Raffelt, H., Rash, J., Rouff, C.A., Steffen, B.: Completing and adapting models of biological processes. In: Proceedings of IFIP Conference on Biologically Inspired Cooperative Computing (BiCC 2006), Santiago (Chile).
  31. 31.
    Steffen, B., Hungar, H.: Behavior-based model construction. In: Mukhopadhyay, S., Zuck, L. (eds.) Proceedings of the 4th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI’03). vol. 2575 of LNCS., Springer, pp. 5–19 (2003)Google Scholar
  32. 32.
    Angluin D.: Learning regular sets from queries and counterexamples. Inf. Comput. 2(75), 87–106 (1987)CrossRefMathSciNetGoogle Scholar
  33. 33.
    Margaria T., Raffelt H., Steffen B.: Knowledge-based relevance filtering for efficient system-level test-based model generation. Innov. Syst. Softw. Eng. 1(2), 147–156 (2005)CrossRefGoogle Scholar
  34. 34.
    Fujiwara S., von Bochmann G., Khendek F., Amalou M., Ghedamsi A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)CrossRefGoogle Scholar
  35. 35.
    Mantis bug tracker. (seen June 2007)
  36. 36.
  37. 37.
    Bhogal, K.S.: HttpUnit: A Civilized Way to Test Web Applications in WebSphere Studio. (2003)
  38. 38.
    Willcock, C., Dei  T., Tobies, S., Keil, S., Engler, F., Schulz, S.: An Introduction to TTCN-3, First edition. Wiley (2005)Google Scholar
  39. 39.
    Troschütz, S.: Web Service Test Framework with TTCN-3 MSc. Thesis, University of Göttingen, June 2007.
  40. 40.
    Testing technologies: TTworkbench Basic.

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Harald Raffelt
    • 1
  • Maik Merten
    • 1
  • Bernhard Steffen
    • 1
  • Tiziana Margaria
    • 2
    Email author
  1. 1.TU Dortmund, Chair of Programming SystemsUniversity of DortmundDortmundGermany
  2. 2.Chair of Service and Software EngineeringUniversität PotsdamPotsdamGermany

Personalised recommendations