Skip to main content
SpringerLink
Log in

Dynamic testing via automata learning

  • Regular Paper
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

This paper presents dynamic testing, a method that exploits automata learning to systematically test (black box) systems almost without prerequisites. Based on interface descriptions and optional sample test cases, our method successively explores the system under test (SUT), in order to extrapolate a behavioural model. This is in turn used to steer the further exploration process. Due to the applied learning technique, our method is optimal in the sense that the extrapolated models are most concise (i.e. state minimal) in consistently representing all the information gathered during the exploration. Using the LearnLib, our framework for automata learning, our method can be elegantly combined with numerous optimisations of the learning procedure, with various choices of model structures, and with the option of dynamically/interactively enlarging the alphabet underlying the learning process. The latter is important in the Web context, where totally new situations may arise when following links. All these features are illustrated using as a case study the web application Mantis, a bug tracking system widely used in practice. In addition, we present another case study that demonstrates the scalability of the approach. We show how the dynamic testing procedure works and how behavioural models arise that concisely summarize the current testing effort. It turns out that these models reveal the system structure from a user perspective. Besides steering the automatic exploration process, they are ideal for user guidance and to support analyses to improve the system understanding, as they reveal the system structure from a user perspective.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Raffelt, H., Steffen, B., Margaria, T.: Dynamic Testing Via Automata Learning. In: Proceedings of Haifa Verification Conference 2007, LNCS 4899, pp.136–152 (2007)

  2. Raffelt, H., Margaria, T., Steffen, B., Merten, M.: Hybrid test of web applications with webtest. In: Proceedings of the 2008 Workshop on Testing, analysis, and verification of web services and applications (TAV-WEB ’08), pp. 1–7

  3. Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., Steffen, B.: On the correspondence between conformance testing and regular inference. In Cerioli, M. (ed.) Proceedings of 8th International Conference on Fundamental Approaches to Software Engineering (FASE’05). vol. 3442 of LNCS, pp. 175–189. Springer, 4–8 April 2005

  4. Raffelt, H., Steffen, B., Berg, T., Margaria, T.: LearnLib: a framework for extrapolating behavioral models. Int. J. Softw. Tools Tech. Transf. (2009). doi:10.1007/s10009-009-0111-8

  5. Niese, O., Margaria, T., Hagerer, A., Steffen, B., Brune, G., Goerigk, W., Ide, H.D.: Automated regression testing of cti-systems. In: Proceedings of the IEEE European Test Workshop (ETW’01), p. 51. IEEE Computer Society, Washington, DC, USA (2001)

  6. Margaria, T., Niese, O., Steffen, B., Erochok, A.: System level testing of virtual switch (re-)configuration over ip. In: Proceedings of the IEEE European Test Workshop (ETW’02), IEEE Computer Society Press, pp. 67–74 (2002)

  7. Hagerer A., Margaria T., Niese O., Steffen B., Brune G., Ide H.D.: Efficient regression testing of cti-systems: testing a complex call-center solution. Ann. Rev. Commun. Int. Eng. Consortium (IEC), Chicago (USA) 55, 1033–1040 (2001)

    Google Scholar 

  8. Jörges, S., Kubczak, C., Nagel, R., Margaria, T., Steffen, B.: Model-driven development with the jabc. In: Proceedings of Haifa verification conference 2006 (HVC 2006). LNCS, Springer. http://www.springerlink.com/content/30v87502kn5h6672/

  9. Margaria, T., Steffen, B., Reitenspieß, M.: Service-oriented design: The jabc approach. In: Proceedings of Service Oriented Computing (SOC 2005), Internationales Begegnungs- und Forschungszentrum für Informatik (IBFI), Schloss Dagstuhl, Germany (2005). http://dblp.uni-trier.de/rec/bibtex/conf/icsoc/MargariaSR05

  10. Karusseit M., Margaria T.: Feature-based modelling of a complex, online-reconfigurable decision support service. Electr. Notes Theor. Comput. Sci. 157(2), 101–118 (2006)

    Article  Google Scholar 

  11. Hungar, H., Margaria, T., Steffen, B.: Test-based model generation for legacy systems. In: Proceedings of 2003 International Test Conference (ITC 2003), pp. 971–980. IEEE Computer Society, Charlotte, NC (2003)

  12. Margaria, T., Steffen, B.: Agile IT: thinking in user-centric models. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece, October 13-15, 2008. Serie “Communications in Computer and Information Science” no. 17, pp. 490–502

  13. Rational unified process. http://www-306.ibm.com/software/awdtools/rup/ (seen June 2007)

  14. Hörmann, M., Margaria, T., Mender, T., Nagel, R., Schuster, M., Steffen, B., Trinh, H.: The jABC approach to rigorous collaborative development of SCM applications. In: Proceedings of ISoLA 2008, Leveraging Applications of Formal Methods, Verification and Validation, Third International Symposium, Porto Sani, Greece. Serie “Communications in Computer and Information Science” no. 17, pp. 724–737, 13–15 October 2008

  15. Steffen, B., Narayan, P.: Full Life-Cycle Support for Endto- End Processes Computer. IEEE Computer Society Press, Los Alamitos (2007). http://portal.acm.org/citation.cfm?id=1320644

  16. Raffelt, H., Steffen, B., Berg, T.: Learnlib: A library for automata learning and experimentation. In: Proc. of the 10th International Workshop on Formal Methods for Industrial Critical Systems (FMICS’05), pp. 62–71. ACM Press, Lisbon, Portugal (2005)

  17. Mariani, L., Pezze, M.: Behaviour capture and test for controlling the quality of component-based integrated systems. In: Proceedings of ESEC/FSE Workshop on Tool Integration in System Development, Helsinki (2003)

  18. Mercury TestDirector. http://www.fronde.com/products_mercury.htm (seen April 2008)

  19. Ernst, M.D., Czeisler, A., Griswold, W.G., Notkin, D.: Quickly detecting relevant program invariants. In: Proceedings of 22th International Conference on Software Engineering (ICSE’00), pp. 449–458 (2000)

  20. Nimmer, J.W., Ernst, M.D.: Automatic generation of program specifications. In: Proceedings of the 2002 International Symposium on Software Testing and Analysis (ISSTA’02), Rome, Italy, pp. 229–239, 22–24 July 2002

  21. Brun, Y., Ernst, M.D.: Finding latent code errors via machine learning over program executions. In: Proceedings of the 26th International Conference on Software Engineering (ICSE’04), Edinburgh, Scotland, pp. 480–490 (2004)

  22. Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model generation by moderated regular extrapolation. In: Kutsche, H.W.R., (ed.) Proceedings of the 5th International Conference on Fundamental Approaches to Software Engineering (FASE’02), vol. 2306 of LNCS., Heidelberg, Germany, Springer, pp. 80–95 (2002)

  23. de la Higuera C.: A bibliographical study of grammatical inference. Pattern Recognit 38, 1332–1348 (2005)

    Article  Google Scholar 

  24. Cook J.E., Wolf A.L.:: Discovering models of software processes from event-based data. (TOSEM) ACM Trans Softw Eng Methodol 7(3), 215–249 (1998)

    Article  Google Scholar 

  25. Xie, T., Notkin, D.: Mutually enhancing test generation and specification inference. In: Petrenk, A., Ulrich, A. (eds.) Proceedings of 3rd International Workshop on Formal Approaches to Testing of Software (FATES’03). vol. 2931 of LNCS, pp. 60–69. Springer (2004)

  26. Peled D., Vardi M.Y., Yannakakis M.: Black box checking. In: Wu, J., Chanson, S.T., Gao, Q. (eds) Proceedings of the Joint International Conference on Formal Description Techniques for Distributed System and Communication/Protocols and Protocol Specification, Testing and Verification FORTE/PSTV’99, pp. 225–240. Kluwer, Dordrecht (1999)

    Google Scholar 

  27. Cook, J.E., Du, Z., Liu, C., Wolf, A.L.: Discovering models of behavior for concurrent systems. Technical report, New Mexico State University, Department of Computer Science (2002) NMSU-CS-2002-010

  28. Mariani, L., Pezzè, M.: A technique for verifying component-based software. In: Proceedings of International Workshop on Test and Analysis of Component Based Systems (TACoS’04), pp. 17–30 (2004)

  29. Steffen, B., Margaria, T., Raffelt, H., Niese, O.: Efficient test-based model generation of legacy systems. In: Proceedings of the 9th IEEE International Workshop on High Level Design Validation and Test (HLDVT’04), Sonoma (CA), USA, IEEE Computer Society Press, pp. 95–100 (2004)

  30. Margaria, T., Hinchey, M.G., Raffelt, H., Rash, J., Rouff, C.A., Steffen, B.: Completing and adapting models of biological processes. In: Proceedings of IFIP Conference on Biologically Inspired Cooperative Computing (BiCC 2006), Santiago (Chile). http://www.springerlink.com/content/2t8279g8g655u826/

  31. Steffen, B., Hungar, H.: Behavior-based model construction. In: Mukhopadhyay, S., Zuck, L. (eds.) Proceedings of the 4th International Conference on Verification, Model Checking, and Abstract Interpretation (VMCAI’03). vol. 2575 of LNCS., Springer, pp. 5–19 (2003)

  32. Angluin D.: Learning regular sets from queries and counterexamples. Inf. Comput. 2(75), 87–106 (1987)

    Article  MathSciNet  Google Scholar 

  33. Margaria T., Raffelt H., Steffen B.: Knowledge-based relevance filtering for efficient system-level test-based model generation. Innov. Syst. Softw. Eng. 1(2), 147–156 (2005)

    Article  Google Scholar 

  34. Fujiwara S., von Bochmann G., Khendek F., Amalou M., Ghedamsi A.: Test selection based on finite state models. IEEE Trans. Softw. Eng. 17(6), 591–603 (1991)

    Article  Google Scholar 

  35. Mantis bug tracker. http://www.mantisbt.org (seen June 2007)

  36. Watij User Guide. http://watij.com/wiki:user_guide

  37. Bhogal, K.S.: HttpUnit: A Civilized Way to Test Web Applications in WebSphere Studio. http://www.ibm.com/developerworks/websphere/library/techarticles/0303_bhogal/bhogal.html (2003)

  38. Willcock, C., Dei  T., Tobies, S., Keil, S., Engler, F., Schulz, S.: An Introduction to TTCN-3, First edition. Wiley (2005)

  39. Troschütz, S.: Web Service Test Framework with TTCN-3 MSc. Thesis, University of Göttingen, June 2007. http://www.swe.informatik.uni-goettingen.de/publications/ST/WebServiceTestFrameworkWithTTCN-3.pdf

  40. Testing technologies: TTworkbench Basic. http://testingtech.de/products/ttwb_basic.php

Download references

Author information

Authors and Affiliations

  1. TU Dortmund, Chair of Programming Systems, University of Dortmund, Otto-Hahn-Str. 14, 44227, Dortmund, Germany

    Harald Raffelt, Maik Merten & Bernhard Steffen

  2. Chair of Service and Software Engineering, Universität Potsdam, August-Bebel-Str. 89, 14482, Potsdam, Germany

    Tiziana Margaria

Authors
  1. Harald Raffelt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maik Merten
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bernhard Steffen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tiziana Margaria
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tiziana Margaria.

Additional information

This work was partially supported by the European Union Specific Targeted Research Project SHADOWS (IST-2006-35157), Self-Healing Approach to Designing cOmplex softWare Systems and the FET Project CONNECT: Emergent Connectors for Eternal Software Intensive Networked Systems (http://connect-forever.eu/).

This paper is an extended version of [1] and [2].

Rights and permissions

Reprints and permissions

About this article

Cite this article

Raffelt, H., Merten, M., Steffen, B. et al. Dynamic testing via automata learning. Int J Softw Tools Technol Transfer 11, 307–324 (2009). https://doi.org/10.1007/s10009-009-0120-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-009-0120-7

Keywords

Search

Navigation