Skip to main content
SpringerLink
Log in

Learning-Based Software Testing: A Tutorial

  • Conference paper
Leveraging Applications of Formal Methods, Verification, and Validation (ISoLA 2011)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 336))

Abstract

We present an overview of the paradigm of learning-based testing (LBT) for software systems. LBT is a fully automated method for specification-based black-box testing using computational learning principles. It applies the principle of tests as queries, where queries are either generated by a learning algorithm or by a model checker through use of a formal requirements specification. LBT can be applied to automate black-box testing of a variety of different software architectures including procedural and reactive systems. We describe some different testing platforms which have been designed using this paradigm and some representative evaluation results. We also compare LBT with related testing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Angluin, D.: A note on the number of queries needed to identify regular languages. Information and Control 51(1), 76–87 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  2. Angluin, D.: Learning regular sets from queries and counterexamples. Information and Computation 75(1), 87–106 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  3. Baader, F., Snyder, W.: Unification theory. In: Handbook of Automated Reasoning, pp. 447–531. Elsevier (2001)

    Google Scholar 

  4. Bergadano, F., Gunetti, D.: Testing by means of inductive program learning. ACM Trans. Software Engineering and Methodology 5(2), 119–145 (1996)

    Article  Google Scholar 

  5. Bohlin, T., Jonsson, B.: Regular inference for communication protocol entities. Technical Report 2008-024, Dept. of Information Technology, Uppsala University (2008)

    Google Scholar 

  6. Budd, T.A., Angluin, D.: Two notions of correctness and their relation to testing. Acta Informatica 18, 31–45 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  7. Caviness, B.F., Johnson, J.R.: Quantifier Elimination and Cylindrical Algebraic Decomposition. Springer (1998)

    Google Scholar 

  8. Chauhan, P., Clarke, E.M., Kukula, J.H., Sapra, S., Veith, H., Wang, D.: Automated Abstraction Refinement for Model Checking Large State Spaces Using Sat Based Conflict Analysis. In: Aagaard, M.D., O’Leary, J.W. (eds.) FMCAD 2002. LNCS, vol. 2517, pp. 33–51. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  9. Cherniavsky, J.C., Smith, C.H.: A recursion theoretic approach to program testing. IEEE Transactions on Software Engineering SE-13(7), 777–784 (1987)

    Article  MathSciNet  Google Scholar 

  10. Cimatti, A., Clarke, E., Giunchiglia, F., Roveri, M.: NUSMV: A New Symbolic Model Verifier. In: Halbwachs, N., Peled, D.A. (eds.) CAV 1999. LNCS, vol. 1633, pp. 495–499. Springer, Heidelberg (1999)

    Chapter  Google Scholar 

  11. Clarke, E., Gupta, A., Kukula, J.H., Strichman, O.: SAT Based Abstraction-Refinement Using ILP and Machine Learning Techniques. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 265–279. Springer, Heidelberg (2002)

    Chapter  Google Scholar 

  12. Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press (1999)

    Google Scholar 

  13. Comon, H.: Disunification: a survey. In: Computational Logic: Essays in Honor of Alan Robinson, pp. 322–359. MIT Press (1991)

    Google Scholar 

  14. de la Higuera, C.: Grammatical Inference. Cambridge University Press (2010)

    Google Scholar 

  15. Dupont, P.: Incremental Regular Inference. In: Miclet, L., de la Higuera, C. (eds.) ICGI 1996. LNCS, vol. 1147, pp. 222–237. Springer, Heidelberg (1996)

    Chapter  Google Scholar 

  16. Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., Pretschner, A. (eds.): Model-Based Testing of Reactive Systems. LNCS, vol. 3472. Springer, Heidelberg (2005)

    MATH  Google Scholar 

  17. Fraser, G., Wotawa, F., Ammann, P.E.: Testing with model checkers: A survey. Tech. rep. 2007-p2-04, TU Graz (2007)

    Google Scholar 

  18. Gold, E.M.: Language identification in the limit. Information and Control 10(5), 447–474 (1967)

    Article  MATH  Google Scholar 

  19. Groce, A., Peled, D., Yannakakis, M.: Adaptive model checking. Logic Journal of the IGPL 14(5), 729–744 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  20. Harel, D.: Statecharts: a visual formalism for complex systems. Science of Computer Programming 8, 231–274 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  21. Hullot, J.M.: Canonical Forms and Unification. In: Proc. Fifth Int. Conf. on Automated Deduction. LNCS, vol. 87, pp. 318–334. Springer, Heidelberg (1980)

    Google Scholar 

  22. Vitter, J.S., Romanik, K.: Using vapnik-chervonenkis dimension to analyze the testing complexity of program, segments. Information and Computation 128(2), 87–108 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  23. Krinke, J.: Program slicing. In: Handbook of Software Engineering and Knowledge Engineering. Recent Advances, vol. 3. World Scientific Publishing (2005)

    Google Scholar 

  24. Loeckx, J., Ehrich, H.D., Wolf, M.: Specification of Abstract Data Types. John Wiley & Sons, Inc., New York (1996)

    MATH  Google Scholar 

  25. Meinke, K.: Automated black-box testing of functional correctness using function approximation. In: ISSTA 2004: Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 143–153. ACM, New York (2004)

    Chapter  Google Scholar 

  26. Meinke, K.: CGE: A Sequential Learning Algorithm for Mealy Automata. In: Sempere, J.M., García, P. (eds.) ICGI 2010. LNCS (LNAI), vol. 6339, pp. 148–162. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  27. Meinke, K., Niu, F.: A Learning-Based Approach to Unit Testing of Numerical Software. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 221–235. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  28. Meinke, K., Niu, F.: Learning-Based Testing for Reactive Systems Using Term Rewriting Technology. In: Wolff, B., Zaïdi, F. (eds.) ICTSS 2011. LNCS, vol. 7019, pp. 97–114. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  29. Meinke, K., Sindhu, M.: Correctness and performance of an incremental learning algorithm for finite automata. Technical report, School of Computer Science and Communication, Royal Institute of Technology, Stockholm (2010)

    Google Scholar 

  30. Meinke, K., Sindhu, M.: Incremental Learning-Based Testing for Reactive Systems. In: Gogolla, M., Wolff, B. (eds.) TAP 2011. LNCS, vol. 6706, pp. 134–151. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  31. Niu, F.: Learning-based software testing using symbolic constraint solving methods. Licentiate thesis, School of Computer Science and Communication, Royal Institute of Technology (2011)

    Google Scholar 

  32. Parekh, R., Nichitiu, C., Honavar, V.G.: A Polynomial Time Incremental Algorithm for Learning DFA. In: Honavar, V.G., Slutzki, G. (eds.) ICGI 1998. LNCS (LNAI), vol. 1433, pp. 37–49. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  33. Peled, D., Vardi, M.Y., Yannakakis, M.: Black-box checking. In: Formal Methods for Protocol Engineering and Distributed Systems FORTE/PSTV, pp. 225–240. Kluwer (1999)

    Google Scholar 

  34. Phadke, M.S.: Planning efficient software tests. Crosstalk 10(10), 11–15 (1997)

    Google Scholar 

  35. Phillips, E.R.: An Introduction to Analysis and Integration Theory. Dover, New York (1984)

    Google Scholar 

  36. Poston, R.M.: Automating Specification-Based Software Testing. IEEE Computer Society Press, Los Alamitos (1997)

    Google Scholar 

  37. Raffelt, H., Steffen, B., Margaria, T.: Dynamic Testing Via Automata Learning. In: Yorav, K. (ed.) HVC 2007. LNCS, vol. 4899, pp. 136–152. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  38. Reimer, M.: Multivariate Polynomial Approximation. Birkhauser, Basel (2003)

    Book  MATH  Google Scholar 

  39. Romanik, K.: Approximate testing and its relationship to learning. Theoret. Comp. Sci. 188, 79–99 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  40. Sindhu, M.: Incremental Learning and Testing of Reactive Systems. Licentiate thesis, School of Computer Science and Communication, Royal Institute of Technology, Stockholm, Sweden (2011)

    Google Scholar 

  41. Walkinshaw, N., Bogdanov, K., Derrick, J., Paris, J.: Increasing Functional Coverage by Inductive Testing: A Case Study. In: Petrenko, A., Simão, A., Maldonado, J.C. (eds.) ICTSS 2010. LNCS, vol. 6435, pp. 126–141. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  42. Weyuker, E.: Assessing test data adequacy through program inference. ACM Trans. Program. Lang. Syst. 5(4), 641–655 (1983)

    Article  MATH  Google Scholar 

  43. Zhu, H.: A formal interpretation of software testing as inductive inference. Journal of Software Testing, Verification and Reliability 6(1), 3–31 (1996)

    Article  Google Scholar 

  44. Zhu, H., Hall, P., May, J.: Inductive inference and software testing. Journal of Software Testing, Verification and Reliability 2(2), 3–31 (1992)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Communication, Royal Institute of Technology, 100-44, Stockholm, Sweden

    Karl Meinke, F. Niu & M. Sindhu

Authors
  1. Karl Meinke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. F. Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Sindhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Computer Science, Software Engineering Group, TU Darmstadt, Hochschulstr. 10, 64289, Darmstadt, Germany

    Reiner Hähnle

  2. Institute of Computer Languages, TU Vienna, Faculty of Informatics, Argentinierstr. 8 / E185.1, 1040, Vienna, Austria

    Jens Knoop

  3. Institute of Informatics, Service and Software Engineering Group, University Potsdam, August-Bebel-Str. 89, 14482, Potsdam, Germany

    Tiziana Margaria

  4. Faculty of Informatics, Institute of Computer Languages, TU Vienna, Argentinierstr. 8 / E185.1, 1040, Vienna, Austria

    Dietmar Schreiner

  5. Faculty of Informatics, Chair of Programming Systems, TU Dortmund, Otto-Hahn-Str. 14, 44227, Dortmund, Germany

    Bernhard Steffen

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Meinke, K., Niu, F., Sindhu, M. (2012). Learning-Based Software Testing: A Tutorial. In: Hähnle, R., Knoop, J., Margaria, T., Schreiner, D., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification, and Validation. ISoLA 2011. Communications in Computer and Information Science, vol 336. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34781-8_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34781-8_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34780-1

  • Online ISBN: 978-3-642-34781-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation