Skip to main content
SpringerLink
Log in

“Testing Can Be Formal Too”: 30 Years Later

  • Chapter
  • First Online:
The French School of Programming

  • 46 Accesses

Abstract

This chapter relates a series of works that contributed to the recognition of the use of formal specifications when testing software.

After presenting a first set of results on test methods based on algebraic data types, generalisations to other formal approaches are described.

The outcome of several case studies is reported, and as conclusion, the place of formal testing within the software validation and verification activities is discussed. A concrete example is developed in appendix.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aiguier, M., Arnould, A., Boin, C., Gall, P.L., Marre, B.: Testing from algebraic specifications: Test data set selection by unfolding axioms. In: Formal Approaches to Software Testing, 5th International Workshop, FATES. Lecture Notes in Computer Science, vol. 3997, pp. 203–217. Springer (2005)

    Google Scholar 

  2. Arnold, A., Gaudel, M., Marre, B.: An experiment on the validation of a specification by heterogeneous formal means: The transit node. In: 5th IFIP Working Conference on Dependable Computing for Critical Applications (DCCA5). pp. 24–34 (1995)

    Google Scholar 

  3. Barbey, S., Buchs, D.: Testing Ada abstract data types using formal specifications. In: 1st Int. Eurospace-Ada-Europe Symposium. pp. 76–89. No. 887 in Lecture Notes in Computer Science, Springer Verlag (1994)

    Google Scholar 

  4. Barbey, S., Buchs, D., Péraire, C.: A theory of specification-based testing for object-oriented software. In: EDCC-2, Second European Dependable Computing Conference,Proceedings. Lecture Notes in Computer Science, vol. 1150, pp. 303–320. Springer (1996)

    Google Scholar 

  5. Barr, E.T., Harman, M., McMinn, P., Shahbaz, M., Yoo, S.: The oracle problem in software testing: A survey. IEEE Trans. Software Eng. 41(5), 507–525 (2015)

    Article  Google Scholar 

  6. Bernot, G., Gaudel, M.C., Marre, B.: Software testing based on formal specifications: a theory and a tool. Software Engineering Journal 6(6), 387–405 (1991)

    Article  Google Scholar 

  7. Bidoit, M., Hennicker, R., Wirsing, M.: Behavioural and abstractor specifications. Science of Computer Programming 25(2–3), 149–186 (1995)

    Article  MathSciNet  Google Scholar 

  8. Bougé, L., Choquet, N., Fribourg, L., Gaudel, M.C.: Test set generation from algebraic specifications using logic programming. Journal of Systems and Software 6(4), 343–360 (1986)

    Article  Google Scholar 

  9. Bougé, L.: Modeling the notion of program testing; application to test set generation. Thesis, Université Pierre et Marie Curie - Paris VI (Oct 1982), https://tel.archives-ouvertes.fr/tel-00416558, directeur: Prof. Marie-Claude Gaudel

  10. Bougé, L., Choquet, N., Fribourg, L., Gaudel, M.C.: Application of Prolog to test sets generation from algebraic specifications. In: International Joint Conference on Theory and Practice of Software Development (TAPSOFT). LNCS, vol. 186, pp. 261–275. Springer (1985)

    Google Scholar 

  11. Brinksma, E.: A theory for the derivation of tests. In: Proc. 8th Int. Conf. Protocol Specification,Testing and Verification. pp. 63–74. North-Holland (1988)

    Google Scholar 

  12. Brucker, A.D., Brügger, L., Wolff, B.: Verifying test-hypotheses – an experiment in test and proof. In: Model-based Testing MBT 2008. Electronic Notes in Theoretical Computer Science, vol. 202, pp. 15–28. Elsevier Science Publishers (2008)

    Google Scholar 

  13. Brucker, A.D., Wolff, B.: Symbolic test case generation for primitive recursive functions. In: Formal Approaches to Testing of Software (FATES 04). Lecture Notes in Computer Science, vol. 3395, pp. 16–32. Springer Verlag (2005)

    Google Scholar 

  14. Brucker, A.D., Wolff, B.: On theorem prover-based testing. Formal Asp. Comput. 25(5), 683–721 (2013)

    Article  MathSciNet  Google Scholar 

  15. Cavalcanti, A., Gaudel, M.C.: Testing for refinement in CSP. In: Formal Methods and Software Engineering, ICFEM 2007. Lecture Notes in Computer Science, vol. 4789, pp. 151–170. Springer Verlag (2007)

    Google Scholar 

  16. Cavalcanti, A., Gaudel, M.C.: Testing for refinement in Circus. Acta Informatica 48(2), 97–147 (2011)

    Article  MathSciNet  Google Scholar 

  17. Chen, H.Y., Tse, T.H., Chan, F.T., Chen, T.Y.: In black and white: an integrated approach to class-level testing of object-oriented programs. ACM transactions on Software Engineering and Methodology 7(3), 250–295 (1998)

    Article  Google Scholar 

  18. Chen, H.Y., Tse, T.H.: Equality to equals and unequals: A revisit of the equivalence and nonequivalence criteria in class-level testing of object-oriented software. IEEE Trans. Software Eng. 39(11), 1549–1563 (2013)

    Article  Google Scholar 

  19. Chen, H.Y., Tse, T.H., Chen, T.Y.: TACCLE: a methodology for object-oriented software testing at the class and cluster levels. ACM Trans. Softw. Eng. Methodol. 10(1), 56–109 (2001)

    Article  Google Scholar 

  20. Chow, T.: Testing software design modeled by finite-state machines. IEEE Transactions on Software Engineering SE-4(3), 178–187 (1978)

    Article  Google Scholar 

  21. Dauchy, P., Gaudel, M.C., Marre, B.: Using algebraic specifications in software testing : a case study on the software of an automatic subway. Journal of Systems and Software 21(3), 229–244 (1993)

    Article  Google Scholar 

  22. Dauchy, P., Ozello, P.: Experiments with formal specifications on MAGGALY. In: Second International Conference on Applications of Advanced Technologies in Transportation Engineering. Mineapolis (1991)

    Google Scholar 

  23. Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: FME ’93: Industrial-Strength Formal Methods, First International Symposium of Formal Methods Europe. LNCS, vol. 670, pp. 268–284. Springer (1993)

    Google Scholar 

  24. Dong, R.K., Frankl, P.G.: The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology 3(2), 103–130 (1994)

    Google Scholar 

  25. Feliachi, A.: Semantics-Based Testing for Circus. Ph.D. thesis, Université Paris Sud (December 2012), directeurs: Prof. Marie-Claude Gaudel and Prof. Burkhart Wolff

    Google Scholar 

  26. Feliachi, A., Gaudel, M.C., Wenzel, M., Wolff, B.: The Circus testing theory revisited in Isabelle/HOL. In: Formal Methods and Software Engineering - 15th International Conference on Formal Engineering Methods, ICFEM 2013. pp. 131–147 (2013)

    Google Scholar 

  27. Fujiwara, S., von Bochmann, G.: Testing non-deterministic state machines with fault coverage. In: Protocol Test Systems, IV, Proceedings of the IFIP TC6/WG6.1 Fourth International Workshop on Protocol Test Systems. IFIP Transactions, vol. C-3, pp. 267–280. North-Holland (1991)

    Google Scholar 

  28. Gannon, J.D., McMullin, P.R., Hamlet, R.G.: Data-abstraction implementation, specification, and testing. ACM Trans. Program. Lang. Syst. 3(3), 211–223 (1981)

    Article  Google Scholar 

  29. Gaudel, M.C.: Testing can be formal, too. In: TAPSOFT9́5, International Joint Conference, Theory And Practice of Software Development. Lecture Notes in Computer Science, vol. 915, pp. 82–96. Springer Verlag (1995)

    Google Scholar 

  30. Gaudel, M.C., James, P.R.: Testing algebraic data types and processes: A unifying theory. Formal Asp. Comput. 10(5–6), 436–451 (1998)

    Article  Google Scholar 

  31. Gaudel, M.C., Le Gall, P.: Testing data types implementations from algebraic specifications. In: Formal Methods and Testing, An Outcome of the FORTEST Network, Revised Selected Papers, R. Hierons, J. Bowen, and M. Harman,eds, LNCS, vol. 4949, pp. 209–239. Springer-Verlag (2007)

    Google Scholar 

  32. Goodenough, J.B., Gerhart, S.: Toward a theory of test data selection. IEEE Transactions on Software Engineering SE-1(2), 156–173 (1975)

    Article  MathSciNet  Google Scholar 

  33. Helke, S., Neustupny, T., Santen, T.: Automating test case generation from Z specifications with Isabelle. In: ZUM. pp. 52–71 (1997)

    Google Scholar 

  34. Hennicker, R., Wirsing, M., Bidoit, M.: Proof systems for structured specifications with observability operators. Theor. Comput. Sci. 173(2), 393–443 (1997)

    Article  MathSciNet  Google Scholar 

  35. Hierons, R.M.: Comparing test sets and criteria in the presence of test hypotheses and fault domains. ACM Trans. Softw. Eng. Methodol. 11(4), 427–448 (2002)

    Article  Google Scholar 

  36. Hierons, R.M., Bogdanov, K., Bowen, J.P., Cleaveland, R., Derrick, J., Dick, J., Gheorghe, M., Harman, M., Kapoor, K., Krause, P.J., Lüttgen, G., Simons, A.J.H., Vilkomir, S.A., Woodward, M.R., Zedan, H.: Using formal specifications to support testing. ACM Comput. Surv. 41(2), 9:1–9:76 (2009)

    Google Scholar 

  37. Hoare, C.A.R., He, J.: Unifying theories of programming. Prentice-Hall (1998)

    Google Scholar 

  38. Jalote, P.: Specification and testing of abstract data types. In: IEEE International Computer Software and Applications Conference COMPSAC. pp. 508–511 (1983)

    Google Scholar 

  39. James, P.R., Endler, M., Gaudel, M.C.: Development of an atomic broadcast protocol using LOTOS. Software Practice and Experience 29(8), 699–719 (1999)

    Article  Google Scholar 

  40. Jard, C., Jéron, T.: TGV: theory, principles and algorithms. Int. J. on Software Tools for Technology Transfer 7(4), 297–315 (2005)

    Article  Google Scholar 

  41. Le Gall, P., Arnould, A.: Formal specification and test: correctness and oracle. In: 11th WADT joint with the 9th general COMPASS workshop. Lecture Notes in Computer Science, vol. 1130, pp. 342–358. Springer (1996)

    Google Scholar 

  42. Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines-a survey. Proceedings of the IEEE 84(8), 1090–1123 (1996)

    Article  Google Scholar 

  43. Legeard, B., Peureux, F.: Generation of functional test sequences from B formal specifications-presentation and industrial case study. In: 16th IEEE International Conference on Automated Software Engineering (ASE 2001). pp. 377–381. IEEE Computer Society (2001)

    Google Scholar 

  44. Lestiennes, G., Gaudel, M.: Testing processes from formal specifications with inputs, outputs and data types. In: 13th International Symposium on Software Reliability Engineering (ISSRE ). pp. 3–14. IEEE Computer Society (2002)

    Google Scholar 

  45. Lestiennes, G., Gaudel, M.C.: Test de systèmes réactifs non réceptifs. Journal européen des systèmes automatisés 39(1 - 3), 255–270 (2005)

    Google Scholar 

  46. Machado, P.: On oracles for interpreting test results against algebraic specifications. In: Algebraic Methodology and Software Technology, AMAST’98. Lecture Notes in Computer Science, vol. 1548, pp. 502–518 (1998)

    Article  Google Scholar 

  47. Machado, P.: Testing from structured algebraic specifications. In: AMAST2000. Lecture Notes in Computer Science, vol. 1816, pp. 529–544 (2000)

    Article  Google Scholar 

  48. Marre, B.: LOFT: a tool for assisting selection of test data sets from algebraic specifications. In: TAPSOFT. Lecture Notes in Computer Science, vol. 915, pp. 799–800. Springer Verlag (1995)

    Google Scholar 

  49. Marre, B., Arnould, A.: Test sequences generation from Lustre descriptions: GATeL. In: ASE-00: The 15th IEEE Conference on Automated Software Engineering. pp. 229–237. IEEE CS Press (2000)

    Google Scholar 

  50. Marre, B., Thévenod-Fosse, P., Waeselink, H., Gall, P.L., Crouzet, Y.: An experimental evaluation of formal testing and statistical testing. In: SAFECOMP’92. pp. 311–316 (1992)

    Google Scholar 

  51. Marre, B.: Toward automatic test data set selection using algebraic specifications and logic programming. In: Logic Programming, Proceedings of the 8th International Conference. pp. 202–219. MIT Press (1991)

    Google Scholar 

  52. Marre, B., Blanc, B.: Test selection strategies for Lustre descriptions in GATeL. Electr. Notes Theor. Comput. Sci. 111, 93–111 (2005)

    Article  Google Scholar 

  53. Péraire, C., Barbey, S., Buchs, D.: Test selection for object-oriented software based on formal specifications. In: IFIP Working Conference on Programming Concepts and Methods (PROCOMET’98). pp. 385–403. Chapman Hall (1998)

    Google Scholar 

  54. Sarikaya, B., von Bochmann, G.: Some experience with test sequence generation for protocols. In: Protocol Specification, Testing and Verification, Proceedings of the IFIP WG6.1 Second International Workshop on Protocol Specification, Testing and Verification. pp. 555–567. North-Holland (1982)

    Google Scholar 

  55. Stocks, P., Carrington, D.A.: Test templates: A specification-based testing framework. In: Proceedings of the 15th International Conference on Software Engineering,ICSE. pp. 405–414. IEEE Computer Society / ACM Press (1993)

    Google Scholar 

  56. Tretmans, J.: Test generation with inputs, outputs and repetitive quiescence. Software - Concepts and Tools 17(3), 103–120 (1996)

    Google Scholar 

  57. Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with Spec Explorer. In: Formal Methods and Testing, An Outcome of the FORTEST Network, revised selected papers, R. Hierons, J. Bowen, and M. Harman, eds. Lecture Notes in Computer Science, vol. 4949, pp. 39–76. Springer (2008)

    Google Scholar 

  58. Weyuker, E.J.: On testing non-testable programs. Comput. J. 25(4), 465–470 (1982)

    Article  Google Scholar 

  59. Weyuker, E.J., Ostrand, T.J.: Theories of program testing and the the application of revealing subdomains. IEEE Trans. Software Eng. 6(3), 236–246 (1980)

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Université de Paris-Sud et CNRS, Paris, France

    Marie-Claude Gaudel

Authors
  1. Marie-Claude Gaudel
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Constructor Institute, Schaffhausen, Switzerland

    Bertrand Meyer

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gaudel, MC. (2024). “Testing Can Be Formal Too”: 30 Years Later. In: Meyer, B. (eds) The French School of Programming. Springer, Cham. https://doi.org/10.1007/978-3-031-34518-0_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34518-0_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34517-3

  • Online ISBN: 978-3-031-34518-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation