Incrementally Discovering Testable Specifications from Program Executions

  • Neil Walkinshaw
  • John Derrick
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6286)


The ProTest project is an EU FP7 project to develop techniques that improve the testing and verification of concurrent and distributed software systems. One of the four main work packages is concerned with the automated identification of specifications that could serve as a suitable basis for testing; this is currently a tedious and error-prone manual task that tends to be neglected in practice. This paper describes how this problem has been addressed in the ProTest project. It describes a technique that uses test executions to refine the specification from which they are generated. It shows how the technique has been implemented and applied to real Erlang systems. It also describes in detail the major challenges that remain to be addressed in future work.


State Machine Transmission Control Protocol Program Execution Label Transition System Execution Trace 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Derrick, J., Walkinshaw, N.: Property-based testing: The protest project. In: FMCO (2009)Google Scholar
  2. 2.
    Armstrong, J.: Programming Erlang: Software for a Concurrent World. Pragmatic Bookshelf (July 2007)Google Scholar
  3. 3.
    Walkinshaw, N., Derrick, J., Guo, Q.: Iterative refinement of reverse-engineered models by model-based testing. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 305–320. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  4. 4.
    Walkinshaw, N., Bogdanov, K., Holcombe, M., Salahuddin, S.: Improving Dynamic Software Analysis by Applying Grammar Inference Principles. Journal of Software Maintenance and Evolution: Research and Practice (2008)Google Scholar
  5. 5.
    Claessen, K., Hughes, J.: Quickcheck: A Lightweight Tool for Random Testing of Haskell Programs. In: Proceedings of the International Conference on Functional Programming (ICFP), pp. 268–279 (2000)Google Scholar
  6. 6.
    Walkinshaw, N., Bogdanov, K., Ali, S., Holcombe, M.: Automated discovery of state transitions and their functions in source code. Software Testing, Verification and Reliability 18(2) (2008)Google Scholar
  7. 7.
    Moore, E.F.: Gedanken–experiments on sequential machines. In: Shannon, C.E., McCarthy, J. (eds.) Annals of Mathematics Studies (34), Automata Studies, pp. 129–153. Princeton University Press, Princeton (1956)Google Scholar
  8. 8.
    Nerode, A.: Linear automata transformations. Proceedings of the American Mathematical Society 9, 541–544 (1958)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Gold, E.: Language Identification in the Limit. Information and Control 10, 447–474 (1967)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Biermann, A., Feldman, J.: On the Synthesis of Finite-State Machines from Samples of their Behavior. IEEE Transactions on Computers 21, 592–597 (1972)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Dupont, P., Lambeau, B., Damas, C., van Lamsweerde, A.: The QSM Algorithm and its Application to Software Behavior Model Induction. Applied Artificial Intelligence 22, 77–115 (2008)CrossRefGoogle Scholar
  12. 12.
    Walkinshaw, N., Bogdanov, K., Holcombe, M., Salahuddin, S.: Reverse Engineering State Machines by Interactive Grammar Inference. In: 14th IEEE International Working Conference on Reverse Engineering, WCRE (2007)Google Scholar
  13. 13.
    Cook, J., Wolf, A.: Discovering Models of Software Processes from Event-Based Data. ACM Transactions on Software Engineering and Methodology 7(3), 215–249 (1998)CrossRefGoogle Scholar
  14. 14.
    Ammons, G., Bodík, R., Larus, J.: Mining Specifications. In: 29th SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), Portland, Oregon, pp. 4–16 (2002)Google Scholar
  15. 15.
    Angluin, D.: Learning Regular Sets from Queries and Counterexamples. Information and Computation 75, 87–106 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Shahbaz, M., Groz, R.: Inferring mealy machines. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 207–222. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Lang, K., Pearlmutter, B., Price, R.: Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm. In: Honavar, V.G., Slutzki, G. (eds.) ICGI 1998. LNCS (LNAI), vol. 1433, pp. 1–12. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  18. 18.
    Biermann, A.W., Krishnaswamy, R.: Constructing programs from example computations. IEEE Trans. on Software Engineering SE 2, 141–153 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Cicchello, O., Kremer, S.: Inducing grammars from sparse data sets: A survey of algorithms and results. Journal of Machine Learning Research 4, 603–632 (2003)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Cheng, K., Krishnakumar, A.: Automatic functional test generation using the extended finite state machine model. In: 30th ACM/IEEE Design Automation Conference, pp. 86–91 (1993)Google Scholar
  21. 21.
    Walkinshaw, N., Bogdanov, K.: Inferring Finite-State Models with Temporal Constraints. In: Proceedings of the 23rd International Conference on Automated Software Engineering, ASE (2008)Google Scholar
  22. 22.
    Paris, J., Arts, T.: Automatic testing of tcp/ip implementations using quickcheck. In: Proceedings of the 8th ACM SIGPLAN workshop on Erlang, Erlang 2009, pp. 83–92. ACM, New York (2009)Google Scholar
  23. 23.
    Postel, J.: Transmission control protocol. Technical Report 793, DDN Network Information Center, SRI International, RFC (September 1981)Google Scholar
  24. 24.
    Ostrand, T., Balcer, M.: The category-partition method for specifying and generating functional tests. Communications of the ACM 31(6), 676–686 (1988)CrossRefGoogle Scholar
  25. 25.
    Damas, C., Lambeau, B., Dupont, P., van Lamsweerde, A.: Generating Annotated Behavior Models from End-User Scenarios. IEEE Transactions on Software Engineering 31(12), 1056–1073 (2005)CrossRefGoogle Scholar
  26. 26.
    Lorenzoli, D., Mariani, L., Pezzè, M.: Automatic generation of software behavioral models. In: Proceedings of the 30th international conference on Software engineering, ICSE 2008, pp. 501–510. ACM, New York (2008)Google Scholar
  27. 27.
    Ernst, M., Cockrell, J., Griswold, W., Notkin, D.: Dynamically Discovering Likely Program Invariants to Support Program Evolution. Transactions on Software Engineering 27(2), 1–25 (2001)CrossRefGoogle Scholar
  28. 28.
    Bongard, J.C., Lipson, H.: Nonlinear system identification using coevolution of models and tests. IEEE Trans. Evolutionary Computation 9(4), 361–384 (2005)CrossRefzbMATHGoogle Scholar
  29. 29.
    Cornelissen, B., Zaidman, A., Holten, D., Moonen, L., van Deursen, A., van Wijk, J.: Execution trace analysis through massive sequence and circular bundle views. Journal of Systems and Software 81(12), 2252–2268 (2008)CrossRefGoogle Scholar
  30. 30.
    Afshan, S., McMinn, P., Walkinshaw, N.: Using dictionary compression algorithms to identify phases in program traces. Technical Report CS-10-01, Department of Computer Science, The University of Sheffield (2010)Google Scholar
  31. 31.
    Walkinshaw, N., Afshan, S., McMinn, P.: Using compression algorithms to support the comprehension of program traces. In: Proceedings of the Eighth International Workshop on Dynamic Analysis (WODA 2010). ACM, New York (2010)Google Scholar
  32. 32.
    Nevill-Manning, C., Witten, I.: Compression and explanation using hierarchical grammars. Computer Journal 40(2/3), 103–116 (1997)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Neil Walkinshaw
    • 1
  • John Derrick
    • 1
  1. 1.Department of Computer ScienceThe University of SheffieldSheffieldUK

Personalised recommendations