Machine Learning

, Volume 96, Issue 1–2, pp 189–224 | Cite as

Improving active Mealy machine learning for protocol conformance testing

  • Fides Aarts
  • Harco Kuppens
  • Jan Tretmans
  • Frits Vaandrager
  • Sicco Verwer


Using a well-known industrial case study from the verification literature, the bounded retransmission protocol, we show how active learning can be used to establish the correctness of protocol implementation I relative to a given reference implementation R. Using active learning, we learn a model MR of reference implementation R, which serves as input for a model-based testing tool that checks conformance of implementation I to MR. In addition, we also explore an alternative approach in which we learn a model MI of implementation I, which is compared to model MR using an equivalence checker. Our work uses a unique combination of software tools for model construction (Uppaal), active learning (LearnLib, Tomte), model-based testing (JTorX, TorXakis) and verification (CADP, MRMC). We show how these tools can be used for learning models of and revealing errors in implementations, present the new notion of a conformance oracle, and demonstrate how conformance oracles can be used to speed up conformance checking.


Active learning Automaton learning Mealy machines State machine synthesis Model-based testing Protocol learning Model checking 


  1. Aarts, F., & Vaandrager, F. (2010). Learning I/O automata. In Lecture notes in computer science: Vol. 6269. Proceedings of the 21st international conference on concurrency theory, CONCUR (pp. 71–85). Berlin: Springer. Google Scholar
  2. Aarts, F., Jonsson, B., & Uijen, J. (2010a). Generating models of infinite-state communication protocols using regular inference with abstraction. In Proceedings of the 22nd IFIP WG 6.1 international conference on testing software and systems, ICTSS’10 (pp. 188–204). Berlin: Springer. Google Scholar
  3. Aarts, F., Schmaltz, J., & Vaandrager, F. (2010b). Inference and abstraction of the biometric passport. In Proceedings of the 4th international conference on leveraging applications of formal methods, verification, and validation—volume part I (pp. 673–686). Berlin: Springer. CrossRefGoogle Scholar
  4. Aarts, F., Heidarian, F., Kuppens, H., Olsen, P., & Vaandrager, F. (2012a). Automata learning through counterexample-guided abstraction refinement. In Lecture notes in computer science: Vol. 7436. Proceedings of the 18th international symposium on formal methods (FM 2012) (pp. 10–27). Berlin: Springer. Google Scholar
  5. Aarts, F., Kuppens, H., Tretmans, J., Vaandrager, F., & Verwer, S. (2012b). Learning and testing the bounded retransmission protocol. In JMLR workshop and conference proceedings: Vol. 21. Proceedings of the 11th international conference on grammatical inference (ICGI 2012) (pp. 4–18). JMLR. Google Scholar
  6. Alur, R., & Dill, D. L. (1994). A theory of timed automata. Theoretical Computer Science, 126, 183–235. MATHMathSciNetCrossRefGoogle Scholar
  7. Ammons, G., Bodik, R., & Larus, J. R. (2002). Mining specifications. In Proceedings of the 29th symposium on principles of programming languages (pp. 4–16). New York: ACM. Google Scholar
  8. Angluin, D. (1987). Learning regular sets from queries and counterexamples. Information and Computation, 75(2), 87–106. MATHMathSciNetCrossRefGoogle Scholar
  9. Antunes, J., Neves, N., & Verissimo, P. (2011). Reverse engineering of protocols from network traces. In Proceedings of the working conference on reverse engineering (pp. 169–178). Google Scholar
  10. Aziz, A., Sanwal, K., Singhal, V., & Brayton, R. (1996). Verifying continuous time Markov chains. In Lecture notes in computer science: Vol. 1102. Proceedings of 8th international conference on computer aided verification (CAV) (pp. 269–276). Berlin: Springer. Google Scholar
  11. Balcázar, J., Dıaz, J., Gavaldá, R., & Watanabe, O. (1997). Algorithms for learning finite automata from queries: a unified view. Advances in Algorithms, Languages, and Complexity, 53–72. Google Scholar
  12. Bartlett, K., Scantlebury, R., & Wilkinson, P. (1969). A note on reliable full–duplex transmission over half–duplex links. Communications of the ACM, 12, 260–261. CrossRefGoogle Scholar
  13. Behrmann, G., David, A., & Larsen, K. (2004). A tutorial on Uppaal. In Lecture notes in computer science: Vol. 3185. Formal methods for the design of real-time systems (pp. 33–35). Berlin: Springer. CrossRefGoogle Scholar
  14. Belinfante, A. (2010). Jtorx: a tool for on-line model-driven test derivation and execution. In Lecture notes in computer science: Vol. 6015. Proceedings of the 16th international conference on tools and algorithms for the construction and analysis of systems (TACAS) (pp. 266–270). Berlin: Springer. CrossRefGoogle Scholar
  15. Berg, T., Grinchtein, O., Jonsson, B., Leucker, M., Raffelt, H., & Steffen, B. (2005). On the correspondence between conformance testing and regular inference. In Lecture notes in computer science: Vol. 3442. Proceedings of the 8th international conference on fundamental approaches to software engineering (FASE) (pp. 175–189). Berlin: Springer. CrossRefGoogle Scholar
  16. Bertolino, A., Inverardi, P., Pelliccione, P., & Tivoli, M. (2009). Automatic synthesis of behavior protocols for composable web-services. In Proceedings of the joint meeting of the 12th European software engineering conference and the 17th ACM SIGSOFT symposium on the foundations of software engineering (pp. 141–150). New York: ACM. Google Scholar
  17. Broy, M., Jonsson, B., Katoen, J.-P., Leucker, M., & Pretschner, A. (Eds.) (2005). Lecture notes in computer science.: Vol. 3472. Model-based testing of reactive systems. Berlin: Springer. MATHGoogle Scholar
  18. Castro, J., & Gavaldà, R. (2008). Towards feasible PAC-learning of probabilistic deterministic finite automata. In Lecture notes in computer science: Vol. 5278. Proceedings of the 9th international colloquium on grammatical inference: algorithms and applications (ICGI) (pp. 163–174). Berlin: Springer. CrossRefGoogle Scholar
  19. Cho, C. Y., Babic, D., Shin, E. C. R., & Song, D. (2010). Inference and analysis of formal models of botnet command and control protocols. In Proceedings of the 17th ACM conference on computer and communications security (pp. 426–439). New York: ACM. CrossRefGoogle Scholar
  20. Clark, A., & Thollard, F. (2004). PAC-learnability of probabilistic deterministic finite state automata. Journal of Machine Learning Research, 473–497. Google Scholar
  21. Clarke, E. (1997). Model checking. In Lecture notes in computer science: Vol. 1346. Proceedings of the 17th conference on foundations of software technology and theoretical computer science (pp. 54–56). Berlin: Springer. CrossRefGoogle Scholar
  22. Combe, D., de la Higuera, C., & Janodet, J.-C. (2010). Zulu: an interactive learning competition. In Lecture notes in computer science: Vol. 6062. Proceedings of the 9th international workshop on finite-state methods and natural language processing (pp. 139–146). Berlin: Springer. CrossRefGoogle Scholar
  23. Comparetti, P., Wondracek, G., Kruegel, C., & Kirda, E. (2009). Prospex: protocol specification extraction. In Proceedings of the 30th IEEE symposium on security and privacy (pp. 110–125). New York: IEEE. Google Scholar
  24. Cook, J. E., & Wolf, A. L. (1998). Discovering models of software processes from event-based data. ACM Transactions on Software Engineering and Methodology, 7, 215–249. CrossRefGoogle Scholar
  25. Cui, W., Kannan, J., & Wang, H. J. (2007). Discoverer: automatic protocol reverse engineering from network traces. In Proceedings of 16th USENIX security symposium (p. 14). Google Scholar
  26. Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C., Patton, G., & Horowitz, B. (1999). Model-based testing in practice. In Proceedings of the 1999 international conference on software engineering, 1999 (pp. 285–294). New York: IEEE. Google Scholar
  27. Dallmeier, V., Lindig, C., Wasylkowski, A., & Zeller, A. (2006). Mining object behavior with ADABU. In Proceedings of the 2006 international workshop on dynamic systems analysis (WODA) (pp. 17–24). New York: ACM. CrossRefGoogle Scholar
  28. D’Argenio, P., Katoen, J.-P., Ruys, T., & Tretmans, J. (1997). The bounded retransmission protocol must be on time! In Lecture notes in computer science: Vol. 1217. Proceedings of the 3rd workshop on tools and algorithms for the construction and analysis of systems (pp. 416–431). Berlin: Springer. CrossRefGoogle Scholar
  29. de la Higuera, C. (2010). Grammatical inference: learning automata and grammars. New York: Cambridge University Press. Google Scholar
  30. de la Higuera, C., & Janodet, J.-C. (2004). Inference of omega-languages from prefixes. Theoretical Computer Science, 313(2), 295–312. MATHMathSciNetCrossRefGoogle Scholar
  31. Denis, F., Lemay, A., & Terlutte, A. (2000). Learning regular languages using non deterministic finite automata. In Proceedings of the 6th international colloquium on grammatical inference (ICGI) (pp. 39–50). Google Scholar
  32. Dijkstra, E. (1969). Notes on structured programming. Google Scholar
  33. Frantzen, L., Tretmans, J., & Willemse, T. (2005). Test generation based on symbolic specifications. In Lecture notes in computer science: Vol. 3395. Proceedings of the 5th international workshop on formal approaches to software testing (pp. 1–15). Berlin: Springer. CrossRefGoogle Scholar
  34. Garavel, H., Lang, F., Mateescu, R., & Serwe, W. (2011). CADP 2010: a toolbox for the construction and analysis of distributed processes. In Lecture notes in computer science: Vol. 6605. Proceedings of the 17th international conference on tools and algorithms for the construction and analysis of systems (pp. 372–387). Berlin: Springer. CrossRefGoogle Scholar
  35. Gold, E. M. (1978). Complexity of automaton identification from given data. Information and Control, 37(3), 302–320. MATHMathSciNetCrossRefGoogle Scholar
  36. Grinchtein, O., Jonsson, B., & Petterson, P. (2006). Inference of event-recording automata using timed decision trees. In Lecture notes in computer science: Vol. 4137. Proceedings of the 17th international conference on concurrency theory (CONCUR) (pp. 435–449). Berlin: Springer. Google Scholar
  37. Hansson, H., & Jonsson, B. (1994). A logic for reasoning about time and reliability. Formal Aspects of Computing, 6, 512–535. MATHCrossRefGoogle Scholar
  38. Helmink, L., Sellink, M., & Vaandrager, F. (1994). Proof-checking a data link protocol. In Lecture notes in computer science: Vol. 806. Proceedings international workshop TYPES’93 (pp. 127–165). Berlin: Springer. Google Scholar
  39. Henzinger, T., Nicollin, X., Sifakis, J., & Yovine, S. (1994). Symbolic model checking for real-time systems. Information and Computation, 111(2), 193–244. MATHMathSciNetCrossRefGoogle Scholar
  40. Hungar, H., Niese, O., & Steffen, B. (2003). Domain-specific optimization in automata learning. In Lecture notes in computer science: Vol. 2725. Proceedings of the 15th international conference on computer aided verification (CAV) (pp. 315–327). Berlin: Springer. CrossRefGoogle Scholar
  41. Ip, C., & Dill, D. (1996). Better verification through symmetry. Formal Methods in System Design, 9(1/2), 41–75. Google Scholar
  42. Katoen, J.-P., Zapreev, I. S., Hahn, E. M., Hermanns, H., & Jansen, D. N. (2011). The ins and outs of the probabilistic model checker MRMC. Performance Evaluation, 68(2), 90–104. CrossRefGoogle Scholar
  43. Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge: MIT Press. Google Scholar
  44. Lee, D., & Yannakakis, M. (1996). Principles and methods for testing finite state machines—a survey. Proceedings of the IEEE, 84(8), 1090–1123. CrossRefGoogle Scholar
  45. Margaria, T., Niese, O., Raffelt, H., & Steffen, B. (2004). Efficient test-based model generation for legacy reactive systems. In Proceedings of the 9th IEEE international high-level design validation and test workshop (HLDVT) (pp. 95–100). Washington: IEEE Computer Society. Google Scholar
  46. Mariani, L., Pastore, F., & Pezze, M. (2011). Dynamic analysis for diagnosing integration faults. IEEE Transactions on Software Engineering, 37, 486–508. CrossRefGoogle Scholar
  47. Meinke, K., & Walkinshaw, N. (2012). Model-based testing and model inference. In Lecture notes in computer science: Vol. 7609. Proceedings of the 5th international symposium on leveraging applications of formal methods, verification and validation (ISoLA) (pp. 440–443). Berlin: Springer. Google Scholar
  48. Mostowski, W., Poll, E., Schmaltz, J., Tretmans, J., & Wichers Schreur, R. (2009). Model-based testing of electronic passports. In Lecture notes in computer science: Vol. 5825. Proceedings of the 15th international workshop on formal methods for industrial critical systems (pp. 207–209). Berlin: Springer. CrossRefGoogle Scholar
  49. Puterman, M. L. (1994). Markov decision processes: discrete stochastic dynamic programming (1st ed.). New York: Wiley. MATHCrossRefGoogle Scholar
  50. Raffelt, H., Steffen, B., Berg, T., & Margaria, T. (2009). Learnlib: a framework for extrapolating behavioral models. International Journal on Software Tools for Technology Transfer, 11, 393–407. CrossRefGoogle Scholar
  51. Settles, B. (2010). Active learning literature survey (Technical report). University of Wisconsin-Madison. Google Scholar
  52. Shafique, M., & Labiche, Y. (2010). A systematic review of model based testing tool support (Technical Report SCE-10-04). Department of Systems and Computer Engineering, Carleton University, Ottawa, Canada. Google Scholar
  53. Sudkamp, T. A. (2006). Languages and machines: an introduction to the theory of computer science (3rd ed.). Reading: Addison-Wesley. Google Scholar
  54. Tretmans, J. (2008). Model based testing with labelled transition systems. In Lecture notes in computer science: Vol. 4949. Formal methods and testing (pp. 1–38). Berlin: Springer. CrossRefGoogle Scholar
  55. Tretmans, J., & Brinksma, E. (2003). Torx: automated model-based testing. In Proceedings of the 1st European conference on model-driven software engineering (pp. 31–43). Google Scholar
  56. Utting, M., & Legeard, B. (2007). Practical model-based testing: a tools approach. San Mateo: Morgan-Kaufmann. Google Scholar
  57. van der Aalst, W. M. P. (2011). Process mining—discovery, conformance and enhancement of business processes. Berlin: Springer. MATHGoogle Scholar
  58. Verwer, S. (2010). Efficient identification of timed automata: theory and practice. PhD thesis, Delft University of Technology. Google Scholar
  59. Verwer, S., de Weerdt, M., & Witteveen, C. (2011). Efficiently identifying deterministic real-time automata from labeled data. Machine Learning, 1–39. Google Scholar
  60. Walkinshaw, N., Bogdanov, K., Holcombe, M., & Salahuddin, S. (2007). Reverse engineering state machines by interactive grammar inference. In Proceedings of the 14th working conference on reverse engineering (pp. 209–218). New York: IEEE. Google Scholar
  61. Walkinshaw, N., Bogdanov, K., Damas, C., Lambeau, B., & Dupont, P. (2010a). A framework for the competitive evaluation of model inference techniques. In Proceedings of the 1st international workshop on model inference in testing (pp. 1–9). New York: ACM. Google Scholar
  62. Walkinshaw, N., Bogdanov, K., Derrick, J., & Paris, J. (2010b). Increasing functional coverage by inductive testing: a case study. In Lecture notes in computer science: Vol. 6435. Proceedings of the 22nd IFIP WG 6.1 international conference on testing software and systems (ICTSS) (pp. 126–141). Berlin: Springer. Google Scholar
  63. Weyuker, E. J. (1983). Assessing test data adequacy through program inference. ACM Transactions on Programming Languages and Systems, 5(4), 641–655. MATHCrossRefGoogle Scholar
  64. Yokomori, T. (1993). Learning non-deterministic finite automata from queries and counterexamples. In Machine Intelligence (pp. 196–189). University Press. Google Scholar

Copyright information

© The Author(s) 2013

Authors and Affiliations

  • Fides Aarts
    • 1
  • Harco Kuppens
    • 1
  • Jan Tretmans
    • 1
    • 2
  • Frits Vaandrager
    • 1
  • Sicco Verwer
    • 1
  1. 1.Institute for Computing and Information SciencesRadboud University NijmegenNijmegenThe Netherlands
  2. 2.TNO—Embedded Systems InnovationEindhovenThe Netherlands

Personalised recommendations