STAMINA: a competition to encourage the development and assessment of software model inference techniques
 Neil Walkinshaw,
 Bernard Lambeau,
 Christophe Damas,
 Kirill Bogdanov,
 Pierre Dupont
 … show all 5 hide
Abstract
Models play a crucial role in the development and maintenance of software systems, but are often neglected during the development process due to the considerable manual effort required to produce them. In response to this problem, numerous techniques have been developed that seek to automate the model generation task with the aid of increasingly accurate algorithms from the domain of Machine Learning. From an empirical perspective, these are extremely challenging to compare; there are many factors that are difficult to control (e.g. the richness of the input and the complexity of subject systems), and numerous practical issues that are just as troublesome (e.g. tool availability). This paper describes the StaMinA (State Machine Inference Approaches) competiton, that was designed to address these problems. The competition attracted numerous submissions, many of which were improved or adapted versions of techniques that had not been subjected to extensive empirical evaluations, and had not been evaluated with respect to their ability to infer models of software systems. This paper shows how many of these techniques substantially improve on the state of the art, providing insights into some of the factors that could underpin the success of the best techniques. In a more general sense it demonstrates the potential for competitions to act as a useful basis for empirical software engineering by (a) spurring the development of new techniques and (b) facilitating their comparative evaluation to an extent that would usually be prohibitively challenging without the active participation of the developers.
 Adriaans P, Jacobs C (2006) Using MDL for grammar induction. In: International colloquium on grammatical inference: algorithms and applications (ICGI). Lecture notes in artificial intelligence, vol 4201, pp 293–306
 Ammons G, Bodík R, Larus J (2002) Mining Specifications. In: Principles of programming languages (POPL). Portland, Oregon, pp 4–16
 Angluin D (1978) On the complexity of minimum inference of regular sets. Inform Control 39:337–350 CrossRef
 Biermann A, Feldman J (1972) On the synthesis of finitestate machines from samples of their behavior. IEEE Trans Comput 21:592–597 CrossRef
 Biermann A, Krishnaswamy R (1976) Constructing programs from example computations. IEEE Trans Softw Eng SE2:141–153 CrossRef
 Combe D, de la Higuera C, Janodet J (2010) Zulu: an interactive learning competition. In: Finitestate methods and natural language processing. Lecture notes in computer science, vol 6062. Springer, pp 139–146
 Cook J, Wolf A (1998) Discovering models of software processes from eventbased data. ACM Trans Softw Eng Methodol 7(3):215–249 CrossRef
 Coste F, Nicolas J (1997) Regular inference as a graph coloring problem. In: Workshop on grammatical inference, automata induction, and language acquisition (ICML’97), pp 9–7
 Damas C, Lambeau B, Dupont P, van Lamsweerde A (2005) Generating annotated behavior models from enduser scenarios. IEEE Trans Softw Eng 31(12):1056–1073 CrossRef
 Dupont P, Lambeau B, Damas C, van Lamsweerde A (2008) The QSM algorithm and its application to software behavior model induction. Appl Artif Intell 22:77–115 CrossRef
 Dupont P, Miclet L, Vidal E (1994) What is the search space of the regular inference? In: Proceedings of the international colloqium on grammatical inference and applications (ICGI’94). LNAI, vol 862. Springer Verlag, pp 25–37
 García P, de Parga M, López D, Ruiz J (2010) Learning automata teams. In: International colloquium on grammatical inference: algorithms and applications (ICGI). Springer, pp 52–65
 Gold M (1978) Complexity of automaton identification from given data. Inform Control 37:302–320 CrossRef
 Heule M, Verwer S (2010) Exact DFA identification using SAT solvers. In: International colloquium on grammatical inference: algorithms and applications (ICGI), vol 6339. Springer, pp 66–79
 Heule M, Verwer S (2012) Software model synthesis by indentifying DFAs using satisfiability solvers. Empir Software Eng (submitted)
 Hopcroft J, Motwani R, Ullman J (2007) Introduction to automata theory, languages, and computation, 3rd edn. AddisonWesley
 Keller RM (1976) Formal verification of parallel programs. Commun ACM 19:371–384 CrossRef
 Kleinberg J (1999) Authoritative sources in a hyperlinked environment. J ACM 46(5):604–632 CrossRef
 Lambeau B, Damas C, Dupont P (2008) Statemerging DFA induction algorithms with mandatory merge constraints. In: International colloquium on grammatical inference: algorithms and applications (ICGI). Springer, pp 139–153
 van Lamsweerde A (2009) Requirements engineering: from system goals to UML models to software specifications. Wiley
 Lang K (1992) Random DFA’s can be approximately learned from sparse uniform examples. In: Proceedings of the international conference on learning theory (COLT’92), pp 45–52
 Lang K, Pearlmutter B, Price R (1998) Results of the abbadingo One DFA learning competition and a new evidencedriven state merging algorithm. In: International colloquium on grammatical inference and applications(ICGI). LNAI, vol 1433, pp 1–12
 Lee D, Yannakakis M (1996) Principles and methods of testing finite state machines—a survey. In: Proceedings of the IEEE, vol 84, pp 1090–1126
 Leskovec J, Kleinberg J, Faloutsos C (2007) Graph evolution: densification and shrinking diameters. ACM Trans Knowl Discov Data (or TKDD) 1(1)
 Lo D, Cheng H, Han J, Khoo S, Sun C (2009) Classification of software behaviors for failure detection: a discriminative pattern mining approach. In: International conference on knowledge discovery and data mining (KDD2009). ACM, pp 557–566
 Lo D, Khoo S (2006) SMArTIC: towards building an accurate, robust and scalable specification miner. In: Foundations of software engineering (FSE), pp 265–275
 Lorenzoli D, Mariani L, Pezzè M (2008) Automatic generation of software behavioral models. In: ICSE ’08: Proceedings of the 30th international conference on software engineering, pp 501–510
 Magee J, Kramer J (1999) Concurrency: state models and java programs. Wiley
 Mulder W, Jacobs C, van Someren M (2009) Collaborative DFA learning applied to grid administration. In: Benelearn, annual BelgianDutch conference on machine learning. Tilburg, the Netherlands, pp 38–46
 van Nieuwpoort R, Wrzesińska G, Jacobs C, Bal H (2010) Satin: a highlevel and efficient grid programming model. ACM Trans Progr Lang Syst 32(3):1–39 CrossRef
 Oncina J, Garcia P (1992) Inferring regular languages in polynomial update time. In: Pattern Recognition and Image Analysis, vol 1. World Scientific, Singapore, pp 49–61
 Raffelt H, Steffen B, Berg T, Margaria T (2009) Learnlib: a framework for extrapolating behavioral models. STTT 11(5):393–407 CrossRef
 Reiss S, Renieris M (2001) Encoding program executions. In: International conference on software engineering(ICSE), pp 221–230
 Rissanen J (1983) A universal prior for integers and estimation by minimum description length. Ann Stat 11(2):416–431 CrossRef
 Rousseeuw P, Ruts I, Tukey J (1999) The bagplot: a bivariate boxplot. Am Stat 53(4)
 Shahbaz M, Groz R (2009) Inferring mealy machines. In: FM 2009: formal methods, second world congress, Proceedings. Lecture notes in computer science, vol 5850. Springer, Eindhoven, The Netherlands, pp 207–222, 2–6 November 2009
 Sim S, Easterbrook S, Holt R (2003) Using benchmarking to advance research: a challenge to software engineering. In: International conference on software engineering (ICSE), pp 74–83
 Tukey J (1975) Mathematics and the picturing of data. In: Proceedings of the international congress of mathematicians, vol 2, pp 523–531
 Walkinshaw N, Bogdanov K (2008) Inferring finitestate models with temporal constraints. In: International conference on automated software engineering (ASE)
 Walkinshaw N, Bogdanov K, Holcombe M, Salahuddin S (2007) Reverse engineering state machines by interactive grammar inference. In: International working conference on reverse engineering (WCRE)
 Walkinshaw N, Bogdanov K, Holcombe M, Salahuddin S (2008) Improving dynamic software analysis by applying grammar inference principles. J Softw Maint Evol: Res Pract 20(4)
 Walkinshaw N, Bogdanov K, Johnson K (2008) Evaluation and comparison of inferred regular grammars. In: International colloquium on grammatical inference: algorithms and applications (ICGI). Lecture notes in computer science, vol 5278. Springer, pp 252–265
 Title
 STAMINA: a competition to encourage the development and assessment of software model inference techniques
 Open Access
 Available under Open Access This content is freely available online to anyone, anywhere at any time.
 Journal

Empirical Software Engineering
Volume 18, Issue 4 , pp 791824
 Cover Date
 20130801
 DOI
 10.1007/s1066401292103
 Print ISSN
 13823256
 Online ISSN
 15737616
 Publisher
 Springer US
 Additional Links
 Topics
 Keywords

 Model inference
 Software specification
 Competition
 State machines
 Industry Sectors
 Authors

 Neil Walkinshaw ^{(1)}
 Bernard Lambeau ^{(2)}
 Christophe Damas ^{(2)}
 Kirill Bogdanov ^{(3)}
 Pierre Dupont ^{(2)}
 Author Affiliations

 1. Department of Computer Science, The University of Leicester, Leicester, UK
 2. ICTEAM Institute, Université catholique de Louvain (UCL), LouvainlaNeuve, Belgium
 3. Department of Computer Science, The University of Sheffield, Sheffield, UK