Skip to main content
SpringerLink
Log in

Learning by sampling: learning behavioral family models from software product lines

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Family-based behavioral analysis operates on a single specification artifact, referred to as family model, annotated with feature constraints to express behavioral variability in terms of conditional states and transitions. Family-based behavioral modeling paves the way for efficient model-based analysis of software product lines. Family-based behavioral model learning incorporates feature model analysis and model learning principles to efficiently unify product models into a family model and integrate the behavior of various products into a behavioral family model. Albeit reasonably effective, the exhaustive analysis of product lines is often infeasible due to the potentially exponential number of valid configurations. In this paper, we first present a family-based behavioral model learning techniques, called FFSMDiff. Subsequently, we report on our experience on learning family models by employing product sampling. Using 105 products of six product lines expressed in terms of Mealy machines, we evaluate the precision of family models learned from products selected from different settings of the T-wise product sampling criterion. We show that product sampling can lead to models as precise as those learned by exhaustive analysis and hence, reduce the costs for family model learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  • Aarts F, Heidarian F, Kuppens H, Olsen P, Vaandrager F (2012) Automata learning through counterexample guided abstraction refinement. In: Giannakopoulou D, Méry D (eds) FM 2012: formal methods. https://doi.org/10.1007/978-3-642-32759-9_4. Springer, Berlin, pp 10–27

  • Aichernig BK, Mostowski W, Mousavi MR, Tappler M, Taromirad M (2018) Model learning and model-based testing. In: Bennaceur A, Hähnle R, Meinke K (eds) Machine learning for dynamic software analysis: potentials and limits: International Dagstuhl Seminar 16172, Dagstuhl Castle, Germany, April 24-27, 2016, Revised Papers. https://doi.org/10.1007/978-3-319-96562-8_3. Springer International Publishing, Cham, pp 74–100

  • Al-Hajjaji M, Krieter S, Thüm T, Lochau M, Saake G (2016) Incling: efficient product-line testing using incremental pairwise sampling. In: Proceedings of the 2016 ACM SIGPLAN international conference on generative programming: concepts and experiences, GPCE 2016. https://doi.org/10.1145/2993236.2993253. Association for Computing Machinery, New York, pp 144–155

  • Al-Hajjaji M, Lity S, Lachmann R, Thüm T, Schaefer I, Saake G (2017) Delta-oriented product prioritization for similarity-based product-line testing. In: Proceedings of the 2nd international workshop on variability and complexity in software design, VACE ’17. https://doi.org/10.1109/VACE.2017.8. IEEE Press, pp 34–40

  • Al-Khiaty MAR, Ahmed M (2017) Matching UML class diagrams using a hybridized greedy-genetic algorithm. In: 2017 12th international scientific and technical conference on computer sciences and information technologies (CSIT). https://doi.org/10.1109/STC-CSIT.2017.8098759. IEEE, pp 161–166

  • Al-Msie’deen R, Huchard M, Seriai A, Urtado C, Vauttier S (2014) Reverse engineering feature models from software configurations using formal concept analysis. In: Bertet K, Rudolph S (eds) Proceedings of the eleventh international conference on concept lattices and their applications, Košice, Slovakia, October 7-10, 2014, CEUR Workshop Proceedings. http://ceur-ws.org/Vol-1252/cla2014_submission_13.pdf, vol 1252. CEUR-WS.org, pp 95–106

  • Angluin D (1987) Learning regular sets from queries and counterexamples. Information and Computation 75(2):87–106. https://doi.org/10.1016/0890-5401(87)90052-6

    Article  MathSciNet  MATH  Google Scholar 

  • Apache (2016) Commons math: the apache commons mathematics library. http://commons.apache.org/proper/commons-math/. [Online; Accessed 28 Mar 2019]

  • Apel S, Kolesnikov S, Siegmund N, Kästner C, Garvin B (2013) Exploring feature interactions in the wild: the new feature-interaction challenge. In: Proceedings of the 5th international workshop on feature-oriented software development. https://doi.org/10.1145/2528265.2528267. ACM, New York, pp 1–8

  • Arcuri A, Briand L (2011) A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proceedings of the 33rd international conference on software engineering, ICSE ’11. https://doi.org/10.1145/1985793.1985795. Association for Computing Machinery, New York, pp 1–10

  • Asirelli P, ter Beek MH, Gnesi S, Fantechi A (2011) Formal description of variability in product families. In: 2011 15th international software product line conference. https://doi.org/10.1109/SPLC.2011.34. IEEE, pp 130–139

  • Asirelli P, ter Beek MH, Fantechi A, Gnesi S, Steffen B (2012) A compositional framework to derive product line behavioural descriptions. In: Margaria T (ed) Leveraging applications of formal methods, verification and validation. Technologies for Mastering Change. https://doi.org/10.1007/978-3-642-34026-0_12. Springer, Berlin, pp 146–161

  • Atlee JM, Beidu S, Fahrenberg U, Legay A (2015) Merging features in featured transition systems. In: Proceedings of the 12th workshop on model-driven engineering, verification and validation co-located with ACM/IEEE 18th international conference on model driven engineering languages and systems. https://hal.inria.fr/hal-01237661, vol 1514. CEUR Workshop Proceedings, Ottawa, Canada, pp 38–43

  • Baier C, Katoen JP (2008) Principles of model checking (representation and mind series). The MIT Press

  • Bauer O, Geske M, Isberner M (2014) Analyzing program behavior through active automata learning. Int J Softw Tools Technol Transfer 16(5):531–542. https://doi.org/10.1007/s10009-014-0333-2

    Article  Google Scholar 

  • Benavides D, Segura S, Ruiz-Cortés A (2010) Automated analysis of feature models 20 years later: a literature review. Info Sys 35 (6):615–636. https://doi.org/10.1016/j.is.2010.01.001. http://www.sciencedirect.com/science/article/pii/S0306437910000025

    Article  Google Scholar 

  • Benduhn F, Thüm T, Lochau M, Leich T, Saake G (2015) A survey on modeling techniques for formal behavioral verification of software product lines. In: Proceedings of the ninth international workshop on variability modelling of software-intensive systems, VaMoS ’15. https://doi.org/10.1145/2701319.2701332. Association for Computing Machinery, New York, pp 80–87

  • Beohar H, Mousavi MR (2016) Input–output conformance testing for software product lines. J Logic Algeb Methods Prog 85(6):1131–1153. https://doi.org/10.1016/j.jlamp.2016.09.007. NWPT 2013

    MathSciNet  MATH  Google Scholar 

  • Beohar H, Varshosaz M, Mousavi MR (2016) Basic behavioral models for software product lines: expressiveness and testing pre-orders. Science of Computer Programming 123:42–60. https://doi.org/10.1016/j.scico.2015.06.005

    Article  Google Scholar 

  • Berger T, She S, Lotufo R, Wasowski A, Czarnecki K (2013) A study of variability models and languages in the systems software domain. IEEE Trans Softw Eng 39(12):1611–1640. https://doi.org/10.1109/TSE.2013.34

    Article  Google Scholar 

  • Bertolino A, Daoudagh S, El Kateb D, Henard C, Le Traon Y, Lonetti F, Marchetti E, Mouelhi T, Papadakis M (2015) Similarity testing for access control. Information and Software Technology 58:355–372. https://doi.org/10.1016/j.infsof.2014.07.003

    Article  Google Scholar 

  • Beuche D, Schulze M, Duvigneau M (2016) When 150 % is too much: supporting product centric viewpoints in an industrial product line. In: Proceedings of the 20th international systems and software product line conference, SPLC ’16. https://doi.org/10.1145/2934466.2934493. Association for Computing Machinery, New York, pp 262–269

  • Brabrand C, Ribeiro M, Tolêdo T, Borba P (2012) Intraprocedural dataflow analysis for software product lines. In: Proceedings of the 11th annual international conference on aspect-oriented software development, AOSD ’12. https://doi.org/10.1145/2162049.2162052. ACM, New York, pp 13–24

  • Broy M, Jonsson B, Katoen JP, Leucker M, Pretschner A (2005) Part i. testing of finite state machines. In: Broy M, Jonsson B, Katoen JP, Leucker M, Pretschner A (eds) Model-based testing of reactive systems: advanced lectures. https://doi.org/10.1007/11498490_1. Springer, Berlin, pp 1–3

  • Cartaxo EG, Machado PDL, Neto FGO (2011) On the use of a similarity function for test case selection in the context of model-based testing. Softw Test Verif Reliab 21(2):75–100. https://doi.org/10.1002/stvr.413

    Article  Google Scholar 

  • Chapman M, Chockler H, Kesseli P, Kroening D, Strichman O, Tautschnig M (2015) Learning the language of error. In: Finkbeiner B, Pu G, Zhang L (eds) Automated technology for verification and analysis. https://doi.org/10.1007/978-3-319-24953-7_9. Springer International Publishing, Cham, pp 114–130

  • Chow TS (1978) Testing software design modeled by finite-state machines. IEEE Trans Softw Eng 4(3):178–187. https://doi.org/10.1109/TSE.1978.231496

    Article  MATH  Google Scholar 

  • Chrszon P, Dubslaff C, Klüppelholz S, Baier C (2018) ProFeat: feature-oriented engineering for family-based probabilistic model checking. Formal Aspects Comput 30(1):45–75. https://doi.org/10.1007/s00165-017-0432-4

    Article  MathSciNet  Google Scholar 

  • Chvatal V (1979) A greedy heuristic for the set-covering problem. Math Oper Res 4(3):233–235. https://doi.org/10.1287/moor.4.3.233

    Article  MathSciNet  MATH  Google Scholar 

  • Classen A (2010) Modelling with fts: a collection of illustrative examples. https://researchportal.unamur.be/en/publications/modelling-with-fts-a-collection-of-illustrative-examples

  • Classen A, Heymans P, Schobbens PY, Legay A, Raskin JF (2010) Model checking lots of systems: efficient verification of temporal properties in software product lines. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering - volume 1, ICSE ’10. https://doi.org/10.1145/1806799.1806850. Association for Computing Machinery, New York, pp 335–344

  • Classen A, Cordy M, Schobbens PY, Heymans P, Legay A, Raskin JF (2013) Featured transition systems: foundations for verifying variability-intensive systems and their application to ltl model checking. IEEE Trans Softw Eng 39 (8):1069–1089. https://doi.org/10.1109/TSE.2012.86

    Article  Google Scholar 

  • Clements PC, Northrop L (2001) Software product lines: practices and patterns. SEI series in software engineering. Addison-Wesley Longman Publishing Co., Inc., Boston

    Google Scholar 

  • Cook JE, Wolf AL (1998) Discovering models of software processes from event-based data. ACM Trans Softw Eng Methodol 7(3):215–249. https://doi.org/10.1145/287000.287001

    Article  Google Scholar 

  • Czarnecki K, Bednasch T, Unger P, Eisenecker U (2002) Generative programming for embedded software: an industrial experience report. In: Batory D, Consel C, Taha W (eds) Generative programming and component engineering. https://doi.org/10.1007/3-540-45821-2_10. Series Title: Lecture Notes in Computer Science, vol 2487. Springer Berlin Heidelberg, pp 156–172

  • Czarnecki K, Helsen S, Eisenecker U (2005) Formalizing cardinality-based feature models and their specialization. 10(1):7–29. https://doi.org/10.1002/spip.213

  • Damasceno CDN, Masiero PC, Simao A (2016) Evaluating test characteristics and effectiveness of fsm-based testing methods on rbac systems. In: Proceedings of the 30th Brazilian symposium on software engineering, SBES ’16. https://doi.org/10.1145/2973839.2973849. ACM, New York, pp 83–92

  • Damasceno CDN, Masiero PC, Simao A (2018) Similarity testing for role-based access control systems. J Softw Eng Res Dev 6(1):1. https://doi.org/10.1186/s40411-017-0045-x

    Article  Google Scholar 

  • Damasceno CDN, Mousavi MR, da Silva Simao A (2019a) Learning to reuse: adaptive model learning for evolving systems. In: Ahrendt W, Tapia Tarifa SL (eds) Integrated formal methods: 15th international conference, IFM 2019, Bergen, Norway, December 2-6, 2019, Proceedings. https://doi.org/10.1007/978-3-030-34968-4_8. Springer International Publishing, Cham, pp 138–156

  • Damasceno CDN, Mousavi MR, Simao A (2019b) Learning from difference: an automated approach for learning family models from software product lines. In: Proceedings of the 23rd international systems and software product line conference - volume A, SPLC ’19. https://doi.org/10.1145/3336294.3336307. ACM, New York, pp 52–63

  • De Ruiter J, Poll E (2015) Protocol state fuzzing of tls implementations. In: Proceedings of the 24th USENIX conference on security symposium, SEC’15. https://doi.org/10.5555/2831143.2831156. USENIX Association, Berkeley, pp 193–206

  • Devroey X, Perrouin G (2014) Vibes: variability intensive system behavioural testing. https://projects.info.unamur.be/vibes/

  • Devroey X, Perrouin G, Legay A, Schobbens PY, Heymans P (2015) Covering spl behaviour with sampled configurations: an initial assessment. In: Proceedings of the ninth international workshop on variability modelling of software-intensive systems, VaMoS ’15. https://doi.org/10.1145/2701319.2701325. ACM, New York, pp 59:59–59:66

  • Devroey X, Perrouin G, Legay A, Schobbens PY, Heymans P (2016) Search-based similarity-driven behavioural SPL testing. In: Proceedings of the tenth international workshop on variability modelling of software-intensive systems - VaMoS ’16. https://doi.org/10.1145/2866614.2866627. ACM Press, pp 89–96

  • Deza MM, Deza E (2013) Distances and similarities in data analysis. In: Encyclopedia of distances. https://doi.org/10.1007/978-3-642-30958-8_17. Springer, Berlin, pp 291–305

  • Don Batory (2005) Feature models, grammars, and propositional formulas. In: Obbink H, Pohl K (eds) Software product lines. https://doi.org/10.1007/11554844_3. Series Title: Lecture Notes in Computer Science, vol 3714. Springer, Berlin, pp 7–20

  • Ensan F, Bagheri E, Gašević D (2012) Evolutionary search-based test generation for software product line feature models. In: Ralyté J, Franch X, Brinkkemper S, Wrycza S (eds) Advanced information systems engineering. https://doi.org/10.1007/978-3-642-31095-9_40. Springer, Berlin, pp 613–628

  • FeatureIDE (2004) FeatureIDE: an extensible framework for feature-oriented software development. https://featureide.github.io/. [Online; Accessed 21 May 2020]

  • Fenske W, Thüm T, Saake G (2014) A taxonomy of software product line reengineering. In: Proceedings of the eighth international workshop on variability modelling of software-intensive systems, VaMoS ’14. https://doi.org/10.1145/2556624.2556643. Association for Computing Machinery, New York

  • Fischbein D, Uchitel S, Braberman V (2006) A foundation for behavioural conformance in software product line architectures. In: Proceedings of the ISSTA 2006 workshop on role of software architecture for testing and analysis, ROSATEA ’06. https://doi.org/10.1145/1147249.1147254. Association for Computing Machinery, pp 39–48

  • Fiterău-Broştean P, Howar F (2017) Learning-based testing the sliding window behavior of tcp implementations. In: Petrucci L, Seceleanu C, Cavalcanti A (eds) Critical systems: formal methods and automated verification: joint 22nd international workshop on formal methods for industrial critical systems and 17th international workshop on automated verification of critical systems, FMICS-AVoCS 2017, Turin, Italy, September 18-20, 2017, Proceedings. https://doi.org/10.1007/978-3-319-67113-0_12. Springer International Publishing, Cham, pp 185–200

  • Fragal VH (2017) Automatic generation of configurable test-suites for software product lines. Ph.D. thesis, Universidade de São Paulo. [Online] http://www.teses.usp.br/teses/disponiveis/55/55134/tde-10012019-085746/

  • Fragal VH, Simao A, Mousavi MR (2019) Hierarchical featured state machines. Sci Comput Program 171:67–88. https://doi.org/10.1016/j.scico.2018.10.001

    Article  Google Scholar 

  • Fragal VH, Simao A, Mousavi MR, Turker UC (2018) Extending HSI test generation method for software product lines. Comput J 62(1):109–129. https://doi.org/10.1093/comjnl/bxy046

    Google Scholar 

  • Gill A (1962) Introduction to the theory of finite state machines. McGraw-Hill, New York

    MATH  Google Scholar 

  • Hafemann Fragal V, Simao A, Mousavi MR (2017) Validated test models for software product lines: featured finite state machines. In: Kouchnarenko O, Khosravi R (eds) Formal aspects of component software: 13th international conference, FACS 2016, Besançon, France, October 19-21, 2016, Revised Selected Papers. https://doi.org/10.1007/978-3-319-57666-4_13. Springer International Publishing, Cham, pp 210–227

  • Harel D (1987) Statecharts: a visual formalism for complex systems. Sci Comput Program 8(3):231–274. https://doi.org/10.1016/0167-6423(87)90035-9

    Article  MathSciNet  MATH  Google Scholar 

  • Haslinger EN, Lopez-Herrejon RE, Egyed A (2011) Reverse engineering feature models from programs’ feature sets. In: 2011 18th working conference on reverse engineering. https://doi.org/10.1109/WCRE.2011.45. IEEE, pp 308–312

  • Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, Le Traon Y (2014) Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650–670

    Article  Google Scholar 

  • Hess MR, Kromrey JD (2004) Robust confidence intervals for effect sizes: a comparative study of cohen’s d and cliff’s delta under non-normality and heterogeneous variances. In: Annual meeting of the american educational research association

  • Holthusen S, Wille D, Legat C, Beddig S, Schaefer I, Vogel-Heuser B (2014) Family model mining for function block diagrams in automation software. In: Proceedings of the 18th international software product line conference on companion volume for workshops, demonstrations and tools - SPLC ’14. https://doi.org/10.1145/2647908.2655965. ACM Press, pp 36–43

  • Huistra D, Meijer J, Pol J (2018) Adaptive learning for learn-based regression testing. In: Howar F, Barnat J (eds) Formal methods for industrial critical systems, lecture notes in computer science. https://doi.org/10.1007/978-3-030-00244-2_11. Springer Publishers, Switzerland, pp 162–177

  • Hungar H, Niese O, Steffen B (2003) Domain-specific optimization in automata learning. In: Hunt WA, Somenzi F (eds) Computer aided verification: 15th international conference, CAV 2003, Boulder, CO, USA, July 8-12, 2003. Proceedings. https://doi.org/10.1007/978-3-540-45069-6_31. Springer, Berlin, pp 315–327

  • Irfan MN, Oriat C, Groz R (2013) Chapter 3 - model inference and testing. In: Memon A (ed) Advances in computers. https://doi.org/10.1016/B978-0-12-408094-2.00003-5, vol 89. Elsevier, pp 89–139

  • Isberner M, Howar F, Steffen B (2015) The open-source learnlib. In: Kroening D, Păsăreanu CS (eds) Computer aided verification: 27th international conference, CAV 2015, San Francisco, CA, USA, July 18-24, 2015, Proceedings, Part I. https://doi.org/10.1007/978-3-319-21690-4_32. Springer International Publishing, Cham, pp 487–495

  • Johansen MF, Haugen Ø, Fleurey F (2011) Properties of realistic feature models make combinatorial testing of product lines feasible. In: Whittle J, Clark T, Kühne T (eds) Model driven engineering languages and systems. https://doi.org/10.1007/978-3-642-24485-8_47. Springer, Berlin, pp 638–652

  • Kampenes VB, Dyba T, Hannay JE, Sjoberg DI (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49 (11):1073–1086. https://doi.org/10.1016/j.infsof.2007.02.015

    Article  Google Scholar 

  • Kang K, Cohen S, Hess J, Novak W, Peterson A (1990) Feature-oriented domain analysis (foda) feasibility study. Tech. Rep. CMU/SEI-90-TR-021, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA. https://resources.sei.cmu.edu/library/asset-view.cfm?assetid=11231

  • Keller RM (1976) Formal verification of parallel programs. Commun ACM 19(7):371–384. https://doi.org/10.1145/360248.360251

    Article  MathSciNet  MATH  Google Scholar 

  • Kuhn DR, Kacker RN, Lei Y (2013) Introduction to combinatorial testing, 1st. Chapman & Hall/CRC, Boca Raton

    MATH  Google Scholar 

  • Kuhn DR, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421. https://doi.org/10.1109/TSE.2004.24

    Article  Google Scholar 

  • Kunze S, Mostowski W, Mousavi MR, Varshosaz M (2016) Generation of failure models through automata learning. In: 2016 workshop on automotive systems/software architectures (WASA). https://doi.org/10.1109/WASA.2016.7. IEEE Computer Society, Los Alamitos, pp 22–25

  • Laguna MA, Crespo Y (2013) A systematic mapping study on software product line evolution: from legacy system reengineering to product line refactoring. Sci Comput Program 78(8):1010–1034. https://doi.org/10.1016/j.scico.2012.05.003

    Article  Google Scholar 

  • Lang KJ, Pearlmutter BA, Price RA (1998) Results of the abbadingo one dfa learning competition and a new evidence-driven state merging algorithm. In: Proceedings of the 4th international colloquium on grammatical inference, ICGI ’98. https://doi.org/10.1007/BFb0054059. Springer, Berlin, pp 1–12

  • Larsen KG, Thomsen B (1988) A modal process logic. In: Proceedings of the third annual symposium on logic in computer science. https://doi.org/10.1109/LICS.1988.5119, pp 203–210

  • Larsen KG, Nyman U, Wa̧sowski A (2007) Modal i/o automata for interface and product line theories. In: De Nicola R (ed) Programming languages and systems, lecture notes in computer science. https://doi.org/10.1007/978-3-540-71316-6_6. Springer, pp 64–79

  • Le Berre D, Parrain A (2010) The SAT4J library, release 2.2, system description. J Satis Boolean Model Comput 7:59–64. http://satassociation.org/jsat/index.php/jsat/article/view/82

    Google Scholar 

  • Linden FJVD, Schmid K, Rommes E (2007) Software product lines in action: the best industrial practice in product line engineering. Springer-Verlag New York, Inc., Secaucus

    Book  Google Scholar 

  • Lopez-Herrejon RE, Ferrer J, Chicano F, Egyed A, Alba E (2014) Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: 2014 IEEE congress on evolutionary computation (CEC). https://doi.org/10.1109/CEC.2014.6900473, pp 387–396

  • Luthmann L, Stephan A, Bürdek J, Lochau M (2017) Modeling and testing product lines with unbounded parametric real-time constraints. In: Proceedings of the 21st international systems and software product line conference - volume A on - SPLC ’17. https://doi.org/10.1145/3106195.3106204. ACM Press, pp 104–113

  • Luthmann L, Gerecht T, Stephan A, Bürdek J, Lochau M (2019) Minimum/maximum delay testing of product lines with unbounded parametric real-time constraints. 149: 535–553. https://doi.org/10.1016/j.jss.2018.12.028. http://www.sciencedirect.com/science/article/pii/S0164121218302851

  • Mariani L, Pezzè M, Zuddas D (2015) Chapter four - recent advances in automatic black-box testing. In: Memon A (ed) Advances in computers. https://doi.org/10.1016/bs.adcom.2015.04.002, vol 99. Elsevier, pp 157–193

  • Marques M, Simmonds J, Rossel PO, Bastarrica MC (2019) Software product line evolution: a systematic literature review. Inf Softw Technol 105:190–208. https://doi.org/10.1016/j.infsof.2018.08.014

    Article  Google Scholar 

  • Mendez D, Graziotin D, Wagner S, Seibold H (2020) Open science in software engineering. In: Felderer M, Travassos GH (eds) Contemporary empirical methods in software engineering. https://doi.org/10.1007/978-3-030-32489-6_17. Springer International Publishing, pp 477–501

  • Nascimento Damasceno CD (2020) damascenodiego/learningFFSM: Learning By Sampling @ EMSE. Zenodo. https://doi.org/10.5281/zenodo.4394350

  • Neider D, Smetsers R, Vaandrager F, Kuppens H (2019) Benchmarks for automata learning and conformance testing. In: Margaria T, Graf S, Larsen KG (eds) Models, mindsets, meta: the what, the how, and the why not? Essays dedicated to Bernhard Steffen on the occasion of his 60th birthday. https://doi.org/10.1007/978-3-030-22348-9_23. Springer International Publishing, pp 390–416

  • Nejati S, Sabetzadeh M, Chechik M, Easterbrook S, Zave P (2012) Matching and merging of variant feature specifications. IEEE Transactions on Software Engineering 38(6):1355–1375. https://doi.org/10.1109/TSE.2011.112

    Article  Google Scholar 

  • Neubauer J, Steffen B, Bauer O, Windmuller S, Merten M, Margaria T, Howar F (2012) Automated continuous quality assurance. In: 2012 first international workshop on formal methods in software engineering: rigorous and agile approaches (FormSERA). https://doi.org/10.1109/FormSERA.2012.6229787. IEEE, pp 37–43

  • Oster S (2012) Feature model-based software product line testing. Ph.D. thesis, Technische Universität. [Online] http://tuprints.ulb.tu-darmstadt.de/2881/

  • Peled D, Vardi MY, Yannakakis M (1999) Black box checking. In: Wu J, Chanson ST, Gao Q (eds) Formal methods for protocol engineering and distributed systems: FORTE XII / PSTV XIX’99 IFIP TC6 WG6.1 Joint international conference on formal description techniques for distributed systems and communication protocols (FORTE XII) and protocol specification, testing and verification (PSTV XIX) October 5–8, 1999, Beijing, China. https://doi.org/10.1007/978-0-387-35578-8_13. Springer US, Boston, pp 225–240

  • Perrouin G, Sen S, Klein J, Baudry Bl, Traon Y (2010) Automated and scalable t-wise test case generation strategies for software product lines. In: 2010 third international conference on software testing, verification and validation. https://doi.org/10.1109/ICST.2010.43, pp 459–468

  • Petke J, Yoo S, Cohen MB, Harman M (2013) Efficiency and early fault detection with lower and higher strength combinatorial interaction testing. In: Proceedings of the 2013 9th joint meeting on foundations of software engineering, ESEC/FSE 2013. https://doi.org/10.1145/2491411.2491436. ACM, New York, pp 26–36

  • Pohl K, Böckle G, Linden FJVD (2005) Software product line engineering: foundations, principles and techniques. Springer-Verlag New York, Inc., Secaucus

    Book  MATH  Google Scholar 

  • Raffelt H, Merten M, Steffen B, Margaria T (2009) Dynamic testing via automata learning. Int J Softw Tools Technol Transfer 11(4):307. https://doi.org/10.1007/s10009-009-0120-7

    Article  Google Scholar 

  • Raffelt H, Steffen B (2006) Learnlib: a library for automata learning and experimentation. In: Baresi L, Heckel R (eds) Fundamental approaches to software engineering: 9th international conference, FASE 2006, held as part of the joint european conferences on theory and practice of software, ETAPS 2006, Vienna, Austria, March 27-28, 2006. Proceedings. https://doi.org/10.1007/11693017_28. Springer, Berlin, pp 377–380

  • RStudio (2019) RStudio: open source and enterprise-ready professional software for data science. https://www.rstudio.com/. [Online; Accessed 19 May 2019]

  • Runeson P, Engström E (2012) Chapter 7 - regression testing in software product line engineering. In: Hurson A, Memon A (eds) Advances in computers. https://doi.org/10.1016/B978-0-12-396535-6.00007-7, vol 86. Elsevier, pp 223–263

  • Ryssel U, Ploennigs J, Kabitzsch K (2011) Extraction of feature models from formal contexts. In: Proceedings of the 15th international software product line conference, volume 2, SPLC ’11. https://doi.org/10.1145/2019136.2019141. ACM, New York, pp 1–8

  • Sabouri H, Khosravi R (2013) Delta modeling and model checking of product families. In: Arbab F, Sirjani M (eds) Fundamentals of software engineering: 5th international conference, FSEN 2013, Tehran, Iran, April 24-26, 2013, Revised Selected Papers. https://doi.org/10.1007/978-3-642-40213-5_4. Springer, Berlin, pp 51–65

  • Samih H, Guen HL, Bogusch R, Acher M, Baudry B (2014) Deriving usage model variants for model-based testing: an industrial case study. In: 2014 19th international conference on engineering of complex computer systems. https://doi.org/10.1109/ICECCS.2014.19, pp 77–80

  • Schaefer I, Bettini L, Bono V, Damiani F, Tanzarella N (2010) Delta-oriented programming of software product lines. In: Bosch J, Lee J (eds) Software product lines: going beyond: 14th international conference, SPLC 2010, Jeju Island, South Korea, September 13-17, 2010. Proceedings. https://doi.org/10.1007/978-3-642-15579-6_6. Springer, Berlin, pp 77–91

  • Schaefer I, Rabiser R, Clarke D, Bettini L, Benavides D, Botterweck G, Pathak A, Trujillo S, Villela K (2012) Software diversity: state of the art and perspectives. Int J Softw Tools Technol Transfer 14(5):477–495. https://doi.org/10.1007/s10009-012-0253-y

    Article  Google Scholar 

  • Schuts M, Hooman J, Vaandrager F (2016) Refactoring of legacy software using model learning and equivalence checking: an industrial experience report. In: Ábrahám E, Huisman M (eds) Integrated formal methods: 12th international conference, IFM 2016, Reykjavik, Iceland, June 1-5, 2016, Proceedings. https://doi.org/10.1007/978-3-319-33693-0_20. Springer International Publishing, Cham, pp 311–325

  • Sery O, Fedyukovich G, Sharygina N (2015) Incremental upgrade checking. In: Chockler H, Kroening D, Mariani L, Sharygina N (eds) Validation of evolving software. https://doi.org/10.1007/978-3-319-10623-6_6. Springer International Publishing, Cham, pp 55–72

  • Shahbaz M, Groz R (2009) Inferring mealy machines. In: Cavalcanti A, Dams DR (eds) FM 2009: formal methods: second world congress, Eindhoven, The Netherlands, November 2-6, 2009. Proceedings. https://doi.org/10.1007/978-3-642-05089-3_14. Springer, Berlin, pp 207–222

  • Sharp DC (2000) Component-based product line development of avionics software. In: Donohoe P (ed) Software product lines. https://doi.org/10.1007/978-1-4615-4339-8_19. Springer US, pp 353–369

  • Sofia Cassel Falk Howar BJ (2015) Ralib: a learnlib extension for inferring efsms. DIFTS 2015. http://www.faculty.ece.vt.edu/chaowang/difts2015/papers/paper_5.pdf

  • Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45 (4):427–437. https://doi.org/10.1016/j.ipm.2009.03.002. http://www.sciencedirect.com/science/article/pii/S0306457309000259

    Article  Google Scholar 

  • SPLC (2020) Hall of fame – SPLC. https://splc.net/fame.html. [Online; Accessed 29 May 2020]

  • Steffens M, Oster S, Lochau M, Fogdal T (2012) Industrial evaluation of pairwise spl testing with moso-polite. In: Proceedings of the sixth international workshop on variability modeling of software-intensive systems, VaMoS ’12. https://doi.org/10.1145/2110147.2110154. ACM, New York, pp 55–62

  • Stevenson A, Cordy JR (2014) A survey of grammatical inference in software engineering. Sci Comput Program 96(P4):444–459. https://doi.org/10.1016/j.scico.2014.05.008

    Article  Google Scholar 

  • Svendsen A, Zhang X, Lind-Tviberg R, Fleurey F, Haugen Ø, Møller-Pedersen B, Olsen GK (2010) Developing a software product line for train control: a case study of CVL. In: Bosch J, Lee J (eds) Software product lines: going beyond. https://doi.org/10.1007/978-3-642-15579-6_8. Series Title: Lecture Notes in Computer Science, vol 6287. Springer, Berlin, pp 106–120

  • ter Beek MH, Fantechi A, Gnesi S, Mazzanti F (2016) Modelling and analysing variability in product families: model checking of modal transition systems with variability constraints. Journal of Logical and Algebraic Methods in Programming 85(2):287–315. https://doi.org/10.1016/j.jlamp.2015.11.006. http://www.sciencedirect.com/science/article/pii/S2352220815001431

    Article  MathSciNet  MATH  Google Scholar 

  • ter Beek MH, de Vink EP, Willemse TAC (2017) Family-based model checking with mcrl2. In: Huisman M, Rubin J (eds) Fundamental approaches to software engineering: 20th international conference, FASE 2017, held as part of the european joint conferences on theory and practice of software, ETAPS 2017, Uppsala, Sweden, April 22-29, 2017, Proceedings. https://doi.org/10.1007/978-3-662-54494-5_23. Springer, Berlin, pp 387–405

  • ter Beek MH, Damiani F, Gnesi S, Mazzanti F, Paolini L (2019a) On the expressiveness of modal transition systems with variability constraints. Science of Computer Programming 169:1–17. https://doi.org/10.1016/j.scico.2018.09.006. http://www.sciencedirect.com/science/article/pii/S0167642318303769

    Article  Google Scholar 

  • ter Beek MH, Damiani F, Lienhardt M, Mazzanti F, Paolini L (2019b) Static analysis of featured transition systems: [research]. In: Proceedings of the 23rd international systems and software product line conference - volume A - SPLC ’19. https://doi.org/10.1145/3336294.3336295. ACM Press, pp 1–13

  • Thüm T, Apel S, Kästner C, Schaefer I, Saake G (2014) A classification and survey of analysis strategies for software product lines. ACM Comput Surv 47(1):1–45. https://doi.org/10.1145/2580950

    Article  Google Scholar 

  • Thüm T, Kästner C, Benduhn F, Meinicke J, Saake G, Leich T (2014) Featureide: an extensible framework for feature-oriented software development. Sci Comput Program 79(Supplement C):70–85. https://doi.org/10.1016/j.scico.2012.06.002

    Article  Google Scholar 

  • Thüm T, Kastner C, Erdweg S, Siegmund N (2011) Abstract features in feature modeling. In: 2011 15th international software product line conference. https://doi.org/10.1109/SPLC.2011.53. IEEE, pp 191–200

  • Torchiano M (2017) effsize: efficient effect size computation (v. 0.7.1). CRAN package repository. https://cran.r-project.org/web/packages/effsize/effsize.pdf [Online; Accessed 20 Nov 2017]

  • Utting M, Pretschner A, Legeard B (2012) A taxonomy of model-based testing approaches. Softw Test Verif Reliab 22(5):297–312. https://doi.org/10.1002/stvr.456

    Article  Google Scholar 

  • Vaandrager F (2017) Model learning. Commun ACM 60(2):86–95. https://doi.org/10.1145/2967606

    Article  Google Scholar 

  • van der Linden F, Schmid K, Rommes E (2007a) Philips consumer electronics software for televisions. In: Software product lines in action. https://doi.org/10.1007/978-3-540-71437-8_14. Springer, Berlin, pp 219–232

  • van der Linden F, Schmid K, Rommes E (2007b) Philips medical systems. In: Software product lines in action. https://doi.org/10.1007/978-3-540-71437-8_15. Springer, Berlin, pp 233–248

  • van der Linden F, Schmid K, Rommes E (2007c) Siemens medical solutions. In: Software product lines in action. https://doi.org/10.1007/978-3-540-71437-8_16. Springer, Berlin, pp 249–263

  • von Rhein A, Grebhahn A, Apel S, Siegmund N, Beyer D, Berger T (2015) Presence-condition simplification in highly configurable systems. In: Proceedings of the 37th international conference on software engineering - volume 1, ICSE ’15. https://doi.org/10.5555/2818754.2818779. IEEE, Piscataway, pp 178–188

  • Vargha A, Delaney HD (2000) A critique and improvement of the CL common language effect size statistics of McGraw and Wong. J Educ Behav Stat 25(2):101–132. https://doi.org/10.3102/10769986025002101

    Google Scholar 

  • Varshosaz M, Al-Hajjaji M, Thüm T, Runge T, Mousavi MR, Schaefer I (2018) A classification of product sampling for software product lines. In: Proceedings of the 22Nd international systems and software product line conference - volume 1, SPLC ’18. https://doi.org/10.1145/3233027.3233035. ACM, New York, pp 1–13

  • Varshosaz M, Khosravi R (2013) Discrete time markov chain families: modeling and verification of probabilistic software product lines. In: Proceedings of the 17th international software product line conference co-located workshops on - SPLC ’13 Workshops. https://doi.org/10.1145/2499777.2500725. ACM Press, p 34

  • Vasilevskii MP (1973) Failure diagnosis of automata. Cybernetics 9(4):653–665. https://doi.org/10.1007/BF01068590

    Article  MathSciNet  Google Scholar 

  • VIBeS (2016) VIBeS - variability intensive system behavioural testing - case study: aero UC5. https://projects.info.unamur.be/vibes/case-study-aerouc5.html. [Online; Accessed 23 Mar 2020]

  • VIBeS (2016) VIBeS - variability intensive system behavioural testing - case study: card payement terminal. https://projects.info.unamur.be/vibes/case-study-cpterminal.html. [Online; Accessed 23 Mar 2020]

  • VIBeS (2016) VIBeS - variability intensive system behavioural testing - case study: minepump. https://projects.info.unamur.be/vibes/case-study-aerouc5.html. [Online; Accessed 23 Mar 2020]

  • Walkinshaw N (2013) Chapter 1 - reverse-engineering software behavior. In: Memon A (ed) Advances in computers. https://doi.org/10.1016/B978-0-12-408089-8.00001-X, vol 91. Elsevier, pp 1–58

  • Walkinshaw N, Bogdanov K (2013) Automated comparison of state-based software models in terms of their language and structure. ACM Trans Softw Eng Methodol 22(2):1–37. https://doi.org/10.1145/2430545.2430549

    Article  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Systematic literature reviews. In: Experimentation in software engineering. https://doi.org/10.1007/978-3-642-29044-2_4. Springer, Berlin, pp 45–54

  • Wohlin C, Runeson P, Hst M, Ohlsson MC, Regnell B, Wessln A (2012) Experimentation in software engineering. Springer Publishing Company, Incorporated, Berlin. https://doi.org/10.1007/978-3-642-29044-2

    Book  Google Scholar 

  • Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification & Reliability 22 (2):67–120. https://doi.org/10.1002/stv.430

    Article  Google Scholar 

Download references

Acknowledgments

This study was financed in part by the Coordenac ao de Aperfeicoamento de Pessoal 1167 de Nível Superior - Brasil (CAPES) - Finance Code 001, FAPESP (grant 2019/06937-0), and University of 1168 Leicester (UK), College of Science & Engineering. Research carried out using the computational resources of 1169 the Center for Mathematical Sciences Applied to Industry (CeMEAI) funded by FAPESP (grant 2013/07375- 1170 0). We have also been partially supported by the UKRI Trustworthy Autonomous Systems Node in Verifiability, Grant Award Reference EP/V026801/1. The authors are grateful to the anonymous reviewers, the VALidation and VErification (VALVE) group of 1171 the University of Leicester and Laboratory of Software Engineering (LabES) at the University of Sao Paulo 1172 for their insightful comments.

Author information

Authors and Affiliations

  1. Universidade de Sao Paulo – Instituto de Ciências Matemáticas e de Computação, Av. Trab. Sao Carlense, 400, Centro, São Carlos, SP, 13566-590, Brazil

    Carlos Diego Nascimento Damasceno & Adenilso da Silva Simao

  2. University of Leicester – Department of Informatics, University Rd, Leicester, LE1 7RH, UK

    Mohammad Reza Mousavi

Authors
  1. Carlos Diego Nascimento Damasceno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Reza Mousavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adenilso da Silva Simao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carlos Diego Nascimento Damasceno.

Additional information

Communicated by: Laurence Duchien, Thomas Thüm and Paul Grünbacher

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Configurable Systems

This study was financed in part by the Coordenaç ão de Aperfeiçoamento de Pessoal de Nível Superior -Brasil (CAPES) -Finance Code 001, FAPESP (grant 2019/06937-0), and University of Leicester (UK), College of Science & Engineering. Research carried out using the computational resources of the Center for Mathematical Sciences Applied to Industry (CeMEAI) funded by FAPESP (grant 2013/07375-0). We have also been partially supported by the UKRI Trustworthy Autonomous Systems Node in Verifiability, Grant Award Reference EP/V026801/1.

Appendix: A Glossary of Symbols

This glossary section lists the main symbols and abbreviations that are used in this manuscript with their following meanings.

Table 9
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Damasceno, C.D.N., Mousavi, M.R. & Simao, A.d.S. Learning by sampling: learning behavioral family models from software product lines. Empir Software Eng 26, 4 (2021). https://doi.org/10.1007/s10664-020-09912-w

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-020-09912-w

Keywords

Search

Navigation