Abstract
We propose a new Statistical Model Checking (SMC) method to identify bugs in variability-intensive systems (VIS). The state-space of such systems is exponential in the number of variants, which makes the verification problem harder than for classical systems. To reduce verification time, we propose to combine SMC with featured transition systems (FTS)—a model that represents jointly the state spaces of all variants. Our new methods allow the sampling of executions from one or more (potentially all) variants. We investigate their utility in two complementary use cases. The first case considers the problem of finding all variants that violate a given property expressed in Linear-Time Logic (LTL) within a given simulation budget. To achieve this, we perform random walks in the featured transition system seeking accepting lassos. We show that our method allows us to find bugs much faster (up to 16 times according to our experiments) than exhaustive methods. As any simulation-based approach, however, the risk of Type-1 error exists. We provide a lower bound and an upper bound for the number of simulations to perform to achieve the desired level of confidence. Our empirical study involving 59 properties over three case studies reveals that our method manages to discover all variants violating 41 of the properties. This indicates that SMC can act as a coarse-grained analysis method to quickly identify the set of buggy variants. The second case complements the first one. In case the coarse-grained analysis reveals that no variant can guarantee to satisfy an intended property in all their executions, one should identify the variant that minimizes the probability of violating this property. Thus, we propose a fine-grained SMC method that quickly identifies promising variants and accurately estimates their violation probability. We evaluate different selection strategies and reveal that a genetic algorithm combined with elitist selection yields the best results.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ammann PE, Black PE, Majurski W (1998) Using model checking to generate tests from specifications. In: Proceedings second international conference on formal engineering methods (Cat.No.98EX241), pp 46–54
Boucher Q, Classen A, Heymans P, Bourdoux A, Demonceau L (2010) Tag and prune: a pragmatic approach to software product line implementation. In: ASE’10. ACM, pp 333–336
Budde CE, D’Argenio PR, Hermanns H (2015) Rare event simulation with fully automated importance splitting. In: Beltrán M, Knottenbelt WJ, Bradley JT (eds) Computer performance engineering–12th European workshop, EPEW 2015, Madrid, Spain, August 31–September 1, 2015, Proceedings, volume 9272 of Lecture Notes in Computer Science. Springer, pp 275–290
Boneh A, Hofri M (1997) The coupon-collector problem revisited—a survey of engineering problems and computational methods. Commun Stat Stoch Models 13(1):39–66
Broy M, Jonsson B, Katoen J-P, Leucker M, Pretschner A (2005) (eds) Model-based testing of reactive systems, advanced lectures [the volume is the outcome of a research seminar that was held in Schloss Dagstuhl in January 2004], volume 3472 of Lecture Notes in Computer Science. Springer
Baier C, Katoen J-P (2008) Principles of model checking. MIT Press
Budde CE, Stoelinga M (2020) Automated rare event simulation for fault tree analysis via minimal cut sets. In: Hermanns H (eds) Measurement, modelling and evaluation of computing systems—20th international GI/ITG conference, MMB 2020, Saarbrücken, Germany, March 16–18, 2020, Proceedings, volume 12040 of Lecture Notes in Computer Science. Springer, pp 259–277
Benavides D, Segura S, Ruiz-Cortés A (September 2010) Automated analysis of feature models 20 years later: a literature review. Inf Syst 35(6):615–636
Classen A, Cordy M, Schobbens P-Y, Heymans P, Legay A, Raskin J-F (2013) Featured transition systems: foundations for verifying variability-intensive systems and their application to LTL model checking. In: Transactions on software engineering, pp 1069–1089
Chechik M, Devereux B, Easterbrook SM, Gurfinkel A (2003) Multi-valued symbolic model-checking. ACMTrans Softw Eng Methodol 12(4):371–408
Cordy M, Heymans P, Legay A, Schobbens P-Y, Dawagne B, Leucker M(2014) Counterexample guided abstraction refinement of product-line behavioural models. In: FSE’14. ACM
Classen A, Heymans P, Schobbens P-Y, Legay A, Raskin J-F (2010) Model checking lots of systems: efficient verification of temporal properties in software product lines. In: ICSE’10. ACM, pp 335–344
Chockler H, Ivrii A , Matsliah A, Rollini SF, Sharygina N (2013) Using cross-entropy for satisfiability. In: Shin SY, Maldonado JC (eds) Proceedings of the 28th annual ACM symposium on applied computing, SAC ’13, Coimbra, Portugal, March 18–22. ACM, pp 1196–1203
Cordy M, Legay A, Lazreg S, Collet P (2019) Towards sampling and simulation-based analysis of featured weighted automata. In: Proceedings of the 7th international workshop on formal methods in software engineering, FormaliSE@ICSE 2019, Montreal, QC, Canada, May 27, 2019. pp 61–64
Clements PC, Northrop L (August 2001) Software product lines: practices and patterns. SEI Series in Software Engineering. Addison-Wesley
Consultative Committee for Space Data Systems (CCSDS). CCSDS file delivery protocol (CFDP): Blue Book, Issue 4. NASA (2007)
Cordy M, Schobbens P-Y, Heymans P, Legay A (2013) Beyond Boolean product-line model checking: dealing with feature attributes and multi-features. In: ICSE’13. IEEE, pp 472–481
Cordy M, Schobbens P-Y, Heymans P, Legay A (2013) Provelines: a product-line of verifiers for software product lines. In: SPLC’13. ACM, pp 141–146
Delahaye B, Fournier P, Lime D (2019) Statistical model checking for parameterized models
Daca P, Henzinger TA, Kretínský J, Petrov T (2017) Faster statistical model checking for unbounded temporal properties. ACM Trans Comput Log 18(2):12:1–12:25
D’Argenio PR, Hartmanns A, Sedwards S (2018) Lightweight statistical model checking in nondeterministic continuous time. In: Margaria T, Steffen B (eds) Leveraging applications of formal methods, verification and validation. Verification—8th international symposium, ISoLA 2018, Limassol, Cyprus, November 5–9, 2018, Proceedings, Part II, volume 11245 of Lecture notes in computer science. Springer, pp 336–353
Dubslaff C, Klüppelholz S, Baier C (2014) Probabilistic model checking for energy analysis in software product lines. In: Binder W, Ernst E, Peternier A, Hirschfeld R (eds) 13th International conference on modularity, Modularity ’14, Lugano, Switzerland, April 22–26, 2014. ACM, pp 169–180
Gruler A, Leucker M, Scheidemann K (2008) Modeling and model checking software product lines. In: International conference on formal methods for open object-based distributed systems. Springer, pp 113–131
Grosu R, Smolka SA (2005) Monte Carlo model checking. In: Halbwachs N, Zuck LD (eds) Tools and algorithms for the construction and analysis of systems. Berlin, Heidelberg, pp 271–286
Guo J, White J, Wang G, Li J, Wang Y (2011) A genetic algorithm for optimized feature selection with resource constraints in software product lines. J Syst Softw 84(12):2208–2221
Harman M, Jia Y, Krinke J, Langdon WB, Petke J, Zhang Y (2014) Search based software engineering for software product line engineering: a survey and directions for future work. In: Proceedings of the 18th international software product line conference-volume 1, pp 5–18
Henard C, Papadakis M, Harman M, Le TY (2015) Combining multi-objective search and constraint solving for configuring large software product lines. In: Proceedings of ICSE ’15. IEEE Press, pp 517–528
Jégourel C, Legay A, Sedwards S (2013) Importance splitting for statistical model checking rare properties. In: Sharygina N, Veith H (eds) Computer aided verification—25th international conference, CAV 2013, Saint Petersburg, Russia, July 13–19, 2013. Proceedings, volume 8044 of lecture notes in computer science. Springer, pp 576–591
Kang K, Cohen S, Hess J, Novak W, Peterson S (1990) Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21
Kaltenecker C, Grebhahn A, Siegmund N, Guo J, Apel S (2019) Distance-based sampling of software configuration spaces. In: Atlee JM, Bultan T, Whittle J (eds) Proceedings of the 41st international conference on software engineering, ICSE 2019, Montreal, QC, Canada, May 25–31, 2019. IEEE/ACM, pp 1084–1094
Kramer J, Magee J, Sloman M, Lister A (1983) Conic: an integrated approach to distributed computer control systems. Comput Digit Tech IEE Proc E 130(1):1–10
Legay A, Delahaye B, Bensalem S (2010) Statistical model checking: an overview. In: Runtime verification—first international conference, RV 2010, St. Julians, Malta, November 1–4, 2010. Proceedings, pp 122–135
Larsen KG, Legay A (2018) Statistical model checking the 2018 edition! In:Margaria T, Steffen B (eds) Leveraging applications of formal methods, verification and validation. Verification—8th international symposium, ISoLA 2018, Limassol, Cyprus, November 5–9, 2018, Proceedings, Part II, volume 11245 of lecture notes in computer science. Springer, pp 261–270
Muschevici R, Clarke D, Proença J (2010) Feature petri nets. In: Proceedings of the 14th international software product line conference (SPLC 2010), volume 2. Lancaster University; Lancaster, United Kingdom
Oudinet J, Denise A, Gaudel M-C, Lassaigne R, Peyronnet S (2011) Uniform Monte-Carlo model checking. In:Giannakopoulou D, Orejas F (eds) Fundamental approaches to software engineering—14th international conference, FASE 2011, held as part of the joint European conferences on theory and practice of software, ETAPS 2011, Saarbrücken, Germany, March 26–April 3, 2011. Proceedings, volume 6603 of lecture notes in computer science. Springer, pp 127–140
Oh J, Gazzillo P, Batory DS (2019) t-wise coverage by uniformsampling. In: Berger T, Collet P, Duchien L, Fogdal T, Heymans P, Kehrer T, Martinez J, Mazo R, Montalvillo L, Salinesi C,Tërnava X,Thüm T, Ziadi T (eds) Proceedings of the 23rd international systems and software product line conference, SPLC 2019, Volume A, Paris, France, September 9–13, 2019. ACM, pp 15:1–15:4
Olaechea R, Rayside D, Guo J, Czarnecki K (2014) Comparison of exact and approximate multi-objective optimization for software product lines. In: Proceedings of the 18th international software product line conference-volume 1, pp 92–101
Olaechea R, Stewart S, Czarnecki K, Rayside D (2012) Modelling and multi-objective optimization of quality attributes in variability-rich software. In: Proceedings of the fourth international workshop on nonfunctional system properties in domain specific modeling languages, pp 1–6
Plazar Q, Acher M, Perrouin G, Devroey X, Cordy M (2019) Uniform sampling of SAT solutions for configurable systems: are we there yet? In: 12th IEEE conference on software testing, validation and verification, ICST 2019, Xi’an, China, April 22–27, 2019. IEEE, pp 240–251
Pnueli A (1977) The temporal logic of programs. In: FOCS’77, pp 46–57
Plath M, Ryan M (2001) Feature integration using a feature construct. SCP 41(1):53–84
Pett T, Thüm T, Runge T, Krieter S, Lochau M, Schaefer I (2019) Product sampling for product lines: the scalability challenge. In: Proceedings of the 23rd international systems and software product line conference—Volume A, SPLC ’19. Association for Computing Machinery, New York, pp 78–83
Rodrigues GN, Alves V, Nunes V, Lanna A, Cordy M, Schobbens P-Y, Sharifloo AM, Legay A (2015) Modeling and verification for probabilistic properties in software product lines. In: HASE 2015, Daytona Beach, FL, USA, January 8–10, 2015, pp 173–180
Rakshit P, Konar A, Das S (2017) Noisy evolutionary optimization algorithms—a comprehensive survey. Swarm Evol Comput 33:18–45
Shioda S (2007) Some upper and lower bounds on the coupon collector problem. J Comput Appl Math 200(1):154–167
Siegmund N, Rosenmüller M, Kuhlemann M, Kästner C, Apel S, Saake G (2012) Spl conqueror: toward optimization of non-functional properties in software product lines. Softw Qual J 20(3):487–517
Sabin D, Weigel R (Jul 1998) Product configuration frameworks-a survey. IEEE Intell Syst Their Appl 13(4):42–49
Thüm T, Apel S, Kästner C, Schaefer I, Saake G (2014) A classification and survey of analysis strategies for software product lines. ACM Comput Surv 47(1):6:1–6:45
ter Beek MH, Fantechi A, Gnesi S, Mazzanti F (2016) Modelling and analysing variability in product families: model checking of modal transition systems with variability constraints. J Log Algebra Methods Programm 85(2):287–315
ter Beek MH, Legay A, Lluch-Lafuente A, Vandin A (2020) A framework for quantitative modeling and analysis of highly (re)configurable systems. IEEE Trans Softw Eng 46(3):321–345
Thüm T, van Hoorn A, Apel S, Bürdek J, Getir S, Heinrich R, Jung R, Kowal M, Lochau M, Schaefer I, Walter J (2019) Performance analysis strategies for software variants and versions. In: Managed software evolution, pp 175–206
Vandin A, ter Beek MH, Legay A, Lluch-Lafuente A (2018) Qflan: A tool for the quantitative analysis of highly reconfigurable systems. In: Havelund K, Peleska J, Roscoe B, de Vink EP (eds) Formal methods—22nd international symposium, FM 2018, Held as Part of the Federated Logic Conference, FloC 2018, Oxford, UK, July 15–17, 2018, Proceedings, volume 10951 of Lecture Notes in Computer Science. Springer, pp 329–337
Vardi MY, Wolper P (1986) An automata-theoretic approach to automatic program verification. In: LICS’86. IEEE CS, pp 332–344
Wagner J, Kuznetsov V, Candea G, Kinder J (2015) High system-code security with low overhead. In: Proceedings of the 2015 IEEE symposium on security and privacy, SP ’15. IEEE Computer Society, USA, pp 866–879
Younes HLS, Clarke EM, Zuliani P (2010) Statistical verification of probabilistic properties with unbounded until. In: Davies J, Silva L, da Silva SA (eds) Formal methods: Foundations and applications—13th Brazilian symposium on formal methods, SBMF 2010, Natal, Brazil, November 8–11, 2010, Revised Selected Papers, volume 6527 of Lecture Notes in Computer Science. Springer, pp 144–160
Younes HLS, Simmons RG (2002) Probabilistic verification of discrete event systems using acceptance sampling. In Brinksma E, LarsenKG(eds) Computer aided verification, 14th international conference, CAV 2002, Copenhagen, Denmark, July 27–31, 2002, Proceedings, volume 2404 of lecture notes in computer science. Springer, pp 223–235
Acknowledgements
Maxime Cordy and Sami Lazreg are supported by FNR Luxembourg (grant C19/IS/13566661/BEEHIVE/Cordy).
Author information
Authors and Affiliations
Corresponding author
Additional information
Jordi Cabot, Heike Wehrheim and Eerke Boiten
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Cordy, M., Lazreg, S., Papadakis, M. et al. Statistical model checking for variability-intensive systems: applications to bug detection and minimization. Form Asp Comp 33, 1147–1172 (2021). https://doi.org/10.1007/s00165-021-00563-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00165-021-00563-2