Skip to main content
SpringerLink
Log in

Statistical prioritization for software product line testing: an experience report

  • Theme Section Paper
  • Published:
Software & Systems Modeling Aims and scope Submit manuscript

Abstract

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion™, an embedded system providing helicopter landing assistance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Complete models are downloadable at https://projects.info.unamur.be/vibes/

  2. http://webcampus.unamur.be

  3. Tool implementation may be downloaded at https://projects.info.unamur.be/yami/

References

  1. Airbus defence & space - sferion: http://www.defenceandsecurity-airbusds.com/fr/sferion

  2. ALL4TEC - MaTeLo: http://all4tec.net/index.php/en/model-based-testing/20-markov-test-logic-matelo

  3. Asirelli, P., ter Beek, M., Gnesi, S., Fantechi, A.: Formal description of variability in product families. In: Software product line conference (SPLC), 15th International, pp. 130–139 (2011). doi:10.1109/SPLC.2011.34

  4. Asirelli, P., ter Beek, M.H., Fantechi, A., Gnesi, S., Mazzanti, F.: Design and validation of variability in product lines. In: Proceedings of the 2nd International Workshop on Product Line Approaches in Software Engineering, PLEASE ’11, ACM, New York, 25–30. (2011). doi:10.1145/1985484.1985492

  5. Baier, C., Katoen, J.P.: Principles of Model Checking. MIT Press, Massachusetts (2008)

    MATH  Google Scholar 

  6. Burch, J.R., Clarke, E.M., McMillan, K.L., Dill, D.L., Hwang, L.J.: Symbolic model checking: 1020 states and beyond. Inf. Comput. 98(2), 142–170 (1992). doi:10.1016/0890-5401(92)90017-A

    Article  MATH  Google Scholar 

  7. Cichos, H., Oster, S., Lochau, M., Schürr, A.: Model-based coverage-driven test suite generation for software product lines. In: Proceedings of the 14th International Conference on Model Driven Engineering Languages and Systems, MODELS’11, Springer, Heidelberg, pp.425–439. (2011). http://dl.acm.org/citation.cfm?id=2050655.2050698

  8. Claroline: http://www.claroline.net/

  9. Classen, A., Boucher, Q., Heymans, P.: A text-based approach to feature modelling: syntax and semantics of TVL. Science of computer programming 76(12), 1130–1143 (2011). doi:10.1016/j.scico.2010.10.005. http://linkinghub.elsevier.com/retrieve/pii/S0167642310001899

  10. Classen, A., Cordy, M., Schobbens, P.Y., Heymans, P., Legay, A., Raskin, J.F.: Featured transition systems: foundations for verifying variability-intensive systems and their application to LTL model checking. IEEE Trans. Softw. Eng. 39(8), 1069–1089 (2013). doi:10.1109/TSE.2012.86

    Article  Google Scholar 

  11. Classen, A., Heymans, P., Schobbens, P.Y.: What’s in a Feature: A requirements engineering perspective. In: Fiadeiro, J.L., Inverardi, P. (eds.) Proceedings of the 11th International Conference on Fundamental Approaches to Software Engineering (FASE’08), Held as Part of the Joint European Conferences on Theory and Practice of Software (ETAPS’08), LNCS, vol. 4961, pp. 16–30. Springer, Heidelberg (2008). http://www.cs.le.ac.uk/events/fase2008/

  12. Classen, A., Heymans, P., Schobbens, P.Y., Legay, A.: Symbolic model checking of software product lines. In: Proceedings of the 33rd International Conference on Software Engineering, ICSE ’11, pp. 321–330. ACM, New York (2011). doi:10.1145/1985793.1985838

  13. Classen, A., Heymans, P., Schobbens, P.Y., Legay, A., Raskin, J.F.: Model checking lots of systems: efficient verification of temporal properties in software product lines. In: Proceedings of the 32Nd ACM/IEEE International Conference on Software Engineering - Vol 1, ICSE ’10, pp. 335–344. ACM, New York (2010). doi:10.1145/1806799.1806850

  14. Cohen, M., Dwyer, M., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008). doi:10.1109/TSE.2008.50. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4564473

  15. Cohen, M.B., Dwyer, M.B., Shi, J.: Coverage and adequacy in software product line testing. In: Proceedings of the ISSTA 2006 Workshop on Role of Software Architecture for Testing and Analysis, ROSATEA ’06, pp. 53–63. ACM, New York (2006). doi:10.1145/1147249.1147257

  16. Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: Proceedings of the 2007 International Symposium on Software Testing and Analysis, ISSTA ’07, pp. 129–139. ACM, New York (2007). doi:10.1145/1273463.1273482

  17. Cordy, M., Classen, A., Heymans, P., Schobbens, P.Y., Legay, A.: Provelines: a product line of verifiers for software product lines. In: Proceedings of the 17th International Software Product Line Conference Co-located Workshops, SPLC ’13 Workshops, pp. 141–146. ACM, New York (2013). doi:10.1145/2499777.2499781

  18. Cordy, M., Heymans, P., Schobbens, P.Y., Sharifloo, A.M., Ghezzi, C., Legay, A.: Verification for reliable product lines. arXiv:1311.1343 (2013)

  19. Czarnecki, K., Wasowski, A.: Feature diagrams and logics: there and back again. In: SPLC ’07, pp. 23–34. IEEE (2007). doi:10.1109/SPLINE.2007.24

  20. Devroey, X., Cordy, M., Perrouin, G., Kang, E.Y., Schobbens, P.Y., Heymans, P., Legay, A., Baudry, B.: A vision for behavioural model-driven validation of software product lines. In: Margaria, T., Steffen, B. (eds.) Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change, Lecture Notes in Computer Science, vol. 7609, pp. 208–222. Springer, Berlin Heidelberg (2012). doi:10.1007/978-3-642-34026-0_16

    Chapter  Google Scholar 

  21. Devroey, X., Perrouin, G.: Variability intensive system behavioural testing framework (VIBeS) (2014). https://projects.info.unamur.be/vibes/

  22. Devroey, X., Perrouin, G., Cordy, M., Schobbens, P., Legay, A., Heymans, P.: Towards statistical prioritization for software product lines testing. In: Collet, P., Wasowski, A., Weyer, T. (eds.) The Eighth International Workshop on Variability Modelling of Software-intensive Systems, VaMoS ’14, Sophia Antipolis, France, January 22–24, 2014, p. 10. ACM (2014). doi:10.1145/2556624.2556635

  23. Devroey, X., Perrouin, G., Legay, A., Cordy, M., Schobbens, P., Heymans, P.: Coverage criteria for behavioural testing of software product lines. In: Margaria, T., Steffen, B. (eds.) Leveraging Applications of Formal Methods, Verification and Validation. Technologies for Mastering Change - 6th International Symposium, ISoLA 2014, Imperial, Corfu, Greece, October 8–11, 2014, Proceedings, Part I, Lecture Notes in Computer Science, vol. 8802, pp. 336–350. Springer (2014). doi:10.1007/978-3-662-45234-9_24

  24. Devroey, X., Perrouin, G., Schobbens, P.: Abstract test case generation for behavioural testing of software product lines. In: Gnesi, S., Fantechi, A., ter Beek, M.H., Botterweck, G., Becker, M. (eds.) 18th International Software Product Lines Conference - Companion Volume for Workshop, Tools and Demo papers, SPLC ’14, Florence, Italy, September 15–19, 2014, pp. 86–93. ACM (2014). doi:10.1145/2647908.2655971

  25. Feliachi, A., Le Guen, H.: Generating transition probabilities for automatic model-based test generation. In: Software testing, verification and validation (ICST), third international conference, pp. 99–102 (2010). doi:10.1109/ICST.2010.26

  26. Fischbein, D., Uchitel, S., Braberman, V.: A foundation for behavioural conformance in software product line architectures. In: Proceedings of the ISSTA 2006 Workshop on Role of Software Architecture for Testing and Analysis, ROSATEA ’06, pp. 39–48. ACM, New York (2006). doi:10.1145/1147249.1147254

  27. Garca-Teodoro, P., Daz-Verdejo, J., Maci-Fernndez, G., Vzquez, E.: Anomaly-based network intrusion detection: techniques, systems and challenges. Comput. Secur. 28(12), 18–28 (2009). doi:10.1016/j.cose.2008.08.003. http://www.sciencedirect.com/science/article/pii/S0167404808000692

  28. Ghezzi, C., Pezzè, M., Sama, M., Tamburrelli, G.: Mining behavior models from user-intensive web applications categories and subject descriptors. In: 36th International Conference on Software Engineering, ICSE ’14. ACM, Hyderabad (2014)

  29. Gouraud, S.D., Denise, A., Gaudel, M.C., Marre, B.: A new way of automating statistical testing methods. In: Automated software engineering, 2001. (ASE 2001). In: Proceedings 16th Annual International Conference on Automated Software Engineering, pp. 5–12 (2001). doi:10.1109/ASE.2001.989785

  30. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Le Traon, Y.: Bypassing the combinatorial explosion: using similarity to generate and prioritize T-wise test configurations for software product lines. IEEE Trans. Soft. Eng. 40(7), 650–670 (2014). doi:10.1109/TSE.2014.2327020. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6823132

  31. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: Proceedings of the 17th International Software Product Line Conference, SPLC ’13, pp. 62–71. ACM, New York (2013). doi:10.1145/2491627.2491635

  32. Johansen, M., Haugen, O., Fleurey, F., Eldegard, A., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: France, R., Kazmeier, J., Breu, R., Atkinson, C. (eds.) Model Driven Engineering Languages and Systems, Lecture Notes in Computer Science, vol. 7590, pp. 269–284. Springer, Berlin Heidelberg (2012). doi:10.1007/978-3-642-33666-9_18

    Chapter  Google Scholar 

  33. Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Spencer Peterson, A.: Feature-oriented domain analysis (FODA) feasibility study. Tech. rep., Software Engineering Institute, Carnegie Mellon University (1990)

  34. Kim, C., Khurshid, S., Batory, D.: Shared execution for efficiently testing product lines. In: Software reliability engineering (ISSRE), IEEE 23rd International Symposium on, pp. 221–230 (2012). doi:10.1109/ISSRE.2012.23

  35. Lauenroth, K., Pohl, K., Toehning, S.: Model checking of domain artifacts in product line engineering. In: Automated software engineering, ASE ’09. 24th IEEE/ACM international conference, pp. 269–280 (2009). doi:10.1109/ASE.2009.16

  36. Lochau, M., Oster, S., Goltz, U., Schürr, A.: Model-based pairwise testing for feature interaction coverage in software product line engineering. Softw. Qual. J. 20(3–4), 567–604 (2012). doi:10.1007/s11219-011-9165-4

    Article  Google Scholar 

  37. Mathur, A.P.: Foundations of Software Testing. Pearson Education, New York (2008)

    Google Scholar 

  38. Metzger, A., Pohl, K.: Software product line engineering and variability management: achievements and challenges. In: Proceedings of the on Future of Software Engineering, FOSE, pp. 70–84. ACM, New York (2014). doi:10.1145/2593882.2593888

  39. Michel, R., Classen, A., Hubaux, A., Boucher, Q.: A formal semantics for feature cardinalities in feature diagrams. In: Proceedings of the 5th Workshop on Variability Modeling of Software-Intensive Systems, VaMoS, pp. 82–89. ACM, New York (2011). doi:10.1145/1944892.1944902

  40. Musa, J.D., Fuoco, G., Irving, N., Kropfl, D., Juhlin, B.: The operational profile. NATO ASI Ser. Comput. Syst. Sci. 154, 333–344 (1996)

    Google Scholar 

  41. Nguyen, H.V., Kästner, C., Nguyen, T.N.: Exploring variability-aware execution for testing plugin-based web applications. In: Jalote, P., Briand, L.C., van der Hoek, A. (eds.) 36th international conference on software engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014, pp. 907–918. ACM (2014). doi:10.1145/2568225.2568300

  42. Oster, S., Wöbbeke, A., Engels, G., Schürr, A.: Model-based software product lines testing survey. In: Model-Based Testing for Embedded Systems, pp. 339–382. CRC Press (2011)

  43. Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., le Traon, Y.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. 20(3–4), 605–643 (2012). doi:10.1007/s11219-011-9160-9

    Article  Google Scholar 

  44. Pohl, K., Böckle, G., Van Der Linden, F.: Software Product Line Engineering: Foundations, Principles, and Techniques. Springer, New York (2005)

    Book  MATH  Google Scholar 

  45. Prowell, S.J.: JUMBL: a tool for model-based statistical testing. In: System sciences. In: Proceedings of the 36th Annual Hawaii International Conference, p. 9 (2003). doi:10.1109/HICSS.2003.1174916

  46. von Rhein, A., Apel, S., Kästner, C., Thüm, T., Schaefer, I.: The PLA model: on the combination of product-line analyses. In: Proceedings of the Seventh International Workshop on Variability Modelling of Software-intensive Systems (VaMoS). pp. 14:1–14:8. ACM, New York (2013). doi:10.1145/2430502.2430522

  47. Samih, H., Acher, M., Bogusch, R., Le Guen, H., Baudry, B.: Deriving usage model variants for model-based testing: an industrial case study. In: IEEE (ed.) 19th international conference on engineering of complex computer systems (ICECCS). Tianjin, Chine (2014). http://hal.inria.fr/hal-01002099

  48. Samih, H., Bogusch, R.: MPLM - MaTeLo Product line manager. In: Proceedings of the 18th International Software Product Line Conference: Companion Volume for Workshops, Demonstrations and Tools - Vol 2, SPLC , pp. 138–142. ACM, New York (2014). doi:10.1145/2647908.2655980

  49. Samih, H., Le Guen, H., Bogusch, R., Acher, M., Baudry, B.: An Approach to Derive Usage Models Variants for Model-based Testing. In: The 26th IFIP International Conference on Testing Software and Systems (2014). Springer, Madrid, Espagne (2014). http://hal.inria.fr/hal-01025124

  50. Sampath, S., Bryce, R.C., Viswanath, G., Kandimalla, V., Koru, a.G.: Prioritizing user-session-based test cases for web applications testing. In: 1st international conference on Software testing, verification, and validation, pp. 141–150. IEEE (2008). doi:10.1109/ICST.2008.42. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=4539541

  51. Schobbens, P.Y., Heymans, P., Trigaux, J.C., Bontemps, Y.: Generic semantics of feature diagrams. Comput. Netw. 51(2), 456–479 (2007). doi:10.1016/j.comnet.2006.08.008. http://www.sciencedirect.com/science/article/pii/S1389128606002179. Feature Interaction

  52. Scrapy: http://scrapy.org/

  53. Segura, S., Sánchez, A.B., Ruiz-Cortés, A.: Automated variability analysis and testing of an e-commerce site.: an experience report. In: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE ’14, pp. 139–150. ACM, New York (2014). doi:10.1145/2642937.2642939

  54. Sprenkle, S.E., Pollock, L.L., Simko, L.M.: Configuring effective navigation models and abstract test cases for web applications by analysing user behaviour. Softw. Test. Verif. Reliab. 23(6), 439–464 (2013). doi:10.1002/stvr.1496

    Article  Google Scholar 

  55. Thévenod-Fosse, P., Waeselynck, H.: An investigation of statistical software testing. Softw. Test. Verif. Reliab. 1(2), 5–25 (1991)

    Article  Google Scholar 

  56. Tretmans, J.: Model based testing with labelled transition systems. In: Hierons, R., Bowen, J., Harman, M. (eds.) Formal Methods and Testing, Lecture Notes in Computer Science, vol. 4949, pp. 1–38. Springer, Heidelberg (2008). doi:10.1007/978-3-540-78917-8_1

    Google Scholar 

  57. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, Burlington (2007)

    Google Scholar 

  58. Utting, M., Pretschner, A., Legeard, B.: A taxonomy of model-based testing approaches (April 2011), 297–312 (2012). doi:10.1002/stvr

  59. Verwer, S., Eyraud, R., De La Higuera, C.: PAutomaC: a probabilistic automata and hidden Markov models learning competition. Mach. Learn. pp. 1–26 (2013). doi:10.1007/s10994-013-5409-9. https://hal.archives-ouvertes.fr/hal-00873981

  60. Viterbi, A.J.: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory 13(2), 260–269 (1967)

    Article  MATH  Google Scholar 

  61. Weißleder, S., Sokenou, D., Schlingloff, B.: Reusing state machines for automatic test generation in product lines. In: 1st workshop on model-based testing in practice (MoTiP), p. 19. Citeseer, Berlin (2008). http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.169.5699&rep=rep1&type=pdf#page=21

  62. Whittaker, J., Thomason Michael, G.: A Markov chain model for statistical software testing. IEEE Trans. Softw. Eng. 20(10), 812–824 (1994). doi:10.1109/32.328991

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank Jean-Roch Meurisse and Didier Belhomme from the University of Namur for providing the Webcampus Apache access log.

Author information

Author notes

  1. Hamza Samih

    Present address: Alcatel-Lucent, IP T&R / Wireless Transmission, Route De VilleJust, 91620, Nozay, France

Authors and Affiliations

  1. PReCISE, University of Namur, Rue Grandgagnage 21, 5000, Namur, Belgium

    Xavier Devroey, Gilles Perrouin, Maxime Cordy, Pierre-Yves Schobbens & Patrick Heymans

  2. All4tec GL, 6 rue Léonard de Vinci, BP 0119, 53001, Laval Cedex, France

    Hamza Samih

  3. Inria Rennes, Bretagne Atlantique, Campus universitaire de Beaulieu, 35042, Rennes Cedex, France

    Hamza Samih & Axel Legay

Authors
  1. Xavier Devroey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gilles Perrouin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maxime Cordy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hamza Samih
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Axel Legay
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Pierre-Yves Schobbens
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Patrick Heymans
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gilles Perrouin.

Additional information

Communicated by Prof. Andrzej Wąsowski and Thorsten Weyer.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Devroey, X., Perrouin, G., Cordy, M. et al. Statistical prioritization for software product line testing: an experience report. Softw Syst Model 16, 153–171 (2017). https://doi.org/10.1007/s10270-015-0479-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-015-0479-8

Keywords

Mathematics Subject Classification

Search

Navigation