Effective product-line testing using similarity-based product prioritization

  • Mustafa Al-Hajjaji
  • Thomas Thüm
  • Malte Lochau
  • Jens Meinicke
  • Gunter Saake
Regular Paper

Abstract

A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible due to the potentially exponential number of products in the number of features. Accordingly, several sampling approaches have been proposed to select a presumably minimal, yet sufficient number of products to be tested. Since the time budget for testing is limited or even a priori unknown, the order in which products are tested is crucial for effective product-line testing. Prioritizing products is required to increase the probability of detecting faults faster. In this article, we propose similarity-based prioritization, which can be efficiently applied on product samples. In our approach, we incrementally select the most diverse product in terms of features to be tested next in order to increase feature interaction coverage as fast as possible during product-by-product testing. We evaluate the gain in the effectiveness of similarity-based prioritization on three product lines with real faults. Furthermore, we compare similarity-based prioritization to random orders, an interaction-based approach, and the default orders produced by existing sampling algorithms considering feature models of various sizes. The results show that our approach potentially increases effectiveness in terms of fault detection ratio concerning faults within real-world product-line implementations as well as synthetically seeded faults. Moreover, we show that the default orders of recent sampling algorithms already show promising results, which, however, can still be improved in many cases using similarity-based prioritization.

Keywords

Software product lines Product-line testing Model-based testing Combinatorial interaction testing Test-case prioritization 

Notes

Acknowledgements

We are grateful to anonymous reviewers for their useful comments. We also thank Martin Fagereng Johansen for his support in integrating the SPLCATool into FeatureIDE. We are grateful to Remo Lachmann, Reimar Schröter, Fabian Benduhn, and Hauke Baller for helpful comments on a draft and for interesting discussions. This work has been supported by the German Research Foundation (DFG) in the Priority Programme SPP 1593: Design For Future Managed Software Evolution (LO 2198/2-1).

References

  1. 1.
    Abal, I., Brabrand, C., Wasowski, A.: 42 variability bugs in the linux kernel: a qualitative analysis. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 421–432. ACM (2014)Google Scholar
  2. 2.
    Al-Hajjaji, M.: Scalable sampling and prioritization for product-line testing. In: Proc. Software Engineering (SE), pp. 295–298. Gesellschaft für Informatik (GI) (2015)Google Scholar
  3. 3.
    Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 197–206. ACM (2014)Google Scholar
  4. 4.
    Al-Hajjaji, M., Krieter, S., Thüm, T., Lochau, M., Saake, G.: IncLing: efficient product-line testing using incremental pairwise sampling. In: Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), pp. 144–155. ACM (2016)Google Scholar
  5. 5.
    Al-Hajjaji, M., Meinicke, J., Krieter, S., Schröter, R., Thüm, T., Leich, T., Saake, G.: Tool demo: testing configurable systems with featureIDE. In: Proc. Int’l Conf. Generative Programming and Component Engineering (GPCE), pp. 173–177. ACM (2016)Google Scholar
  6. 6.
    Apel, S., Batory, D., Kästner, C., Saake, G.: Feature-Oriented Software Product Lines: Concepts and Implementation. Springer, Berlin (2013)CrossRefGoogle Scholar
  7. 7.
    Apel, S., von Rhein, A., Wendler, P., Größlinger, A., Beyer, D.: Strategies for product-line verification: case studies and experiments. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 482–491. IEEE (2013)Google Scholar
  8. 8.
    Arcuri, A., Briand, L.: A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 1–10. ACM (2011)Google Scholar
  9. 9.
    Bagheri, E., Ensan, F., Gasevic, D.: Grammar-based test generation for software product line feature models. In: Proc. Conf. Centre for Advanced Studies on Collaborative Research (CASCON), pp. 87–101. IBM Corp. (2012)Google Scholar
  10. 10.
    Baller, H., Lochau, M.: Towards incremental test suite optimization for software product lines. In: Proc. Int’l Workshop Feature-Oriented Software Development (FOSD), pp. 30–36. ACM (2014)Google Scholar
  11. 11.
    Baller, H., Lity, S., Lochau, M., Schaefer, I.: Multi-objective test suite optimization for incremental product family testing. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 303–312. IEEE (2014)Google Scholar
  12. 12.
    Bryce, R.C., Colbourn, C.J.: One-test-at-a-time heuristic search for interaction test suites. In: Proc. Int’l Conf. on Genetic and Evolutionary Computation (GECCO), pp. 1082–1089. ACM (2007)Google Scholar
  13. 13.
    Bryce, R.C., Memon, A.M.: Test suite prioritization by interaction coverage. In: Workshop on Domain Specific Approaches to Software Test Automation: Conjunction with the 6th ESEC/FSE Joint Meeting, DOSTA ’07, pp. 1–7. ACM (2007)Google Scholar
  14. 14.
    Bürdek, J., Lochau, M., Bauregger, S., Holzer, A., von Rhein, A., Apel, S., Beyer, D.: Facilitating reuse in multi-goal test-suite generation for software product lines. In: Egyed, A., Schaefer, I. (eds.) Fundamental Approaches to Software Engineering, volume 9033 of Lecture Notes in Computer Science, pp. 84–99. Springer (2015)Google Scholar
  15. 15.
    Carmo Machado, I.D., McGregor, J.D., Cavalcanti, YaC, De Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. J. Inf. Softw. Technol. (IST) 56(10), 1183–1199 (2014)CrossRefGoogle Scholar
  16. 16.
    Cartaxo, E.G., Machado, P.D., Neto, F.G.O.: On the use of a similarity function for test case selection in the context of model-based testing. Softw. Test. Verif. Reliab. (STVR) 21(2), 75–100 (2011)CrossRefGoogle Scholar
  17. 17.
    Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pp. 129–139. ACM (2007)Google Scholar
  19. 19.
    Devroey, X., Perrouin, G., Cordy, M., Schobbens, P.-Y., Legay, A., Heymans, P.: Towards statistical prioritization for software product lines testing. In: Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 10:1–10:7. ACM (2014)Google Scholar
  20. 20.
    Devroey, X., Perrouin, G., Legay, A., Schobbens, P.-Y., Heymans, P.: Search-based similarity-driven behavioural SPL testing. In: Proc. Int’l Workshop Variability Modelling of Software-Intensive Systems (VaMoS), pp. 89–96. ACM (2016)Google Scholar
  21. 21.
    Dietrich, C., Tartler, R., Schröder-Preikschat, W., Lohmann, D.: Understanding linux feature distribution. In: Proc. of Workshop on Modularity in Systems Software (MISS), pp. 15–20. ACM (2012)Google Scholar
  22. 22.
    Elbaum, S., Malishevsky, A.G., Rothermel, G.: Prioritizing test cases for regression testing. SIGSOFT Softw. Eng. Notes 25(5), 102–112 (2000)CrossRefGoogle Scholar
  23. 23.
    Engström, E., Runeson, P.: Software product line testing—a systematic mapping study. J. Inf. Softw. Technol. (IST) 53(1), 2–13 (2011)CrossRefGoogle Scholar
  24. 24.
    Ensan, A., Bagheri, E., Asadi, M., Gasevic, D., Biletskiy, Y.: Goal-oriented test case selection and prioritization for product line feature models. In: Proc. Int’l Conf. on Information Technology: New Generations (ITNG), pp. 291–298. IEEE (2011)Google Scholar
  25. 25.
    Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Proc. Int’l Conf. Advanced Information Systems Engineering (CAiSE), vol. 7328, pp. 613–628. Springer (2012)Google Scholar
  26. 26.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir. Softw. Eng. (EMSE) 16(1), 61–102 (2011)CrossRefGoogle Scholar
  27. 27.
    Hall, R.J.: Fundamental nonmodularity in electronic mail. Autom. Softw. Eng. 12(1), 41–79 (2005)CrossRefGoogle Scholar
  28. 28.
    Hamming, R.W.: Error detecting and error correcting codes. Bell Syst. Tech. J. 29(2), 147–160 (1950)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Harrold, M.J.: Testing: a roadmap. In: Proc. of the Conf. on The Future of Software Engineering (FSE), pp. 61–72. ACM (2000)Google Scholar
  30. 30.
    Hemmati, H., Briand, L.: An industrial investigation of similarity measures for model-based test case selection. In: Proc. Int’l Symposium Software Reliability Engineering (ISSRE), pp. 141–150. IEEE (2010)Google Scholar
  31. 31.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: PLEDGE: a product line editor and test generation tool. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 126–129. ACM (2013)Google Scholar
  32. 32.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Le Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize T-wise test configurations for software product lines. IEEE Trans. Softw. Eng. (TSE) 40(7), 650–670 (2014)CrossRefGoogle Scholar
  33. 33.
    Jackson, M., Zave, P.: Distributed feature composition: a virtual architecture for telecommunications services. IEEE Trans. Softw. Eng. (TSE) 24(10), 831–847 (1998)CrossRefGoogle Scholar
  34. 34.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 638–652. Springer (2011)Google Scholar
  35. 35.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating T-wise covering arrays from large feature models. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 46–55. ACM (2012)Google Scholar
  36. 36.
    Johansen, M.F., Haugen, Ø., Fleurey, F., Eldegard, A.G., Syversen, T.: Generating better partial covering arrays by modeling weights on sub-product lines. In: Proc. Int’l Conf. Model Driven Engineering Languages and Systems (MODELS), pp. 269–284. Springer (2012)Google Scholar
  37. 37.
    Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute (1990)Google Scholar
  38. 38.
    Kästner, C., Apel, S., ur Rahman, S.S., Rosenmüller, M., Batory, D., Saake, G.: On the impact of the optional feature problem: analysis and case studies. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 181–190. Software Engineering Institute (2009)Google Scholar
  39. 39.
    Kramer, J., Magee, J., Sloman, M., Lister, A.: CONIC: an integrated approach to distributed computer control systems. IEE Proc. Comput. Dig. Tech. 130(1), 1 (1983)CrossRefGoogle Scholar
  40. 40.
    Kuhn, D.R., Wallace, D.R., Gallo Jr., A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. (TSE) 30(6), 418–421 (2004)CrossRefGoogle Scholar
  41. 41.
    Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., Schaefer, I.: Delta-oriented test case prioritization for integration testing of software product lines. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 81–90. ACM (2015)Google Scholar
  42. 42.
    Lei, Y., Kacker, R.N., Kuhn, D.R., Okun, V., Lawrence, J.: IPOG: a general strategy for T-way software testing. In: Proc. Int’l Conf. Engineering of Computer-Based Systems (ECBS), pp. 549–556. IEEE (2007)Google Scholar
  43. 43.
    McGregor, J.: Testing a software product line. In: Testing Techniques in Software Engineering, pp. 104–140. Springer, Berlin (2010)Google Scholar
  44. 44.
    Meinicke, J., Wong, C.-P., Kästner, C., Thüm, T., Saake, G.: On essential configuration complexity: measuring interactions in highly-configurable systems. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 483–494. ACM (2016)Google Scholar
  45. 45.
    Mendonça, M., Branco, M., Cowan, D.: SPLOT: software product lines online tools. In: Proc. Conf. Object-Oriented Programming, Systems, Languages and Applications (OOPSLA), pp. 761–762. ACM (2009)Google Scholar
  46. 46.
    Myers, G.J., Sandler, C., Badgett, T.: The Art of Software Testing, 3rd edn. Wiley, New York (2011)Google Scholar
  47. 47.
    Nguyen, H.V., Kästner, C., Nguyen, T.N.: Exploring variability-aware execution for testing plugin-based web applications. In: Proc. Int’l Conf. Software Engineering (ICSE), pp. 907–918. ACM (2014)Google Scholar
  48. 48.
    Northrop, L., Clements, P.: A framework for software product line practice, version 5.0. SEI (2007)Google Scholar
  49. 49.
    Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 196–210. Springer (2010)Google Scholar
  50. 50.
    Perrouin, G., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Automated and scalable T-wise test case generation strategies for software product lines. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 459–468. IEEE (2010)Google Scholar
  51. 51.
    Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. (SQJ) 20(3–4), 605–643 (2012)CrossRefGoogle Scholar
  52. 52.
    Plath, M., Ryan, M.: Feature integration using a feature construct. Sci. Comput. Program. (SCP) 41(1), 53–84 (2001)CrossRefMATHGoogle Scholar
  53. 53.
    Rothermel, G., Untch, R., Chu, C., Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. (TSE) 27(10), 929–948 (2001)CrossRefGoogle Scholar
  54. 54.
    Sánchez, A.B., Segura, S., Ruiz-Cortés, A.: A comparison of test case prioritization criteria for software product lines. In: Proc. Int’l Conf. Software Testing, Verification and Validation (ICST), pp. 41–50. IEEE (2014)Google Scholar
  55. 55.
    Sánchez, A., Segura, S., Parejo, J., Ruiz-Cortés, A.: Variability testing in the wild: the drupal case study. Softw. Syst. Model. 1–22 (2015)Google Scholar
  56. 56.
    Shi, J., Cohen, M.B., Dwyer, M.B.: Integration testing of software product lines using compositional symbolic execution. In: Proc. Int’l Conf. Fundamental Approaches to Software Engineering (FASE), pp. 270–284. Springer (2012)Google Scholar
  57. 57.
    Thüm, T., Kästner, C., Erdweg, S., Siegmund, N.: Abstract features in feature modeling. In: Proc. Int’l Software Product Line Conf. (SPLC), pp. 191–200. IEEE (2011)Google Scholar
  58. 58.
    Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6:1–6:45 (2014)CrossRefGoogle Scholar
  59. 59.
    Thüm, T., Kästner, C., Benduhn, F., Meinicke, J., Saake, G., Leich, T.: FeatureIDE: an extensible framework for feature-oriented software development. Sci. Comput. Program. (SCP) 79, 70–85 (2014)CrossRefGoogle Scholar
  60. 60.
    Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc., Burlington (2007)Google Scholar
  61. 61.
    Visser, W., Havelund, K., Brat, G.P., Park, S.: Model checking programs. In: Proc. Int’l Conf. Automated Software Engineering (ASE), pp. 3–12. Springer (2000)Google Scholar
  62. 62.
    Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Time-aware test suite prioritization. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), pp. 1–12. ACM (2006)Google Scholar
  63. 63.
    Weiss, D.M.: The product line hall of fame. In: Proc. Int’l Software Product Line Conf. (SPLC), p. 395. IEEE (2008)Google Scholar
  64. 64.
    Yoo, S., Harman, N.: Regression testing minimization, selection and prioritization: a survey. Softw. Test. Verif. Reliab. (STVR) 22(2), 67–120 (2012)CrossRefGoogle Scholar
  65. 65.
    Yoo, S., Harman, M., Tonella, P., Susi, A.: Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In: Proc. Int’l Symposium in Software Testing and Analysis (ISSTA), ISSTA’09, pp. 201–212. ACM (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Mustafa Al-Hajjaji
    • 1
  • Thomas Thüm
    • 2
  • Malte Lochau
    • 3
  • Jens Meinicke
    • 4
  • Gunter Saake
    • 1
  1. 1.University of MagdeburgMagdeburgGermany
  2. 2.TU BraunschweigBraunschweigGermany
  3. 3.TU DarmstadtDarmstadtGermany
  4. 4.METOP GmbH, University of MagdeburgMagdeburgGermany

Personalised recommendations