Software Product Line Test Suite Reduction with Constraint Optimization

  • Mats CarlssonEmail author
  • Arnaud Gotlieb
  • Dusica Marijan
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 743)


In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

In this article, we revisit this problem with a new approach based on constraint optimization with the NValue and GlobalCardinality constraints and a sophisticated search heuristic. These constraints enforce the coverage of all features through the computation of max flows in a network flow representing the coverage relation. The computed max flows represent possible solutions which are further processed in order to determine the solution that optimizes the given objective function, e.g., the smallest test execution costs.

Our approach is implemented in a tool called Flower/C and experimentally evaluated on both randomly generated instances and standard benchmarks. Comparing Flower/C with MiniSAT+ and Cplex, state-of-the-art tools for constraint optimization, we show that our approach is competitive with both tools on random instances and benchmarks. Our results show that MiniSAT+ is not competitive at all, whereas when the priority-value of each test case is uniformly set to 1, that Flower/C approaches Cplex in performance. We compare four different models of Flower/C, using different global constraints, and the one mixing different constraints shows the best performance with high reduction rates. These results open the door to an industrial adoption of the proposed technology.


Test suite reduction Test suite optimization Software product line testing Feature coverage 



We are grateful to Marius Liaeen from Cisco Systems, Norway and Alexandre Petillon for their participation to the discussion and initial work related to the approach described in the article. This work is partly supported by the Research Council of Norway (RCN) through the research-based innovation center Certus, under the SFI program.


  1. 1.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Whittle, J., Clark, T., Kühne, T. (eds.) MODELS 2011. LNCS, vol. 6981, pp. 638–652. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-24485-8_47CrossRefGoogle Scholar
  2. 2.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: 17th International Software Product Line Conference, SPLC 2013, pp. 62–71 (2013)Google Scholar
  3. 3.
    Li, D., Jin, Y., Sahin, C., Clause, J., Halfond, W.G.J.: Integrated energy-directed test suite optimization. In: International Symposium on Software Testing and Analysis, ISSTA 2014, San Jose, CA, USA, 21–26 July 2014, pp. 339–350 (2014)Google Scholar
  4. 4.
    Stolberg, S.: Enabling agile testing through continuous integration. In: Agile Conference, AGILE 2009, pp. 369–374. IEEE (2009)Google Scholar
  5. 5.
    Wang, S., Ali, S., Gotlieb, A.: Cost-effective test suite minimization in product lines using search techniques. J. Syst. Softw. 103, 370–391 (2015)CrossRefGoogle Scholar
  6. 6.
    Rothermel, G., Harrold, M.J., Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Soft. Test. Verif. Reliab. 12, 219–249 (2002)CrossRefGoogle Scholar
  7. 7.
    Tallam, S., Gupta, N.: A concept analysis inspired greedy algorithm for test suite minimization. In: 6th Workshop on Program Analysis for Software Tools and Engineering (PASTE 2005), pp. 35–42 (2005)Google Scholar
  8. 8.
    Jeffrey, D., Gupta, N.: Test suite reduction with selective redundancy. In: 21st International Confernce on Software Maintenance, pp. 549–558 (2005)Google Scholar
  9. 9.
    Ferrer, J., Kruse, P.M., Chicano, F., Alba, E.: Search based algorithms for test sequence generation in functional testing. Inf. Softw. Technol. 58, 419–432 (2015)CrossRefGoogle Scholar
  10. 10.
    Hsu, H.Y., Orso, A.: MINTS: a general framework and tool for supporting test-suite minimization. In: 31st International Conference on Software Engineering (ICSE 2009), pp. 419–429 (2009)Google Scholar
  11. 11.
    Chen, Z., Zhang, X., Xu, B.: A degraded ILP approach for test suite reduction. In: 20th International Conference on Software Engineering and Knowledge Engineering (2008)Google Scholar
  12. 12.
    Campos, J., Riboira, A., Perez, A., Abreu, R.: GZoltar: an eclipse plug-in for testing and debugging. In: IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pp. 378–381 (2012)Google Scholar
  13. 13.
    Gotlieb, A., Marijan, D.: FLOWER: optimal test suite reduction as a network maximum flow. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA 2014), San José, CA, USA (2014)Google Scholar
  14. 14.
    Chvátal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM TOSEM 2, 270–285 (1993)CrossRefGoogle Scholar
  16. 16.
    Offutt, A.J., Pan, J., Voas, J.M.: Procedures for reducing the size of coverage-based test sets. In: 12th International Conference on Testing Computer Software (1995)Google Scholar
  17. 17.
    Agrawal, H.: Efficient coverage testing using global dominator graphs. In: Workshop on Program Analysis for Software Tools and Engineering (PASTE 1999) (1999)Google Scholar
  18. 18.
    Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: Genetic and Evolutionary Computation Conference (GECCO 2013), Amsterdam, The Netherlands (2013)Google Scholar
  19. 19.
    Black, J., Melachrinoudis, E., Kaeli, D.: Bi-criteria models for all-uses test suite reduction. In: 26th International Conference on Software Engineering, pp. 106–115 (2004)Google Scholar
  20. 20.
    Uzuncaova, E., Khurshid, S., Batory, D.: Incremental test generation for software product lines. IEEE Trans. Soft. Eng. 36, 309–322 (2010)CrossRefGoogle Scholar
  21. 21.
    Hao, D., Zhang, L., Wu, X., Mei, H., Rothermel, G.: On-demand test suite reduction. In: International Conference on Software Engineering, pp. 738–748 (2012)Google Scholar
  22. 22.
    Mouthuy, S., Deville, Y., Dooms, G.: Global constraint for the set covering problem. In: Journées Francophones de Programmation par Contraintes, pp. 183–192 (2007)Google Scholar
  23. 23.
    Gotlieb, A., Carlsson, M., Liaeen, M., Marijan, D., Petillon, A.: Automated regression testing using constraint programming. In: Proceedings of Innovative Applications of Artificial Intelligence (IAAI 2016), Phoenix, AZ, USA (2016)Google Scholar
  24. 24.
    Pachet, F., Roy, P.: Automatic generation of music programs. In: Jaffar, J. (ed.) CP 1999. LNCS, vol. 1713, pp. 331–345. Springer, Heidelberg (1999). doi: 10.1007/978-3-540-48085-3_24CrossRefGoogle Scholar
  25. 25.
    Régin, J.C.: Generalized arc consistency for global cardinality constraint. In: 13th International Conference on Artificial Intelligence (AAAI 1996), pp. 209–215 (1996)Google Scholar
  26. 26.
    Quimper, C.-G., López-Ortiz, A., Beek, P., Golynski, A.: Improved algorithms for the global cardinality constraint. In: Wallace, M. (ed.) CP 2004. LNCS, vol. 3258, pp. 542–556. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-30201-8_40CrossRefzbMATHGoogle Scholar
  27. 27.
    Gotlieb, A., Carlsson, M., Marijan, D., Petillon, A.: A new approach to feature-based test suite reduction in software product line testing. In: Maciaszek, L.A., Cardoso, J.S., Ludwig, A., van Sinderen, M., Cabello, E. (eds.) 11th International Joint Conference on Software Technologies (ICSOFT 2016), pp. 48–58. SciTePress (2016)Google Scholar
  28. 28.
    Rossi, F., Beek, P.V., Walsh, T.: Handbook of Constraint Programming (Foundations of Artificial Intelligence). Elsevier Science Inc., New York (2006)zbMATHGoogle Scholar
  29. 29.
    Régin, J.C.: Global constraints: a survey. In: van Hentenryck, P., Milano, M. (eds.) Hybrid Optimization, pp. 63–134. Springer, Heidelberg (2011). doi: 10.1007/978-1-4419-1644-0_3CrossRefGoogle Scholar
  30. 30.
    Carlsson, M., et al.: SICStus Prolog User’s Manual. Swedish Institute of Computer Science. Release 4.3.0 edn. (2014). ISBN 91-630-3648-7Google Scholar
  31. 31.
    Carlsson, M., Ottosson, G., Carlson, B.: An open-ended finite domain constraint solver. In: Glaser, H., Hartel, P., Kuchen, H. (eds.) PLILP 1997. LNCS, vol. 1292, pp. 191–206. Springer, Heidelberg (1997). doi: 10.1007/BFb0033845CrossRefGoogle Scholar
  32. 32.
    Beasley, J.E.: An algorithm for set covering problem. Eur. J. Oper. Res. 31, 85–93 (1987)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Beasley, J.E.: A lagrangian heuristic for set-covering problems. Naval Res. Logist. (NRL) 37, 151–164 (1990)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Grossman, T., Wool, A.: Computational experience with approximation algorithms for the set covering problem. Eur. J. Oper. Res. 101, 81–92 (1997)CrossRefGoogle Scholar
  35. 35.
    Caprara, A., Fischetti, M., Toth, P.: A heuristic method for the set covering problem. Oper. Res. 47, 730–743 (1999)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Eén, N., Biere, A.: Effective preprocessing in SAT through variable and clause elimination. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 61–75. Springer, Heidelberg (2005). doi: 10.1007/11499107_5CrossRefzbMATHGoogle Scholar
  37. 37.
    Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23, 437–444 (1997)CrossRefGoogle Scholar
  38. 38.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30, 418–421 (2004)CrossRefGoogle Scholar
  39. 39.
    Mendonca, M., Wasowski, A., Czarnecki, K.: SAT-based analysis of feature models is easy. In: 13th International Software Product Line Conference, SPLC 2009, pp. 231–240. Carnegie Mellon University, Pittsburgh (2009)Google Scholar
  40. 40.
    Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Pairwise testing for software product lines: comparison of two approaches. Software Qual. J. 20, 605–643 (2012)CrossRefGoogle Scholar
  41. 41.
    Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Bosch, J., Lee, J. (eds.) SPLC 2010. LNCS, vol. 6287, pp. 196–210. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15579-6_14CrossRefGoogle Scholar
  42. 42.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating \(t\)-wise covering arrays from large feature models. In: 16th International Software Product Line Conference, SPLC 2012, pp. 46–55. ACM, New York (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.RISE SICSKistaSweden
  2. 2.Simula Research LaboratoryLysakerNorway

Personalised recommendations