Skip to main content
SpringerLink
Log in
Book cover

International Conference on Software Technologies

ICSOFT 2016: Software Technologies pp 68–87Cite as

Software Product Line Test Suite Reduction with Constraint Optimization

  • Conference paper
  • First Online:

  • 480 Accesses

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 743))

Abstract

In many cases, Software Product Line Testing (SPLT) targets only the selection of test cases which cover product features or feature interactions. However, higher testing efficiency can be achieved through the selection of test cases with improved fault-revealing capabilities. By associating each test case a priority-value representing (or aggregating) different criteria, such as importance (in terms of fault discovered in previous test campaigns), duration or cost, it becomes possible to select a feature-covering test suite with improved capabilities. A crucial objective in SPLT then becomes to identify a test suite that optimizes reaching a specific goal (lower test duration or cost), while preserving full feature coverage.

In this article, we revisit this problem with a new approach based on constraint optimization with the NValue and GlobalCardinality constraints and a sophisticated search heuristic. These constraints enforce the coverage of all features through the computation of max flows in a network flow representing the coverage relation. The computed max flows represent possible solutions which are further processed in order to determine the solution that optimizes the given objective function, e.g., the smallest test execution costs.

Our approach is implemented in a tool called Flower/C and experimentally evaluated on both randomly generated instances and standard benchmarks. Comparing Flower/C with MiniSAT+ and Cplex, state-of-the-art tools for constraint optimization, we show that our approach is competitive with both tools on random instances and benchmarks. Our results show that MiniSAT+ is not competitive at all, whereas when the priority-value of each test case is uniformly set to 1, that Flower/C approaches Cplex in performance. We compare four different models of Flower/C, using different global constraints, and the one mixing different constraints shows the best performance with high reduction rates. These results open the door to an industrial adoption of the proposed technology.

This is a preview of subscription content, log in via an institution.

Notes

  1. 1.

    http://people.brunel.ac.uk/~mastjjb/jeb/orlib/scpinfo.html.

References

  1. Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of realistic feature models make combinatorial testing of product lines feasible. In: Whittle, J., Clark, T., Kühne, T. (eds.) MODELS 2011. LNCS, vol. 6981, pp. 638–652. Springer, Heidelberg (2011). doi:10.1007/978-3-642-24485-8_47

    Chapter  Google Scholar 

  2. Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: 17th International Software Product Line Conference, SPLC 2013, pp. 62–71 (2013)

    Google Scholar 

  3. Li, D., Jin, Y., Sahin, C., Clause, J., Halfond, W.G.J.: Integrated energy-directed test suite optimization. In: International Symposium on Software Testing and Analysis, ISSTA 2014, San Jose, CA, USA, 21–26 July 2014, pp. 339–350 (2014)

    Google Scholar 

  4. Stolberg, S.: Enabling agile testing through continuous integration. In: Agile Conference, AGILE 2009, pp. 369–374. IEEE (2009)

    Google Scholar 

  5. Wang, S., Ali, S., Gotlieb, A.: Cost-effective test suite minimization in product lines using search techniques. J. Syst. Softw. 103, 370–391 (2015)

    Article  Google Scholar 

  6. Rothermel, G., Harrold, M.J., Ronne, J., Hong, C.: Empirical studies of test-suite reduction. Soft. Test. Verif. Reliab. 12, 219–249 (2002)

    Article  Google Scholar 

  7. Tallam, S., Gupta, N.: A concept analysis inspired greedy algorithm for test suite minimization. In: 6th Workshop on Program Analysis for Software Tools and Engineering (PASTE 2005), pp. 35–42 (2005)

    Google Scholar 

  8. Jeffrey, D., Gupta, N.: Test suite reduction with selective redundancy. In: 21st International Confernce on Software Maintenance, pp. 549–558 (2005)

    Google Scholar 

  9. Ferrer, J., Kruse, P.M., Chicano, F., Alba, E.: Search based algorithms for test sequence generation in functional testing. Inf. Softw. Technol. 58, 419–432 (2015)

    Article  Google Scholar 

  10. Hsu, H.Y., Orso, A.: MINTS: a general framework and tool for supporting test-suite minimization. In: 31st International Conference on Software Engineering (ICSE 2009), pp. 419–429 (2009)

    Google Scholar 

  11. Chen, Z., Zhang, X., Xu, B.: A degraded ILP approach for test suite reduction. In: 20th International Conference on Software Engineering and Knowledge Engineering (2008)

    Google Scholar 

  12. Campos, J., Riboira, A., Perez, A., Abreu, R.: GZoltar: an eclipse plug-in for testing and debugging. In: IEEE/ACM International Conference on Automated Software Engineering, ASE 2012, pp. 378–381 (2012)

    Google Scholar 

  13. Gotlieb, A., Marijan, D.: FLOWER: optimal test suite reduction as a network maximum flow. In: Proceedings of International Symposium on Software Testing and Analysis (ISSTA 2014), San José, CA, USA (2014)

    Google Scholar 

  14. Chvátal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)

    Article  MathSciNet  Google Scholar 

  15. Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM TOSEM 2, 270–285 (1993)

    Article  Google Scholar 

  16. Offutt, A.J., Pan, J., Voas, J.M.: Procedures for reducing the size of coverage-based test sets. In: 12th International Conference on Testing Computer Software (1995)

    Google Scholar 

  17. Agrawal, H.: Efficient coverage testing using global dominator graphs. In: Workshop on Program Analysis for Software Tools and Engineering (PASTE 1999) (1999)

    Google Scholar 

  18. Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: Genetic and Evolutionary Computation Conference (GECCO 2013), Amsterdam, The Netherlands (2013)

    Google Scholar 

  19. Black, J., Melachrinoudis, E., Kaeli, D.: Bi-criteria models for all-uses test suite reduction. In: 26th International Conference on Software Engineering, pp. 106–115 (2004)

    Google Scholar 

  20. Uzuncaova, E., Khurshid, S., Batory, D.: Incremental test generation for software product lines. IEEE Trans. Soft. Eng. 36, 309–322 (2010)

    Article  Google Scholar 

  21. Hao, D., Zhang, L., Wu, X., Mei, H., Rothermel, G.: On-demand test suite reduction. In: International Conference on Software Engineering, pp. 738–748 (2012)

    Google Scholar 

  22. Mouthuy, S., Deville, Y., Dooms, G.: Global constraint for the set covering problem. In: Journées Francophones de Programmation par Contraintes, pp. 183–192 (2007)

    Google Scholar 

  23. Gotlieb, A., Carlsson, M., Liaeen, M., Marijan, D., Petillon, A.: Automated regression testing using constraint programming. In: Proceedings of Innovative Applications of Artificial Intelligence (IAAI 2016), Phoenix, AZ, USA (2016)

    Google Scholar 

  24. Pachet, F., Roy, P.: Automatic generation of music programs. In: Jaffar, J. (ed.) CP 1999. LNCS, vol. 1713, pp. 331–345. Springer, Heidelberg (1999). doi:10.1007/978-3-540-48085-3_24

    Chapter  Google Scholar 

  25. Régin, J.C.: Generalized arc consistency for global cardinality constraint. In: 13th International Conference on Artificial Intelligence (AAAI 1996), pp. 209–215 (1996)

    Google Scholar 

  26. Quimper, C.-G., López-Ortiz, A., Beek, P., Golynski, A.: Improved algorithms for the global cardinality constraint. In: Wallace, M. (ed.) CP 2004. LNCS, vol. 3258, pp. 542–556. Springer, Heidelberg (2004). doi:10.1007/978-3-540-30201-8_40

    Chapter  MATH  Google Scholar 

  27. Gotlieb, A., Carlsson, M., Marijan, D., Petillon, A.: A new approach to feature-based test suite reduction in software product line testing. In: Maciaszek, L.A., Cardoso, J.S., Ludwig, A., van Sinderen, M., Cabello, E. (eds.) 11th International Joint Conference on Software Technologies (ICSOFT 2016), pp. 48–58. SciTePress (2016)

    Google Scholar 

  28. Rossi, F., Beek, P.V., Walsh, T.: Handbook of Constraint Programming (Foundations of Artificial Intelligence). Elsevier Science Inc., New York (2006)

    MATH  Google Scholar 

  29. Régin, J.C.: Global constraints: a survey. In: van Hentenryck, P., Milano, M. (eds.) Hybrid Optimization, pp. 63–134. Springer, Heidelberg (2011). doi:10.1007/978-1-4419-1644-0_3

    Chapter  Google Scholar 

  30. Carlsson, M., et al.: SICStus Prolog User’s Manual. Swedish Institute of Computer Science. Release 4.3.0 edn. (2014). ISBN 91-630-3648-7

    Google Scholar 

  31. Carlsson, M., Ottosson, G., Carlson, B.: An open-ended finite domain constraint solver. In: Glaser, H., Hartel, P., Kuchen, H. (eds.) PLILP 1997. LNCS, vol. 1292, pp. 191–206. Springer, Heidelberg (1997). doi:10.1007/BFb0033845

    Chapter  Google Scholar 

  32. Beasley, J.E.: An algorithm for set covering problem. Eur. J. Oper. Res. 31, 85–93 (1987)

    Article  MathSciNet  Google Scholar 

  33. Beasley, J.E.: A lagrangian heuristic for set-covering problems. Naval Res. Logist. (NRL) 37, 151–164 (1990)

    Article  MathSciNet  Google Scholar 

  34. Grossman, T., Wool, A.: Computational experience with approximation algorithms for the set covering problem. Eur. J. Oper. Res. 101, 81–92 (1997)

    Article  Google Scholar 

  35. Caprara, A., Fischetti, M., Toth, P.: A heuristic method for the set covering problem. Oper. Res. 47, 730–743 (1999)

    Article  MathSciNet  Google Scholar 

  36. Eén, N., Biere, A.: Effective preprocessing in SAT through variable and clause elimination. In: Bacchus, F., Walsh, T. (eds.) SAT 2005. LNCS, vol. 3569, pp. 61–75. Springer, Heidelberg (2005). doi:10.1007/11499107_5

    Chapter  MATH  Google Scholar 

  37. Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23, 437–444 (1997)

    Article  Google Scholar 

  38. Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30, 418–421 (2004)

    Article  Google Scholar 

  39. Mendonca, M., Wasowski, A., Czarnecki, K.: SAT-based analysis of feature models is easy. In: 13th International Software Product Line Conference, SPLC 2009, pp. 231–240. Carnegie Mellon University, Pittsburgh (2009)

    Google Scholar 

  40. Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Pairwise testing for software product lines: comparison of two approaches. Software Qual. J. 20, 605–643 (2012)

    Article  Google Scholar 

  41. Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In: Bosch, J., Lee, J. (eds.) SPLC 2010. LNCS, vol. 6287, pp. 196–210. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15579-6_14

    Chapter  Google Scholar 

  42. Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating \(t\)-wise covering arrays from large feature models. In: 16th International Software Product Line Conference, SPLC 2012, pp. 46–55. ACM, New York (2012)

    Google Scholar 

Download references

Acknowledgement

We are grateful to Marius Liaeen from Cisco Systems, Norway and Alexandre Petillon for their participation to the discussion and initial work related to the approach described in the article. This work is partly supported by the Research Council of Norway (RCN) through the research-based innovation center Certus, under the SFI program.

Author information

Authors and Affiliations

  1. RISE SICS, P.O. Box 1263, 164 29, Kista, Sweden

    Mats Carlsson

  2. Simula Research Laboratory, P.O. Box 134, 1325, Lysaker, Norway

    Arnaud Gotlieb & Dusica Marijan

Authors
  1. Mats Carlsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arnaud Gotlieb
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dusica Marijan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mats Carlsson .

Editor information

Editors and Affiliations

  1. Edifici Ampliación del Rectorado, Universidad Rey Juan Carlos, Madrid, Spain

    Enrique Cabello

  2. Departamento de Engenharia Informática, Universidade de Coimbra, Coimbra, Portugal

    Jorge Cardoso

  3. Kühne Logistics University – KLU, Hamburg, Germany

    André Ludwig

  4. Wroclaw University of Economics, Wroclaw, Poland

    Leszek A. Maciaszek

  5. Information Systems Group, Enschede, The Netherlands

    Marten van Sinderen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Carlsson, M., Gotlieb, A., Marijan, D. (2017). Software Product Line Test Suite Reduction with Constraint Optimization. In: Cabello, E., Cardoso, J., Ludwig, A., Maciaszek, L., van Sinderen, M. (eds) Software Technologies. ICSOFT 2016. Communications in Computer and Information Science, vol 743. Springer, Cham. https://doi.org/10.1007/978-3-319-62569-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-62569-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-62568-3

  • Online ISBN: 978-3-319-62569-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation