Abstract
Because of economical, technological and marketing reasons today’s software systems are more frequently being built as families where each product variant implements a different combination of features. Software families are commonly called Software Product Lines (SPLs) and over the past three decades have been the subject of extensive research and application. Among the benefits of SPLs are: increased software reuse, faster and easier product customization, and reduced time to market. However, testing SPLs is specially challenging as the number of product variants is usually large making it infeasible to test every single variant. In recent years there has been an increasing interest in applying evolutionary computation techniques for SPL testing. In this chapter, we provide a concise overview of the state of the art and practice in SPL testing with evolutionary techniques as well as to highlight open questions and areas for future research.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
An early version is available in [24].
- 3.
- 4.
- 5.
For notational brevity we omit on the vector function and the objective vectors the T that denotes the transpose on vectors.
- 6.
Available at URL: http://minisat.se/MiniSat+.html.
- 7.
An early version is available in [24].
- 8.
- 9.
References
Pohl, K., Bockle, G., van der Linden, F.J.: Software Product Line Engineering: Foundations, Principles and Techniques. Springer, Berlin (2005)
Batory, D.S., Sarvela, J.N., Rauschmayer, A.: Scaling step-wise refinement. IEEE Trans. Softw. Eng. 30(6), 355–371 (2004)
van der Linden, F., Schmid, K., Rommes, E.: Software Product Lines in Action—The Best Industrial Practice in Product Line Engineering. Springer, Berlin (2007)
Engström, E., Runeson, P.: Software product line testing—A systematic mapping study. Inf. Softw. Technol. 53(1), 2–13 (2011)
da Mota Silveira Neto, P.A., do Carmo Machado, I., McGregor, J.D., de Almeida, E.S., de Lemos Meira, S.R.: A systematic mapping study of software product lines testing. Inf. Softw. Technol. 53(5), 407–423 (2011)
Lee, J., Kang, S., Lee, D.: A survey on software product line testing. 16th International Software Product Line Conference, pp. 31–40 (2012)
do Carmo Machado, I., McGregor, J.D., de Almeida, E.S.: Strategies for testing products in software product lines. ACM SIGSOFT Softw. Eng. Notes 37(6), 1–8 (2012)
Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. 45(1), 11 (2012)
Eiben, A., Smith, J.: Introduction to Evolutionary Computing. Springer, Berlin (2003)
McMinn, P.: Search-based software testing: past, present and future. In: ICST Workshops, pp. 153–163. IEEE Computer Society (2011)
Kang, K., Cohen, S., Hess, J., Novak, W., Peterson, A.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon University (1990)
Benavides, D., Segura, S., Cortés, A.R.: Automated analysis of feature models 20 years later: a literature review. Inf. Syst. 35(6), 615–636 (2010)
Lopez-Herrejon, R.E., Batory, D.S.: A standard problem for evaluating product-line methodologies. In: Bosch, J. (ed.) GCSE. Volume 2186 of Lecture Notes in Computer Science, pp. 10–24. Springer, Berlin (2001)
Michalewicz, Z., Fogel, D.B.: How to Solve It: Modern Heuristics, 2nd edn. Springer, Berlin (2010)
Goldberg, D.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading (1989)
da Mota Silveira Neto, P.A., Runeson, P., do Carmo Machado, I., de Almeida, E.S., de Lemos Meira, S.R., Engström, E.: Testing software product lines. IEEE Software 28(5), 16–20 (2011)
Wohlin, C., Runeson, P., da Mota Silveira Neto, P.A., Engström, E., do Carmo Machado, I., de Almeida, E.S.: On the reliability of mapping studies in software engineering. J. Syst. Softw. 86(10), 2594–2610 (2013)
do Carmo Machado, I., McGregor, J.D., Cavalcanti, Y.C., de Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. Inf. Softw. Technol. 56(10), 1183–1199 (2014)
Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)
Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11:1–11:29 (February 2011)
Yilmaz, C., Fouché, S., Cohen, M.B., Porter, A.A., Demiröz, G., Koc, U.: Moving forward with combinatorial interaction testing. IEEE Comput. 47(2), 37–45 (2014)
de Freitas, F.G., de Souza, J.T.: Ten years of search based software engineering: a bibliometric analysis. In: Cohen, M.B., Cinnéide, M.Ó. (eds.) SSBSE. Volume 6956 of Lecture Notes in Computer Science, pp. 18–32. Springer, Berlin (2011)
Lopez-Herrejon, R.E., Linsbauer, L., Egyed, A.: A systematic mapping study of search-based software engineering for software product lines. Inf. Softw. Technol. J. (to appear)
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Linsbauer, L., Egyed, A., Alba, E.: A hitchhiker’s guide to search-based software engineering for software product lines. CoRR abs/1406.2823 (2014)
Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. 16th International Software Product Line Conference, pp. 46–55 (2012)
Lopez-Herrejon, R.E., Egyed, A.: Towards interactive visualization support for pairwise testing software product lines. In: Telea, A., Kerren, A., Marcus, A. (eds.) VISSOFT, pp. 1–4. IEEE (2013)
Ferrer, J., Kruse, P.M., Chicano, J.F., Alba, E.: Evolutionary algorithm for prioritized pairwise test data generation. InL: Soule, T., Moore, J.H. (eds.) GECCO, pp. 1213–1220. ACM (2012)
Durillo, J.J., Nebro, A.J.: jmetal: a java framework for multi-objective optimization. Adv. Eng. Softw. 42(10), 760–771 (2011)
Trinidad, P., Benavides, D., Ruiz-Cortes, A., Segura, S., Jimenez, A.: Fama framework. In: Software Product Line Conference, 2008. SPLC’08. 12th International (Sept.), pp. 359–359
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: Towards a benchmark and a comparison framework for combinatorial interaction testing of software product lines. CoRR abs/1401.5367 (2014)
Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Ralyté, J., Franch, X., Brinkkemper, S., Wrycza, S. (eds.) CAiSE. Volume 7328 of Lecture Notes in Computer Science, pp. 613–628. Springer, Berlin (2012)
Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans. Softw. Eng. 40(7), 650–670 (2014)
Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Pledge: a product line editor and test generation tool. In: SPLC Workshops, pp. 126–129. ACM (2013)
Xu, Z., Cohen, M.B., Motycka, W., Rothermel, G.: Continuous test suite augmentation in software product lines. In: Proceedings SPLC, pp. 52–61 (2013)
Henard, C., Papadakis, M., Traon, Y.L.: Mutation-based generation of software product line test configurations. In: SSBSE, pp. 92–106 (2014)
Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)
Perrouin, G., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Automated and scalable t-wise test case generation strategies for software product lines. In: ICST, pp. 459–468. IEEE Computer Society (2010)
Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In Bosch, J., Lee, J. (eds.) SPLC. Volume 6287 of Lecture Notes in Computer Science, pp. 196–210. Springer, Berlin (2010)
Hervieu, A., Baudry, B., Gotlieb, A.: Pacogen: automatic generation of pairwise test configurations from feature models. In: Dohi, T., Cukic, B. (eds.) ISSRE, pp. 120–129. IEEE (2011)
Lochau, M., Oster, S., Goltz, U., Schürr, A.: Model-based pairwise testing for feature interaction coverage in software product line engineering. Softw. Qual. J. 20(3–4), 567–604 (2012)
Cichos, H., Oster, S., Lochau, M., Schürr, A.: Model-based coverage-driven test suite generation for software product lines. In: Whittle, J., Clark, T., Kühne, T. (eds.) MoDELS. Volume 6981 of Lecture Notes in Computer Science, pp. 425–439. Springer, Berlin (2011)
Calvagna, A., Gargantini, A., Vavassori, P.: Combinatorial testing for feature models using citlab. In: ICST Workshops, pp. 338–347 (2013)
Coello, C.C.: Evolutionary multi-objective optimization website. http://delta.cs.cinvestav.mx/ccoello/EMOO/
Zhang, Y.: Search Based Software Engineering Repository. http://crestweb.cs.ucl.ac.uk/resources/sbse_repository/
Coello, C.C., Lamont, G.B., Veldhuizen, D.A.: Evolutionary Algorithms for Solving Multi-objective Problems, 2nd edn. Genetic and Evolutionary Computation. Springer, Berlin (2007)
Deb, K.: Multi-objective Optimization Using Evolutionary Algorithms, 1st edn. Wiley, New York (June 2001)
Zitzler, E.: Evolutionary multiobjective optimization. In: Handbook of Natural Computing, pp. 871–904 (2012)
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E.: Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2014, Beijing, China, 6–11 July 2014, pp. 387–396. IEEE (2014)
Lopez-Herrejon, R.E., Chicano, J.F., Ferrer, J., Egyed, A., Alba, E.: Multi-objective optimal test suite computation for software product line pairwise testing. In: ICSM, pp. 404–407. IEEE (2013)
Arito, F., Chicano, F., Alba, E.: On the application of sat solvers to the test suite minimization problem. In: Proceedings of the Symposium of Search Based Software Engineering. Volume 7515 of LNCS, pp. 45–59 (2012)
Wolsey, L.A.: Integer Programming. Wiley, New York (1998)
Sutton, A.M., Whitley, L.D., Howe, A.E.: A polynomial time computation of the exact correlation structure of k-satisfiability landscapes. In: Proceedings of GECCO, pp. 365–372 (2009)
Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: GECCO, pp. 1493–1500 (2013)
Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: Proceedings of SPLC, pp. 62–71 (2013)
Marler, R., Arora, J.: Survey of multi-objective optimization methods for engineering. Struct. Multi. Optim. 26(6), 369–395 (2004)
Cruz, J., Neto, P.S., Britto, R., Rabelo, R., Ayala, W., Soares, T., Mota, M.: Toward a hybrid approach to generate software product line portfolios. In: IEEE Congress on Evolutionary Computation, pp. 2229–2236 (2013)
Sayyad, A.S., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: a case study in software product lines. In: Proceedings of ICSE, pp. 492–501 (2013)
Sayyad, A.S., Ingram, J., Menzies, T., Ammar, H.: Scalable product line configuration: a straw to break the camel’s back. In: ASE, pp. 465–474 (2013)
Pascual, G.G., Lopez-Herrejon, R.E., Pinto, M., Fuentes, L., Egyed, A.: Applying multiobjective evolutionary algorithms to dynamic software product lines for reconfiguring mobile applications. J. Syst. Softw. (2015, to appear)
Olaechea, R., Rayside, D., Guo, J., Czarnecki, K.: Comparison of exact and approximate multi-objective optimization for software product lines. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 92–101. Florence, Italy, 15–19 Sept 2014. ACM (2014)
Murashkin, A., Antkiewicz, M., Rayside, D., Czarnecki, K.: Visualization and exploration of optimal variants in product line engineering. In: Proceedings of SPLC, pp. 111–115 (2013)
Dubinsky, Y., Rubin, J., Berger, T., Duszynski, S., Becker, M., Czarnecki, K.: An exploratory study of cloning in industrial software product lines. In: Cleve, A., Ricca, F., Cerioli, M. (eds.) CSMR, pp. 25–34. IEEE Computer Society (2013)
Chen, L., Babar, M.A.: A systematic review of evaluation of variability management approaches in software product lines. Inf. Softw. Technol. 53(4), 344–362 (2011)
Lopez-Herrejon, R.E., Linsbauer, L., Galindo, J.A., Parejo, J.A., Benavides, D., Segura, S., Egyed, A.: An assessment of search-based techniques for reverse engineering feature models. J. Syst. Softw. Spec. Issue Search-Based Softw. Eng. (2015)
Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Feature model synthesis with genetic programming. In: Goues, C.L., Yoo, S. (eds.) Search-Based Software Engineering—6th International Symposium, SSBSE 2014, Fortaleza, Brazil, 26–29 Aug 2014. Proceedings. Volume 8636 of Lecture Notes in Computer Science, pp. 153–167. Springer, Berlin (2014)
She, S., Ryssel, U., Andersen, N., Wasowski, A., Czarnecki, K.: Efficient synthesis of feature models. Inf. Softw. Technol. 56(9), 1122–1143 (2014)
Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., Liaaen, M.: Multi-objective test prioritization in software product line testing: an industrial case study. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 32–41. Florence, Italy, 15–19 Sept 2014. ACM (2014)
Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. 20(3–4), 605–643 (2012)
Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Using feature model knowledge to speed up the generation of covering arrays. In: Gnesi, S., Collet, P., Schmid, K. (eds.) VaMoS, p. 16. ACM (2013)
Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Improving casa runtime performance by exploiting basic feature model analysis. CoRR abs/1311.7313 (2013)
Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6 (2014)
Fischer, S., Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Enhancing clone-and-own with systematic reuse for developing software variants. 30th International Conference on Software Maintenance and Evolution (2014, to appear)
Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: SPLC (1), pp. 46–55 (2012)
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In: Arnold, D.V. (ed.) Genetic and Evolutionary Computation Conference, GECCO’14, Vancouver, BC, Canada, 12–16 July 2014, pp. 1255–1262. ACM (2014)
Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 197–206. Florence, Italy, 15–19 Sept 2014. ACM (2014)
Sánchez, A.B., Segura, S., Cortés, A.R.: A comparison of test case prioritization criteria for software product lines. In: ICST, pp. 41–50 (2014)
Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test., Verif. Reliab. 22(2), 67–120 (2012)
Acknowledgments
This research is partially funded by the Austrian Science Fund (FWF) projects P 25513-N15, P 25289-N15, and Lise Meitner Fellowship M1421-N15, and by the Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2011-28194 and fellowship BES-2012-055967. It is also partially founded by projects 8.06/5.47.4142 (collaboration with the VSB-Tech. Univ. of Ostrava) and 8.06/5.47.4356 (Andalusian Agency of Public Works).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E. (2016). Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges. In: Pedrycz, W., Succi, G., Sillitti, A. (eds) Computational Intelligence and Quantitative Software Engineering. Studies in Computational Intelligence, vol 617. Springer, Cham. https://doi.org/10.1007/978-3-319-25964-2_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-25964-2_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25962-8
Online ISBN: 978-3-319-25964-2
eBook Packages: EngineeringEngineering (R0)