Advertisement

Evolutionary Computation for Software Product Line Testing: An Overview and Open Challenges

  • Roberto E. Lopez-Herrejon
  • Javier Ferrer
  • Francisco Chicano
  • Alexander Egyed
  • Enrique Alba
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 617)

Abstract

Because of economical, technological and marketing reasons today’s software systems are more frequently being built as families where each product variant implements a different combination of features. Software families are commonly called Software Product Lines (SPLs) and over the past three decades have been the subject of extensive research and application. Among the benefits of SPLs are: increased software reuse, faster and easier product customization, and reduced time to market. However, testing SPLs is specially challenging as the number of product variants is usually large making it infeasible to test every single variant. In recent years there has been an increasing interest in applying evolutionary computation techniques for SPL testing. In this chapter, we provide a concise overview of the state of the art and practice in SPL testing with evolutionary techniques as well as to highlight open questions and areas for future research.

Keywords

Software product lines Product line testing Search based software engineering Feature models Feature set Reverse engineering Variability modeling 

Notes

Acknowledgments

This research is partially funded by the Austrian Science Fund (FWF) projects P 25513-N15, P 25289-N15, and Lise Meitner Fellowship M1421-N15, and by the Spanish Ministry of Economy and Competitiveness and FEDER under contract TIN2011-28194 and fellowship BES-2012-055967. It is also partially founded by projects 8.06/5.47.4142 (collaboration with the VSB-Tech. Univ. of Ostrava) and 8.06/5.47.4356 (Andalusian Agency of Public Works).

References

  1. 1.
    Pohl, K., Bockle, G., van der Linden, F.J.: Software Product Line Engineering: Foundations, Principles and Techniques. Springer, Berlin (2005)CrossRefGoogle Scholar
  2. 2.
    Batory, D.S., Sarvela, J.N., Rauschmayer, A.: Scaling step-wise refinement. IEEE Trans. Softw. Eng. 30(6), 355–371 (2004)CrossRefGoogle Scholar
  3. 3.
    van der Linden, F., Schmid, K., Rommes, E.: Software Product Lines in Action—The Best Industrial Practice in Product Line Engineering. Springer, Berlin (2007)Google Scholar
  4. 4.
    Engström, E., Runeson, P.: Software product line testing—A systematic mapping study. Inf. Softw. Technol. 53(1), 2–13 (2011)CrossRefGoogle Scholar
  5. 5.
    da Mota Silveira Neto, P.A., do Carmo Machado, I., McGregor, J.D., de Almeida, E.S., de Lemos Meira, S.R.: A systematic mapping study of software product lines testing. Inf. Softw. Technol. 53(5), 407–423 (2011)Google Scholar
  6. 6.
    Lee, J., Kang, S., Lee, D.: A survey on software product line testing. 16th International Software Product Line Conference, pp. 31–40 (2012)Google Scholar
  7. 7.
    do Carmo Machado, I., McGregor, J.D., de Almeida, E.S.: Strategies for testing products in software product lines. ACM SIGSOFT Softw. Eng. Notes 37(6), 1–8 (2012)Google Scholar
  8. 8.
    Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. 45(1), 11 (2012)CrossRefGoogle Scholar
  9. 9.
    Eiben, A., Smith, J.: Introduction to Evolutionary Computing. Springer, Berlin (2003)Google Scholar
  10. 10.
    McMinn, P.: Search-based software testing: past, present and future. In: ICST Workshops, pp. 153–163. IEEE Computer Society (2011)Google Scholar
  11. 11.
    Kang, K., Cohen, S., Hess, J., Novak, W., Peterson, A.: Feature-oriented domain analysis (FODA) feasibility study. Technical Report CMU/SEI-90-TR-21, Software Engineering Institute, Carnegie Mellon University (1990)Google Scholar
  12. 12.
    Benavides, D., Segura, S., Cortés, A.R.: Automated analysis of feature models 20 years later: a literature review. Inf. Syst. 35(6), 615–636 (2010)CrossRefGoogle Scholar
  13. 13.
    Lopez-Herrejon, R.E., Batory, D.S.: A standard problem for evaluating product-line methodologies. In: Bosch, J. (ed.) GCSE. Volume 2186 of Lecture Notes in Computer Science, pp. 10–24. Springer, Berlin (2001)Google Scholar
  14. 14.
    Michalewicz, Z., Fogel, D.B.: How to Solve It: Modern Heuristics, 2nd edn. Springer, Berlin (2010)Google Scholar
  15. 15.
    Goldberg, D.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, Reading (1989)Google Scholar
  16. 16.
    da Mota Silveira Neto, P.A., Runeson, P., do Carmo Machado, I., de Almeida, E.S., de Lemos Meira, S.R., Engström, E.: Testing software product lines. IEEE Software 28(5), 16–20 (2011)Google Scholar
  17. 17.
    Wohlin, C., Runeson, P., da Mota Silveira Neto, P.A., Engström, E., do Carmo Machado, I., de Almeida, E.S.: On the reliability of mapping studies in software engineering. J. Syst. Softw. 86(10), 2594–2610 (2013)Google Scholar
  18. 18.
    do Carmo Machado, I., McGregor, J.D., Cavalcanti, Y.C., de Almeida, E.S.: On strategies for testing software product lines: a systematic literature review. Inf. Softw. Technol. 56(10), 1183–1199 (2014)Google Scholar
  19. 19.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)CrossRefGoogle Scholar
  20. 20.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 11:1–11:29 (February 2011)Google Scholar
  21. 21.
    Yilmaz, C., Fouché, S., Cohen, M.B., Porter, A.A., Demiröz, G., Koc, U.: Moving forward with combinatorial interaction testing. IEEE Comput. 47(2), 37–45 (2014)CrossRefGoogle Scholar
  22. 22.
    de Freitas, F.G., de Souza, J.T.: Ten years of search based software engineering: a bibliometric analysis. In: Cohen, M.B., Cinnéide, M.Ó. (eds.) SSBSE. Volume 6956 of Lecture Notes in Computer Science, pp. 18–32. Springer, Berlin (2011)Google Scholar
  23. 23.
    Lopez-Herrejon, R.E., Linsbauer, L., Egyed, A.: A systematic mapping study of search-based software engineering for software product lines. Inf. Softw. Technol. J. (to appear)Google Scholar
  24. 24.
    Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Linsbauer, L., Egyed, A., Alba, E.: A hitchhiker’s guide to search-based software engineering for software product lines. CoRR abs/1406.2823 (2014)Google Scholar
  25. 25.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. 16th International Software Product Line Conference, pp. 46–55 (2012)Google Scholar
  26. 26.
    Lopez-Herrejon, R.E., Egyed, A.: Towards interactive visualization support for pairwise testing software product lines. In: Telea, A., Kerren, A., Marcus, A. (eds.) VISSOFT, pp. 1–4. IEEE (2013)Google Scholar
  27. 27.
    Ferrer, J., Kruse, P.M., Chicano, J.F., Alba, E.: Evolutionary algorithm for prioritized pairwise test data generation. InL: Soule, T., Moore, J.H. (eds.) GECCO, pp. 1213–1220. ACM (2012)Google Scholar
  28. 28.
    Durillo, J.J., Nebro, A.J.: jmetal: a java framework for multi-objective optimization. Adv. Eng. Softw. 42(10), 760–771 (2011)CrossRefGoogle Scholar
  29. 29.
    Trinidad, P., Benavides, D., Ruiz-Cortes, A., Segura, S., Jimenez, A.: Fama framework. In: Software Product Line Conference, 2008. SPLC’08. 12th International (Sept.), pp. 359–359Google Scholar
  30. 30.
    Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: Towards a benchmark and a comparison framework for combinatorial interaction testing of software product lines. CoRR abs/1401.5367 (2014)Google Scholar
  31. 31.
    Ensan, F., Bagheri, E., Gasevic, D.: Evolutionary search-based test generation for software product line feature models. In: Ralyté, J., Franch, X., Brinkkemper, S., Wrycza, S. (eds.) CAiSE. Volume 7328 of Lecture Notes in Computer Science, pp. 613–628. Springer, Berlin (2012)Google Scholar
  32. 32.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., Traon, Y.L.: Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans. Softw. Eng. 40(7), 650–670 (2014)CrossRefGoogle Scholar
  33. 33.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Pledge: a product line editor and test generation tool. In: SPLC Workshops, pp. 126–129. ACM (2013)Google Scholar
  34. 34.
    Xu, Z., Cohen, M.B., Motycka, W., Rothermel, G.: Continuous test suite augmentation in software product lines. In: Proceedings SPLC, pp. 52–61 (2013)Google Scholar
  35. 35.
    Henard, C., Papadakis, M., Traon, Y.L.: Mutation-based generation of software product line test configurations. In: SSBSE, pp. 92–106 (2014)Google Scholar
  36. 36.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Softw. Eng. 16(1), 61–102 (2011)CrossRefGoogle Scholar
  37. 37.
    Perrouin, G., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Automated and scalable t-wise test case generation strategies for software product lines. In: ICST, pp. 459–468. IEEE Computer Society (2010)Google Scholar
  38. 38.
    Oster, S., Markert, F., Ritter, P.: Automated incremental pairwise testing of software product lines. In Bosch, J., Lee, J. (eds.) SPLC. Volume 6287 of Lecture Notes in Computer Science, pp. 196–210. Springer, Berlin (2010)Google Scholar
  39. 39.
    Hervieu, A., Baudry, B., Gotlieb, A.: Pacogen: automatic generation of pairwise test configurations from feature models. In: Dohi, T., Cukic, B. (eds.) ISSRE, pp. 120–129. IEEE (2011)Google Scholar
  40. 40.
    Lochau, M., Oster, S., Goltz, U., Schürr, A.: Model-based pairwise testing for feature interaction coverage in software product line engineering. Softw. Qual. J. 20(3–4), 567–604 (2012)CrossRefGoogle Scholar
  41. 41.
    Cichos, H., Oster, S., Lochau, M., Schürr, A.: Model-based coverage-driven test suite generation for software product lines. In: Whittle, J., Clark, T., Kühne, T. (eds.) MoDELS. Volume 6981 of Lecture Notes in Computer Science, pp. 425–439. Springer, Berlin (2011)Google Scholar
  42. 42.
    Calvagna, A., Gargantini, A., Vavassori, P.: Combinatorial testing for feature models using citlab. In: ICST Workshops, pp. 338–347 (2013)Google Scholar
  43. 43.
    Coello, C.C.: Evolutionary multi-objective optimization website. http://delta.cs.cinvestav.mx/ccoello/EMOO/
  44. 44.
    Zhang, Y.: Search Based Software Engineering Repository. http://crestweb.cs.ucl.ac.uk/resources/sbse_repository/
  45. 45.
    Coello, C.C., Lamont, G.B., Veldhuizen, D.A.: Evolutionary Algorithms for Solving Multi-objective Problems, 2nd edn. Genetic and Evolutionary Computation. Springer, Berlin (2007)Google Scholar
  46. 46.
    Deb, K.: Multi-objective Optimization Using Evolutionary Algorithms, 1st edn. Wiley, New York (June 2001)Google Scholar
  47. 47.
    Zitzler, E.: Evolutionary multiobjective optimization. In: Handbook of Natural Computing, pp. 871–904 (2012)Google Scholar
  48. 48.
    Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Egyed, A., Alba, E.: Comparative analysis of classical multi-objective evolutionary algorithms and seeding strategies for pairwise testing of software product lines. In: Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2014, Beijing, China, 6–11 July 2014, pp. 387–396. IEEE (2014)Google Scholar
  49. 49.
    Lopez-Herrejon, R.E., Chicano, J.F., Ferrer, J., Egyed, A., Alba, E.: Multi-objective optimal test suite computation for software product line pairwise testing. In: ICSM, pp. 404–407. IEEE (2013)Google Scholar
  50. 50.
    Arito, F., Chicano, F., Alba, E.: On the application of sat solvers to the test suite minimization problem. In: Proceedings of the Symposium of Search Based Software Engineering. Volume 7515 of LNCS, pp. 45–59 (2012)Google Scholar
  51. 51.
    Wolsey, L.A.: Integer Programming. Wiley, New York (1998)Google Scholar
  52. 52.
    Sutton, A.M., Whitley, L.D., Howe, A.E.: A polynomial time computation of the exact correlation structure of k-satisfiability landscapes. In: Proceedings of GECCO, pp. 365–372 (2009)Google Scholar
  53. 53.
    Wang, S., Ali, S., Gotlieb, A.: Minimizing test suites in software product lines using weight-based genetic algorithms. In: GECCO, pp. 1493–1500 (2013)Google Scholar
  54. 54.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective test generation for software product lines. In: Proceedings of SPLC, pp. 62–71 (2013)Google Scholar
  55. 55.
    Marler, R., Arora, J.: Survey of multi-objective optimization methods for engineering. Struct. Multi. Optim. 26(6), 369–395 (2004)zbMATHMathSciNetCrossRefGoogle Scholar
  56. 56.
    Cruz, J., Neto, P.S., Britto, R., Rabelo, R., Ayala, W., Soares, T., Mota, M.: Toward a hybrid approach to generate software product line portfolios. In: IEEE Congress on Evolutionary Computation, pp. 2229–2236 (2013)Google Scholar
  57. 57.
    Sayyad, A.S., Menzies, T., Ammar, H.: On the value of user preferences in search-based software engineering: a case study in software product lines. In: Proceedings of ICSE, pp. 492–501 (2013)Google Scholar
  58. 58.
    Sayyad, A.S., Ingram, J., Menzies, T., Ammar, H.: Scalable product line configuration: a straw to break the camel’s back. In: ASE, pp. 465–474 (2013)Google Scholar
  59. 59.
    Pascual, G.G., Lopez-Herrejon, R.E., Pinto, M., Fuentes, L., Egyed, A.: Applying multiobjective evolutionary algorithms to dynamic software product lines for reconfiguring mobile applications. J. Syst. Softw. (2015, to appear)Google Scholar
  60. 60.
    Olaechea, R., Rayside, D., Guo, J., Czarnecki, K.: Comparison of exact and approximate multi-objective optimization for software product lines. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 92–101. Florence, Italy, 15–19 Sept 2014. ACM (2014)Google Scholar
  61. 61.
    Murashkin, A., Antkiewicz, M., Rayside, D., Czarnecki, K.: Visualization and exploration of optimal variants in product line engineering. In: Proceedings of SPLC, pp. 111–115 (2013)Google Scholar
  62. 62.
    Dubinsky, Y., Rubin, J., Berger, T., Duszynski, S., Becker, M., Czarnecki, K.: An exploratory study of cloning in industrial software product lines. In: Cleve, A., Ricca, F., Cerioli, M. (eds.) CSMR, pp. 25–34. IEEE Computer Society (2013)Google Scholar
  63. 63.
    Chen, L., Babar, M.A.: A systematic review of evaluation of variability management approaches in software product lines. Inf. Softw. Technol. 53(4), 344–362 (2011)CrossRefGoogle Scholar
  64. 64.
    Lopez-Herrejon, R.E., Linsbauer, L., Galindo, J.A., Parejo, J.A., Benavides, D., Segura, S., Egyed, A.: An assessment of search-based techniques for reverse engineering feature models. J. Syst. Softw. Spec. Issue Search-Based Softw. Eng. (2015)Google Scholar
  65. 65.
    Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Feature model synthesis with genetic programming. In: Goues, C.L., Yoo, S. (eds.) Search-Based Software Engineering—6th International Symposium, SSBSE 2014, Fortaleza, Brazil, 26–29 Aug 2014. Proceedings. Volume 8636 of Lecture Notes in Computer Science, pp. 153–167. Springer, Berlin (2014)Google Scholar
  66. 66.
    She, S., Ryssel, U., Andersen, N., Wasowski, A., Czarnecki, K.: Efficient synthesis of feature models. Inf. Softw. Technol. 56(9), 1122–1143 (2014)CrossRefGoogle Scholar
  67. 67.
    Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., Liaaen, M.: Multi-objective test prioritization in software product line testing: an industrial case study. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 32–41. Florence, Italy, 15–19 Sept 2014. ACM (2014)Google Scholar
  68. 68.
    Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., Traon, Y.L.: Pairwise testing for software product lines: comparison of two approaches. Softw. Qual. J. 20(3–4), 605–643 (2012)CrossRefGoogle Scholar
  69. 69.
    Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Using feature model knowledge to speed up the generation of covering arrays. In: Gnesi, S., Collet, P., Schmid, K. (eds.) VaMoS, p. 16. ACM (2013)Google Scholar
  70. 70.
    Haslinger, E.N., Lopez-Herrejon, R.E., Egyed, A.: Improving casa runtime performance by exploiting basic feature model analysis. CoRR abs/1311.7313 (2013)Google Scholar
  71. 71.
    Thüm, T., Apel, S., Kästner, C., Schaefer, I., Saake, G.: A classification and survey of analysis strategies for software product lines. ACM Comput. Surv. 47(1), 6 (2014)CrossRefGoogle Scholar
  72. 72.
    Fischer, S., Linsbauer, L., Lopez-Herrejon, R.E., Egyed, A.: Enhancing clone-and-own with systematic reuse for developing software variants. 30th International Conference on Software Maintenance and Evolution (2014, to appear)Google Scholar
  73. 73.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: An algorithm for generating t-wise covering arrays from large feature models. In: SPLC (1), pp. 46–55 (2012)Google Scholar
  74. 74.
    Lopez-Herrejon, R.E., Ferrer, J., Chicano, F., Haslinger, E.N., Egyed, A., Alba, E.: A parallel evolutionary algorithm for prioritized pairwise testing of software product lines. In: Arnold, D.V. (ed.) Genetic and Evolutionary Computation Conference, GECCO’14, Vancouver, BC, Canada, 12–16 July 2014, pp. 1255–1262. ACM (2014)Google Scholar
  75. 75.
    Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., Saake, G.: Similarity-based prioritization in software product-line testing. In: Gnesi, S., Fantechi, A. (eds.) 18th International Software Product Line Conference, SPLC’14, pp. 197–206. Florence, Italy, 15–19 Sept 2014. ACM (2014)Google Scholar
  76. 76.
    Sánchez, A.B., Segura, S., Cortés, A.R.: A comparison of test case prioritization criteria for software product lines. In: ICST, pp. 41–50 (2014)Google Scholar
  77. 77.
    Yoo, S., Harman, M.: Regression testing minimization, selection and prioritization: a survey. Softw. Test., Verif. Reliab. 22(2), 67–120 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Roberto E. Lopez-Herrejon
    • 1
  • Javier Ferrer
    • 2
  • Francisco Chicano
    • 2
  • Alexander Egyed
    • 1
  • Enrique Alba
    • 2
  1. 1.Software Systems Engineering InstituteJohannes Kepler UniversityLinzAustria
  2. 2.Universidad de MálagaAndalucía TechSevillaSpain

Personalised recommendations