Empirical Software Engineering

, Volume 23, Issue 1, pp 1–51 | Cite as

Genetic Algorithm-based Test Generation for Software Product Line with the Integration of Fault Localization Techniques

  • Xuelin Li
  • W. Eric Wong
  • Ruizhi Gao
  • Linghuan Hu
  • Shigeru Hosono


In response to the highly competitive market and the pressure to cost-effectively release good-quality software, companies have adopted the concept of software product line to reduce development cost. However, testing and debugging of each product, even from the same family, is still done independently. This can be very expensive. To solve this problem, we need to explore how test cases generated for one product can be used for another product. We propose a genetic algorithm-based framework which integrates software fault localization techniques and focuses on reusing test specifications and input values whenever feasible. Case studies using four software product lines and eight fault localization techniques were conducted to demonstrate the effectiveness of our framework. Discussions on factors that may affect the effectiveness of the proposed framework is also presented. Our results indicate that test cases generated in such a way can be easily reused (with appropriate conversion) between different products of the same family and help reduce the overall testing and debugging cost.


Software product line Genetic algorithm Test generation Debugging/fault localization Coverage EXAM score 


  1. Abreu R, Zoeteweij P, Golsteijn R, Van Gemund AJC (2009) A practical evaluation of spectrum-based fault localization. J Syst Softw 82(11):1780–1792CrossRefGoogle Scholar
  2. Andrews JH, Briand LC, Labiche Y (2005) Is mutation an appropriate tool for testing experiments? In: Proceedings of the 27th International Conference on Software Engineering (ICSE ‘05), pp 402–411, St. Louis, USAGoogle Scholar
  3. Ahmed MA, Hermadi I (2008) GA-based multiple paths test data generator. Comp Operat Res 35(10):3107–3124CrossRefGoogle Scholar
  4. Ahmed ZH (2010) Genetic algorithm for the traveling salesman problem using sequential constructive crossover operator. J Biom Bioinformat 3(6):96–105Google Scholar
  5. Beohar H, Varshosaz M, Mousavi MR (2016) Basic behavioral models for software product lines: expressivenss and testing pre-orders. Sci Comput Program 123:42–60CrossRefGoogle Scholar
  6. Bergey JK, Chastek G, Cohen S, Donohoe P, Jones LG, Northrop L (2010) Software product lines: report of the 2010 U.S. Army software product line workshop. Technical report. CMU/SEI-2010-TR-014. Retrieved from http://resources.sei.cmu.edu/asset_files/TechnicalReport/2010_005_001_15236.pdf
  7. Bertolino A, Gnesi S (2003) Pluto: a test methodology for product families, 5th Interantional Workshop on Software Product-Family. Engineering 3014:181–197Google Scholar
  8. Burdek J, Lochau M, Bauregger S, Holzer A, von Rhein A, Apel S, Beyer D (2015) Facilitating Reuse in Multi-goal Test-Suite Generation for Software Product Lines, in Proceedings of the 18th International Conference on Fundamental Approaches to Software Engineering (FASE), pp 84–99, April 11–18, 2015Google Scholar
  9. Chen TY, Kuo F, Liu H, Wong WE (2013) Code coverage of adaptive random testing. IEEE Trans Reliab 62(1):226–237CrossRefGoogle Scholar
  10. Clements P, Northrop L (2001) Software product lines: practices and patterns. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  11. CodeCover (2016) Available via http://codecover.org. Accessed 17 Dec 2016
  12. Do H, Rothermel G (2006) On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans Softw Eng 32(9):733–752CrossRefGoogle Scholar
  13. Dustin E, Garrentt T, Gauf (2009) Why automate? Automated software testing ROI explained, Inform IT, April 2009Google Scholar
  14. Engström E, Runeson P (2011) Software product line testing – a systematic mapping study. Informat Softw Technol 53(1):2–13CrossRefGoogle Scholar
  15. Ensan F, Bagheri E, Gašević D (2012) Evolutionary search-based test generation for software product line feature models. In: Proceedings of the 24th International Conference on Advanced Information Systems Engineering, pp 613–628, June 2012Google Scholar
  16. Everitt BS (1977) The analysis of contingency tables. Chapman & Hall, Ltd., LondonGoogle Scholar
  17. Fischer S, Lopez-Herrejon RR, Ramler R, Egyed A (2016) A preliminary empirical assessment of similarity for combinatorial interaction testing of software product lines. In: Proceedings of the 9th International Workshop on Search-Based Software Testing (SBST), pp 15–18, May 16–17, 2016Google Scholar
  18. Forgy EW (1965) Cluster analysis of multivariate data: efficiency versus interpretability of classifications. Biometrics 21:768–769Google Scholar
  19. Freeman D (1987) Applied categorical data analysis. Marcel Dekker, Inc., New YorkGoogle Scholar
  20. Gao R, Wong WE, Chen Z, Wang Y (2015) Effective software fault localization using predicated execution results. Softw Qual J. doi: 10.1007/s11219-015-9295-1, online since November 11, 2015Google Scholar
  21. Goel AL (1985) Software reliability models: assumptions, limitations and applicability. IEEE Trans Softw Eng 11(12):1411–1423CrossRefGoogle Scholar
  22. Goodman LA, Duncan OD (1984) The analysis of cross-classification data having ordered categories. Harvard University Press, CambridgeGoogle Scholar
  23. Guo J, White J, Wang G, Li J, Wang Y (2011) a genetic algorithm for optimized feature selection with resource constraints in software product lines. J Syst Softw 84(12):2208–2221Google Scholar
  24. Hall RJ (2015) Fundamental nonmodularity in electronic mail. Automat Softw Eng 12(1):41–79CrossRefGoogle Scholar
  25. Harman M, Jia Y, Krinke J, Langdon WB, Petke J, Zhang Y (2014) Search based software engineering for software product line engineering: a survey and directions for future work. In: Proceedings of the 18th International Software Product Line Conference, Vol. 1, pp 5–18, September 2014, Florence, ItalyGoogle Scholar
  26. Hartigan JA, Wong MA (1979) Algorithm AS 136: a K-Means clustering algorithm. Appl Stat 28(1):100–108CrossRefMATHGoogle Scholar
  27. Henard C, Papadakis M, Perrouin G, Klein J, Traon YL (2013) Towards automated testing and fixing of re-engineered feature models. In: Proceedings of the 35th International Conference on Software Engineering, pp 1245–1248Google Scholar
  28. Henard C, Papadakis M, Perrouin G, Klein J, Heymans P, Le Traon Y (2014) Bypassing the combinatorial explosion: using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Trans Softw Eng 40(7):650–670Google Scholar
  29. Henard C, Papadakis M, Traon YL (2014) Mutation-based generation of software product line test configurations. In: Proceedings of the 6th Symposium on Search Based Software Engineering, pp 92–106, August 2014Google Scholar
  30. Jones JA, Harrold MJ (2005) Empirical evaluation of the tarantula automatic fault-localization technique. In: Proceedings of the 20th IEEE/ACM Conference on Automated Software Engineering, pp 273–282, Long Beach, California, USA, December, 2005Google Scholar
  31. Kakarontzas G, Stamelos I, Katsaros P (2008) Product Line variability with elastic components and test-driven development. In: Proceedings of the 5th IEEE International Conference on Computational Intelligence for Modeling Control and Automation, Vienna, Austria, pp 146–151, DecemberGoogle Scholar
  32. Lee K, Kang KC, Lee J (2002) Concepts and guidelines of feature modeling for product line software engineering. In: Proceedings of the 7th International Conference on Software Reuse: Methods, Techniques, and Tools. ICSR-7. pp 62–77, London, UK, 2002Google Scholar
  33. Liu C, Fei L, Yan X, Han J, Midkiff SP (2006) Statistical debugging: a hypothesis testing-based approach. IEEE Trans Softw Eng 32(10):831–848CrossRefGoogle Scholar
  34. Lopez-Herrejon RE, Fischer S, Ramler R, Egyed A (2015) A first systematic mapping study on combinatorial interaction testing for software product lines. In: Companion of the Eighth IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2015), pp 1–10, Graz, Austria, April 13–17, 2015Google Scholar
  35. MacQueen J (1967) Some methods for classification and analysis of multivariate observations. In: Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability. vol. 1, pp 281–297, June, 1967Google Scholar
  36. Mathur AP (2008) Foundation of software testing. Addison-Wesley Professional, IndianapolisGoogle Scholar
  37. Mohamed Ali M, Moawad R (2010) An approach for requirements based software product line testing. The 7th International Conference on Informatics and Systems, pp 1–10, March 2010Google Scholar
  38. McGregor JD (2001) Testing a software product line. Technical Report, CMU/SEI-2001-TR-022, December 2001Google Scholar
  39. Michael CC, McGraw G, Schatz MA (2001) Generating software test data by evolution. IEEE Trans Softw Eng 27(12):1085–1110CrossRefGoogle Scholar
  40. Mitchell M (1998) An introduction to genetic algorithms. MIT Press, CambridgeGoogle Scholar
  41. Namin AS, Andrews JH, Labiche Y (2006) Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans Softw Eng 32(8):608–624CrossRefGoogle Scholar
  42. Naish L, Lee HJ, Ramamohanarao K (2011) A model for spectra-based software diagnosis. ACM Trans Softw Eng Method 20(3):11:1–11:32Google Scholar
  43. Nebut C, Pickin S, Le Traon Y, Jezequel J (2003) Automated requirements-based generation of test cases for product families. In: Proceedings of 18th IEEE International Conference on Automated Software Engineering, pp 263–266, October 2003Google Scholar
  44. NIST Report (2002) Software errors cost U.S. Economy $59.5 Billion Annually, NIST Planning Report 02-3, May 2002Google Scholar
  45. Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Method 5(2):99–118CrossRefGoogle Scholar
  46. Perrouin G, Oster S, Sen S, Klein J, Baudry B, le Traon Y (2012) Pairwise testing for software product lines: comparison of two approaches. Softw Qual J 22(3–4):605–643CrossRefGoogle Scholar
  47. Perrouin G, Sen S, Klein J, Baudry B, le Traon Y (2010) Automated and scalable t-wise test case generation strategies for software product lines. In: Proceedings of International Conference on Software Testing, Verification and Validation, pp 459–468, Paris, France, April 2010Google Scholar
  48. Pohl K, Böckle G, Linden FJ (2005) Software product line engineering: foundations, principles, and techniques. Springer, HeidelbergCrossRefMATHGoogle Scholar
  49. Pohl K, Metzger A (2006) Software product line testing. Commun ACM 49(12):78–81CrossRefGoogle Scholar
  50. Reis S, Metzger A, Pohl K (2006) A reuse technique for performance testing of software product lines. In: Proceedings of the 11th International Software Product Line Conference, pp 5–10, Baltimore, MD, USA, August 2006Google Scholar
  51. Sinha S, Dasch T, Ruf R (2011) Governance and cost reduction through multi-tier preventive performance tests in a large-scale product line development. In: Proceedings of 15th IEEE International Software Product Line Conference, pp 295–302, Munich, Germany, August 2011Google Scholar
  52. Sivanandam SN, Deepa SN (2008) Introduction to genetic algorithms. Springer, HeidelbergMATHGoogle Scholar
  53. Telcordia Technologies (formerly Bellcore) (1998) the user’s manual for the χSuds Toolsuite. Available via http://www.utdallas.edu/~ewong/SE6367/01-Project/xsuds-user-manual.pdf. Accessed 17 Dec 2016
  54. Tijms H (2004) Understanding probability: chance rules in everyday life. Cambridge University Press, CambridgeGoogle Scholar
  55. Vessy I (1985) Expertise in debugging computer programs: a process analysis. Int J Man Machine Stud 23(5):459–494CrossRefGoogle Scholar
  56. Wang S, Ali S, Gotlieb A (2015) Cost-effective test suite minimization in product lines using search techniques. J Syst Softw 103:370–391CrossRefGoogle Scholar
  57. Wang S, Gotlieb A, Ali S, Liaaen M (2012) Automated selection of test cases using feature model: an industrial case study. Technical Report (2012-20), Simula Research Laboratory, 2012Google Scholar
  58. Wong WE, Debroy V, Choi B (2010) A family of code coverage-based heuristics for effective fault localization. J Syst Softw 83(2):188–208CrossRefGoogle Scholar
  59. Wong WE, Debroy V, Li Y, Gao R (2012) Software fault localization using DStar (D*). In: Proceedings of The 6th IEEE International Conference on Software Security and Reliability (SERE), pp 21–30, Washington D.C., June, 2012Google Scholar
  60. Wong WE, Debroy V, Li Y, Gao R (2014) The DStar method for effective software fault localization. IEEE Trans Reliab 62(4):290–308CrossRefGoogle Scholar
  61. Wong WE, Debroy V, Xu D (2012b) Towards better fault localization: a crosstab-based statistical approach. IEEE Trans Syst, Man, Cybernet − Part C 42(3):378–396CrossRefGoogle Scholar
  62. Wong WE, Mathur AP (1995a) Fault detection effectiveness of mutation and data flow testing. Softw Qual J 4(1):69–83CrossRefGoogle Scholar
  63. Wong WE, Mathur AP (1995b) Reducing the cost of mutation testing: an empirical study. J Syst Softw 31(3):185–196CrossRefGoogle Scholar
  64. Wong WE, Qi Y, Zhao L, Cai KY (2007) Effective fault localization using code coverage. In: Proceedings of the 31st Annual International Computer Software and Applications Conference (COMPSAC), pp 449–456, Beijing, China, July, 2007Google Scholar
  65. Xie X, Chen TY, Kuo FC, Xu B (2013) A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Trans Softw Eng Method 22(4). doi: 10.1145/2522920.2522924
  66. Xie X, Wong WE, Chen TY, Xu B (2013b) Metamorphic slice: an application in spectrum-based fault localization. Informat Softw Technol 55(5):866–879CrossRefGoogle Scholar
  67. Xu Z, Cohen MB, Motycka W, Rothermel G (2013) Continuous test suite augmentation in software product lines. In: Proceedings of the 17th International Software Product Line Conference, pp 52–61, Tokyo, Japan, August, 2013Google Scholar
  68. Yu Y, Jones JA, Harrold MJ (2008) An empirical study on the effects of test-suite reduction on fault localization. In: Proceedings of the International Conference on Software Engineering (ICSE), pp 201–210, Leipzig, Germany, May, 2008Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of Texas at DallasRichardsonUSA
  2. 2.Service Business Development DivisionNEC CorporationTokyoJapan

Personalised recommendations