Validation of Constraints Among Configuration Parameters Using Search-Based Combinatorial Interaction Testing

  • Angelo Gargantini
  • Justyna Petke
  • Marco Radavelli
  • Paolo VavassoriEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9962)


The appeal of highly-configurable software systems lies in their adaptability to users’ needs. Search-based Combinatorial Interaction Testing (CIT) techniques have been specifically developed to drive the systematic testing of such highly-configurable systems. In order to apply these, it is paramount to devise a model of parameter configurations which conforms to the software implementation. This is a non-trivial task. Therefore, we extend traditional search-based CIT by devising 4 new testing policies able to check if the model correctly identifies constraints among the various software parameters. Our experiments show that one of our new policies is able to detect faults both in the model and the software implementation that are missed by the standard approaches.


Combinatorial testing Feature models Configurable systems CIT 


  1. 1.
    Arcaini, P., Gargantini, A., Vavassori, P.: Automatic detection and removal of conformance faults in feature models. In: 2016 IEEE 9th International Conference on Software Testing, Verification and Validation (ICST), April 2016Google Scholar
  2. 2.
    Calvagna, A., Gargantini, A.: A formal logic approach to constrained combinatorial testing. J. Autom. Reason. 45(4), 331–358 (2010). SpringerMathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Calvagna, A., Gargantini, A., Vavassori, P.: Combinatorial interaction testing with CitLab. In: Sixth IEEE International Conference on Software Testing, Verification and Validation - Testing Tool Track (2013)Google Scholar
  4. 4.
    Cohen, M., Dwyer, M., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)CrossRefGoogle Scholar
  5. 5.
    Davril, J.-M., Delfosse, E., Hariri, N., Acher, M., Clelang-Huang, J., Heymans, P.: Feature model extraction from large collections of informal product descriptions, 22 Aug 2013Google Scholar
  6. 6.
    Ferreira, J.M., Vergilio, S.R., Quináiaferreia, M.A.: A mutation approach to feature testing of software product lines. In: The 25th International Conference on Software Engineering and Knowledge (SEKE) Engineering, Boston, MA, USA, 27–29 June, pp. 232–237 (2013)Google Scholar
  7. 7.
    Gargantini, A., Vavassori, P.: CitLab: a laboratory for combinatorial interaction testing. In: Workshop on Combinatorial Testing (CT) In conjunction with International Conference on Software Testing, ICST 2012, Montreal, Canada, 17–21 April, pp. 559–568 (2012)Google Scholar
  8. 8.
    Garvin, B.J., Cohen, M.B., Dwyer, M.B.: An improved meta-heuristic search for constrained interaction testing. In: 1st International Symposium on Search Based Software Engineering, pp. 13–22, May 2009Google Scholar
  9. 9.
    Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test Verif. Reliab 15(3), 167–199 (2005)CrossRefGoogle Scholar
  10. 10.
    Harman, M., Jia, Y., Krinke, J., Langdon, W.B., Petke, J., Zhang, Y.: Search based software engineering for software product line engineering: a survey and directions for future work. In: Proceedings of the 18th International Software Product Line Conference (SPLC 2014), pp. 5–18. ACM (2014)Google Scholar
  11. 11.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Towards automated testing and fixing of re-engineered feature models. In: 35th International Conference on Software Engineering, ICSE 2013, San Francisco, CA, USA, 18–26 May, pp. 1245–1248 (2013)Google Scholar
  12. 12.
    Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng. 37(5), 649–678 (2011)CrossRefGoogle Scholar
  13. 13.
    Kuhn, D.R., Okun, V.: Pseudo-exhaustive testing for software. In: 30th Annual IEEE/NASA Software Engineering Workshop (SEW-30 2006), 25–28 April 2006, Loyola College Graduate Center, Columbia, MD, USA, pp. 153–158 (2006)Google Scholar
  14. 14.
    Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng 30(6), 418–421 (2004)CrossRefGoogle Scholar
  15. 15.
    Kuhn, R., Kacker, R., Lei, Y., Hunter, J.: Combinatorial software testing. Computer 42(8), 94–96 (2009)CrossRefGoogle Scholar
  16. 16.
    Lopez-Herrejon, R.E., Galindo, J.A., Benavides, D., Segura, S., Egyed, A.: Reverse engineering feature models with evolutionary algorithms: an exploratory study. In: Fraser, G., Teixeira de Souza, J. (eds.) SSBSE 2012. LNCS, vol. 7515, pp. 168–182. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  17. 17.
    Lopez-Herrejon, R.E., Linsbauer, L., Egyed, A.: A systematic mapping study of search-based software engineering for software product lines. Inf. Softw. Technol. 61, 33–51 (2015)CrossRefGoogle Scholar
  18. 18.
    Nadi, S., Berger, T., Kästner, C., Czarnecki, K.: Mining configuration constraints: static analyses and empirical results. In: Jalote, P., Briand, L.C., van der Hoek, A. (eds.) ICSE, pp. 140–151. ACM (2014)Google Scholar
  19. 19.
    Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv 43(2), 11 (2011)CrossRefzbMATHGoogle Scholar
  20. 20.
    Perrouin, G., Sen, S., Klein, J., Baudry, B., Le Traon, Y.: Automated and scalable t-wise test case generation strategies for software product lines. In: Proceedings of the International Conference on Software Testing (ICST), Paris, France, pp. 459–468. IEEE, April 2010Google Scholar
  21. 21.
    Petke, J., Cohen, M.B., Harman, M., Yoo, S.: Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection. IEEE Trans. Softw. Eng. 41(9), 901–924 (2015)CrossRefGoogle Scholar
  22. 22.
    Segall, I., Tzoref-Brill, R., Farchi, E.: Using binary decision diagrams for combinatorial test design. In: Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 2011, pp. 254–264. ACM (2011)Google Scholar
  23. 23.
    Shi, J., Cohen, M.B., Dwyer, M.B.: Integration testing of software product lines using compositional symbolic execution. In: de Lara, J., Zisman, A. (eds.) Fundamental Approaches to Software Engineering. LNCS, vol. 7212, pp. 270–284. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  24. 24.
    Tartler, R., Lohmann, D., Sincero, J., Schröder-Preikschat, W.: Feature consistency in compile-time-configurable system software: facing the linux 10,000 feature problem. In: Proceedings of the Sixth Conference on Computer Systems, EuroSys 2011, pp. 47–60. ACM, New York (2011)Google Scholar
  25. 25.
    Yilmaz, C., Dumlu, E., Cohen, M.B., Porter, A.A.: Reducing masking effects in combinatorial interaction testing: a feedback driven adaptive approach. IEEE Trans. Softw. Eng. 40(1), 43–66 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Angelo Gargantini
    • 1
  • Justyna Petke
    • 2
  • Marco Radavelli
    • 1
  • Paolo Vavassori
    • 1
    Email author
  1. 1.University of BergamoBergamoItaly
  2. 2.University College LondonLondonUK

Personalised recommendations