Skip to main content
SpringerLink
Log in

Dynamic test prioritization of product lines: An application on configurable simulation models

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Product line testing is challenging due to the potentially huge number of configurations. Several approaches have tackled this challenge; most of them focused on reducing the number of tested products by selecting a representative subset. However, little attention has been paid to product line test optimization using test results, while tests are executed. This paper aims at optimizing the testing process of product lines by increasing the fault detection rate. To this end we propose a dynamic test prioritization approach. In contrast to traditional static test prioritization, our dynamic test prioritization leverages information of tests being executed in specific products. Processing this information, the initially prioritized tests are rearranged in order to find non-discovered faults. The proposed approach is valid for any kind of product lines, but we have adapted it to the context of configurable simulation models, an area where testing is especially time-consuming and optimization methods are paramount. The approach was empirically evaluated by employing two case studies. The results of this evaluation reveal that the proposed test prioritization approach improves both the static prioritization algorithm and the selected baseline technique. The results provide a basis for suggesting that the proposed dynamic test prioritization approach is appropriate to optimize the testing process of product lines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

For the sake of replicability, experimental material is available in: https://github.com/umarkiegi/DynamicSPLPrioritization.

Notes

  1. Note that PS and TS consist of a generalization of the definitions proposed in section 2.4 of SMPS and \(TS_{150}\%\), respectively, which are redefined in the generic product line engineering context of the approach described in this section.

  2. In our study, all generated test cases can be executed with all derived products. Refer to sub-section 5.2 for details of the product and test case generation.

  3. For this study, we have employed a similarity-based criterion to prioritize pairs statically, based on weighted measures that combines both product similarity and test cases similarity of valid pairs. Detailed explanation it is provided in Sect. 3.2.1

  4. We have implemented the prototype in MATLAB due to its coupling with Simulink, which was the modeling tool used for the simulation of the employed models.

  5. ICPL is a recursive acronym that stands for “ICPL Covering array generation algorithm for Product Lines”

  6. To make the box plots of a reasonable size, the results for the dynamic test prioritization algorithms are shown as minimum (i.e., worst configuration), average, and maximum (i.e., best configuration) values for each test scenario

References

  • Abdessalem, R. B., Nejati, S., Briand, L. C., & Stifter, T. (2018). Testing vision-based control systems using learnable evolutionary algorithms. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE), IEEE, pp. 1016–1026.

  • Acher, M., Lopez-Herrejon, R. E., & Rabiser, R. (2017). Teaching software product lines: A snapshot of current practices and challenges. ACM Transactions on Computing Education, 18, 1, 2:1–2:31.

  • Al-Hajjaji, M., Thüm, T., Lochau, M., Meinicke, J., & Saake, G. (2016). Effective product-line testing using similarity-based product prioritization. Software & Systems Modeling, 1–23.

  • Al-Hajjaji, M., Thüm, T., Meinicke, J., Lochau, M., & Saake, G. (2014). Similarity-based prioritization in software product-line testing. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (New York, NY, USA), SPLC ’14, ACM, pp. 197–206.

  • Apel, S., Batory, D., Kstner, C., & Saake, G. (2016). Feature-Oriented Software Product Lines: Concepts and Implementation (1st ed.). Incorporated: Springer Publishing Company.

    Google Scholar 

  • Arcuri, A., Iqbal, M. Z., & Briand, L. (2010). Black-box system testing of real-time embedded systems using random and search-based testing. In Proceedings of the 22Nd IFIP WG 6.1 International Conference on Testing Software and Systems (Berlin, Heidelberg), ICTSS’10, Springer-Verlag, pp. 95–110.

  • Arrieta, A., Wang, S., Arruabarrena, A., Markiegi, U., Sagardui, G., & Etxeberria, L. (2018). Multi-objective black-box test case selection for cost-effectively testing simulation models. In Proceedings of the Genetic and Evolutionary Computation Conference (New York, NY, USA), GECCO ’18, ACM, pp. 1411–1418.

  • Arrieta, A., Wang, S., Sagardui, G., & Etxeberria, L. (2016a). Search-based test case selection of cyber-physical system product lines for simulation-based validation. In Proceedings of the 20th International Systems and Software Product Line Conference, pp. 297–306.

  • Arrieta, A., Wang, S., Sagardui, G., & Etxeberria, L. (2016b). Test case prioritization of configurable cyber-physical systems with weight-based search algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference 2016 (New York, NY, USA), GECCO ’16, ACM, pp. 1053–1060.

  • Arrieta, A., Wang, S., Markiegi, U., Arruabarrena, A., Etxeberria, L., & Sagardui, G. (2019a). Pareto efficient multi-objective black-box test case selection for simulation-based testing. Information & Software Technology, 114, 137–154.

    Article  Google Scholar 

  • Arrieta, A., Wang, S., Sagardui, G., & Etxeberria, L. (2019b). Search-based test case prioritization for simulation-based testing of cyber-physical system product lines. Journal of Systems and Software, 149, 1–34.

    Article  Google Scholar 

  • Asadollah, S. A., Inam, R., & Hansson, H. (2015). A survey on testing for cyber physical system. In IFIP International Conference on Testing Software and Systems, Springer, pp. 194–207.

  • Baller, H., Lity, S., Lochau, M., & Schaefer, I. (2014). Multi-objective test suite optimization for incremental product family testing. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, pp. 303–312.

  • Batory, D. (2005). Feature models, grammars, and propositional formulas. In Proceedings of the 9th International Conference on Software Product Lines (Berlin, Heidelberg), SPLC’05, Springer-Verlag, pp. 7–20.

  • Benavides, D., Segura, S., & Ruiz-Cortés, A. (2010). Automated analysis of feature models 20 years later: A literature review. Information Systems, 35(6), 615–636.

    Article  Google Scholar 

  • Berger, T., Rublack, R., Nair, D., Atlee, J. M., Becker, M., Czarnecki, K., & Wasowski, A. (2013). A survey of variability modeling in industrial practice. In Variability Modelling of Software-intensive Systems (VaMoS), pp. 7:1–7:8.

  • Berger, T., She, S., Lotufo, R., Wasowski, A., & Czarnecki, K. (2013). A study of variability models and languages in the systems software domain. IEEE Transactions on Software Engineering, 39(12), 1611–1640.

    Article  Google Scholar 

  • Briand, L., Nejati, S., Sabetzadeh, M., & Bianculli, D. (2016). Testing the untestable: Model testing of complex software-intensive systems. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE ’16, ACM, pp. 789–792.

  • Busjaeger, B., & Xie, T. (2016). Learning for test prioritization: an industrial case study. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, pp. 975–980.

  • Catal, C., & Mishra, D. (2013). Test case prioritization: A systematic mapping study. Software Quality Journal, 21(3), 445–478.

    Article  Google Scholar 

  • Chowdhury, S. A., Mohian, S., Mehra, S., Gawsane, S., Johnson, T. T., & Csallner, C. (2018). Automatically finding bugs in a commercial cyber-physical system development tool chain with slforge. In Proceedings of the 40th International Conference on Software Engineering (New York, NY, USA), ICSE ’18, ACM, pp. 981–992.

  • Clements, P., & Northrop, L. (2001). Software Product Lines: Practices and Patterns. SEI Series in Software Engineering. Addison–Wesley.

  • Cohen, M. B., Dwyer, M. B., & Shi, J. (2008). Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering, 34(5), 633–650.

    Article  Google Scholar 

  • Devroey, X., Perrouin, G., Cordy, M., Samih, H., Legay, A., Schobbens, P.-Y., & Heymans, P. (2015). Statistical prioritization for software product line testing: an experience report. Software & Systems Modeling, 1–19.

  • Devroey, X., Perrouin, G., Cordy, M., Schobbens, P.-Y., Legay, A., & Heymans, P. (2014). Towards statistical prioritization for software product lines testing. In Proceedings of the Eight International Workshop on Variability Modelling of Software-Intensive Systems, ACM, p. 10.

  • Di Nardo, D., Alshahwan, N., Briand, L., & Labiche, Y. (2013). Coverage-based test case prioritisation: An industrial case study. In Sixth International Conference on Software Testing, Verification and Validation (Los Alamitos, CA, USA), pp. 302 – 11.

  • do Carmo Machado, I., McGregor, J. D., Cavalcanti, Y. C., & De Almeida, E. S. (2014). On strategies for testing software product lines: A systematic literature review. Information and Software Technology 56, 10, 1183–1199.

  • Elbaum, S., Rothermel, G., & Penix, J. (2014). Techniques for improving regression testing in continuous integration development environments. In Proceedings of the 22Nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14), ACM, pp. 235–245.

  • Elbaum, S., Malishevsky, A. G., & Rothermel, G. (2002). Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering, 28(2), 159–182.

    Article  Google Scholar 

  • Engström, E., Runeson, P., & Ljung, A. (2011). Improving regression testing transparency and efficiency with history-based prioritization–an industrial case study. In 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation (2011), IEEE, pp. 367–376.

  • Engström, E., & Runeson, P. (2011). Software product line testing-a systematic mapping study. Information and Software Technology, 53(1), 2–13.

    Article  Google Scholar 

  • Ensan, A., Bagheri, E., Asadi, M., Gasevic, D., & Biletskiy, Y. (2011). Goal-oriented test case selection and prioritization for product line feature models. In Proceedings of the 2011 Eight International Conference on Information Technology: New Generations (Washington, DC, USA), ITNG ’11, IEEE Computer Society, pp. 291–298.

  • Epitropakis, M. G., Yoo, S., Harman, M., & Burke, E. K. (2015). Empirical evaluation of pareto efficient multi-objective regression test case prioritisation. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (New York, NY, USA), ISSTA, ACM, pp. 234–245.

  • Fang, C., Chen, Z., Wu, K., & Zhao, Z. (2014). Similarity-based test case prioritization using ordered sequences of program entities. Software Quality Journal, 22(2), 335–361.

    Article  Google Scholar 

  • Feldt, R., Poulding, S. M., Clark, D., & Yoo, S. (2016). Test set diameter: Quantifying the diversity of sets of test cases. In 2016 IEEE International Conference on Software Testing, Verification and Validation, ICST 2016, Chicago, IL, USA, April 11-15, 2016, pp. 223–233.

  • Hajri, I., Goknil, A., Pastore, F., & Briand, L. C. (2020). Automating system test case classification and prioritization for use case-driven testing in product lines. Empirical Software Engineering, 25(5), 3711–3769.

    Article  Google Scholar 

  • Hanh, L. T. M., & Binh, N. T. (2012). Mutation operators for simulink models. In 2012 Fourth International Conference on Knowledge and Systems Engineering, IEEE, pp. 54–59.

  • Hanh, L. T. M., Binh, N. T., & Tung, K. T. (2016). A novel fitness function of metaheuristic algorithms for test data generation for simulink models based on mutation analysis. Journal of Systems and Software 120, C, 17–30.

  • Hao, D., Zhang, L., Zang, L., Wang, Y., Wu, X., & Xie, T. (2016). To be optimal or not in test-case prioritization. IEEE Transactions on Software Engineering, 42(5), 490–505.

    Article  Google Scholar 

  • Hemmati, H., Arcuri, A., & Briand, L. (2013). Achieving scalable model-based testing through test case diversity. ACM Transactions on Software Engineering and Methodology 22, 1, 6:1–6:42.

  • Hemmati, H., Fang, Z., & Mäntylä, M. V. (2015). Prioritizing manual test cases in traditional and rapid release environments. In Proceedings of the 8th International Conference on Software Testing, Verification and Validation (ICST’15), pp. 1–10.

  • Henard, C., Papadakis, M., Harman, M., Jia, Y., & Le Traon, Y. (2016). Comparing white-box and black-box test prioritization. In Proceedings of the 38th International Conference on Software Engineering, ACM, pp. 523–534.

  • Henard, C., Papadakis, M., Harmany, M., & Traon, Y. L. (2015). Combining multi-objective search and constraint solving for configuring large scale software product lines. In 37th International Conference on Software Engineering (ICSE’15), pp. 517–528.

  • Henard, C., Papadakis, M., Perrouin, G., Klein, J., Heymans, P., & Le Traon, Y. (2014). Bypassing the combinatorial explosion: Using similarity to generate and prioritize t-wise test configurations for software product lines. IEEE Transactions on Software Engineering, 40(7), 650–670.

    Article  Google Scholar 

  • Heradio, R., Perez-Morago, H., Fernandez-Amoros, D., Cabrerizo, F. J., & Herrera-Viedma, E. (2016). A bibliometric analysis of 20 years of research on software product lines. Information and Software Technology, 72, 1–15.

    Article  Google Scholar 

  • Huang, Y.-C., Peng, K.-L., & Huang, C.-Y. (2012). A history-based cost-cognizant test case prioritization technique in regression testing. Journal of Systems and Software, 85(3), 626–637.

    Article  MathSciNet  Google Scholar 

  • Jia, Y., & Harman, M. (2011). An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering, 37(5), 649–678.

    Article  Google Scholar 

  • Jiang, B., & Chan, W. (2015). Input-based adaptive randomized test case prioritization: A local beam search approach. Journal of Systems and Software, 105, 91–106.

    Article  Google Scholar 

  • Johansen, M. F., Haugen, Ø., & Fleurey, F. (2011). A survey of empirics of strategies for software product line testing. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, IEEE, pp. 266–269.

  • Johansen, M. F., Haugen, Ø., Fleurey, F., Eldegard, A. G., & Syversen, T. (2012). Generating better partial covering arrays by modeling weights on sub-product lines. In Model Driven Engineering Languages and Systems (Berlin, Heidelberg), R. B. France, J. Kazmeier, R. Breu, and C. Atkinson, Eds., Springer Berlin Heidelberg, pp. 269–284.

  • Jones, J. A., & Harrold, M. J. (2003). Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Transactions on software Engineering, 29(3), 195–209.

    Article  Google Scholar 

  • Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014). Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, pp. 654–665.

  • Kang, K. C., Cohen, S. G., Hess, J. A., Novak, W. E., & Peterson, A. S. (1990). Feature-oriented domain analysis (foda) feasibility study. Tech. rep.: Carnegie-Mellon Univ Pittsburgh Pa Software Engineering Inst.

    Book  Google Scholar 

  • Khalilian, A., Azgomi, M. A., & Fazlalizadeh, Y. (2012). An improved method for test case prioritization by incorporating historical test case data. Science of Computer Programming, 78(1), 93–116.

    Article  Google Scholar 

  • Khatibsyarbini, M., Isa, M. A., Jawawi, D. N. A., & Tumeng, R. (2018). Test case prioritization approaches in regression testing: A systematic literature review. Information and Software Technology, 93, 74–93.

    Article  Google Scholar 

  • Kim, J.-M., & Porter, A. (2002). A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering, ACM, pp. 119–129.

  • Korel, B., Koutsogiannakis, G., & Tahat, L. H. (2008). Application of system models in regression test suite prioritization. In Software Maintenance, 2008. ICSM 2008. IEEE International Conference on, IEEE, pp. 247–256.

  • Korel, B., Tahat, L. H., & Harman, M. (2005). Test prioritization using system models. In Software Maintenance, 2005. ICSM’05. Proceedings of the 21st IEEE International Conference on, IEEE, pp. 559–568.

  • Kuhn, R., Kacker, R., Lei, Y., & Hunter, J. (2009). Combinatorial software testing. Computer, 42, 94–96.

    Article  Google Scholar 

  • Lachmann, R., Lity, S., Lischke, S., Beddig, S., Schulze, S., & Schaefer, I. (2015). Delta-oriented test case prioritization for integration testing of software product lines. In Proceedings of the 19th International Conference on Software Product Line (New York, NY, USA), SPLC ’15, ACM, pp. 81–90.

  • Lackner, H., & Schmidt, M. (2014). Towards the assessment of software product line tests: A mutation system for variable systems. In Proceedings of the 18th International Software Product Line Conference: Companion Volume for Workshops, Demonstrations and Tools - Volume 2 (New York, NY, USA), SPLC ’14, ACM, pp. 62–69.

  • Lamancha, B. P., Usaola, M. P., & Velthius, M. P. (2009). Software product line testing. A Systematic Review. ICSOFT (1), 23–30.

  • Le Thi My Hanh, K. T., & Tung, N. T. B. (2014). Mutation-based test data generation for simulink models using genetic algorithm and simulated annealing. International Journal of Computer and Information Technology 3, 04, 763–771.

  • Ledru, Y., Petrenko, A., Boroday, S., & Mandran, N. (2012). Prioritizing test cases with string distances. Automated Software Engineering, 19(1), 65–95.

    Article  Google Scholar 

  • Lee, J., Kang, S., & Lee, D. (2012). A survey on software product line testing. In Proceedings of the 16th International Software Product Line Conference-Volume 1, ACM, pp. 31–40.

  • Lee, J., & Kang, S. (2020). Jung, P (p. 106272). Test coverage criteria for software product line testing: Systematic literature review. Information and Software Technology.

    Google Scholar 

  • Lin, C.-T., Chen, C.-D., Tsai, C.-S., & Kapfhammer, G. M. (2013). History-based test case prioritization with software version awareness. In 2013 18th International Conference on Engineering of Complex Computer Systems, IEEE, pp. 171–172.

  • Linden, F. J. v. d., Schmid, K., & Rommes, E. (2007). Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering. Springer-Verlag, Berlin, Heidelberg.

  • Liu, B., Lucia, Nejati, S., Briand, L. C., & Bruckmann, T. (2016). Simulink fault localization: an iterative statistical debugging approach. Software Testing, Verification and Reliability 26, 6, 431–459.

  • Liu, B., Lucia, L., Nejati, S., & Briand, L. (2017). Improving fault localization for simulink models using search-based testing and prediction models. In 24th IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2017).

  • Liu, B., Nejati, S., Lucia, & Briand, L. C. (2019). Effective fault localization of automotive simulink models: achieving the trade-off between test oracle effort and fault localization accuracy. Empirical Software Engineering 24, 1, 444–490.

  • Lopez-Herrejon, R. E., Fischer, S., Ramler, R., & Egyed, A. (2015). A first systematic mapping study on combinatorial interaction testing for software product lines. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on, IEEE, pp. 1–10.

  • Lopez-Herrejon, R. E., Linsbauer, L., & Egyed, A. (2015). A systematic mapping study of search-based software engineering for software product lines. Information and Software Technology, 61, 33–51.

    Article  Google Scholar 

  • Luo, Q., Moran, K., & Poshyvanyk, D. (2016). A large-scale empirical comparison of static and dynamic test case prioritization techniques. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, pp. 559–570.

  • Luo, Q., Moran, K., Zhang, L., & Poshyvanyk, D. (2019). How do static and dynamic test case prioritization techniques perform on modern software systems? an extensive study on github projects. IEEE Transactions on Software Engineering, 45(11), 1054–1080.

    Article  Google Scholar 

  • Marijan, D., Gotlieb, A., & Sen, S. (2013). Test case prioritization for continuous regression testing: An industrial case study. In Proceedings of the 2013 IEEE International Conference on Software Maintenance (ICSM’13), IEEE Computer Society, pp. 540–543.

  • Markiegi, U., Arrieta, A., Etxeberria, L., & Sagardui, G. (2019). Test case selection using structural coverage in software product lines for time-budget constrained scenarios. In Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing (New York, NY, USA), SAC ’19, Association for Computing Machinery, pp. 2362–2371.

  • Markiegi, U., Arrieta, A., Etxeberria, L., & Sagardui, G. (2019). White-box and black-box test quality metrics for configurable simulation models. In Proceedings of the 23rd International Systems and Software Product Line Conference, SPLC 2019, Volume B, Paris, France, September 9-13, 2019, ACM, pp. 92:1–92:4.

  • Markiegi, U., Arrieta, A., Sagardui, G., & Etxeberria, L. (2017). Search-based product line fault detection allocating test cases iteratively. In Proceedings of the 21st International Systems and Software Product Line Conference - Volume A (New York, NY, USA), SPLC ’17, ACM, pp. 123–132.

  • Matinnejad, R., Nejati, S., Briand, L. C., & Bruckmann, T. (2016). Automated test suite generation for time-continuous simulink models. In Proceedings of the 38th International Conference on Software Engineering (New York, NY, USA), ICSE ’16, ACM, pp. 595–606.

  • Matinnejad, R., Nejati, S., Briand, L. C., & Bruckmann, T. (2015). Effective test suites for mixed discrete-continuous stateflow controllers. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ACM, pp. 84–95.

  • Matinnejad, R. (2015). Simulink fault patterns. Tech. rep.

  • Matinnejad, R., Nejati, S., Briand, L. C., & Bruckmann, T. (2019). Test generation and test prioritization for simulink models with dynamic behavior. IEEE Transactions on Software Engineering, 45(9), 919–944.

    Article  Google Scholar 

  • Menghi, C., Nejati, S., Briand, L. C., & Parache, Y. I. (2019). Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification.

  • Miranda, B., Cruciani, E., Verdecchia, R., & Bertolino, A. (2018). Fast approaches to scalable similarity-based test case prioritization. In Proceedings of the 40th International Conference on Software Engineering, ACM, pp. 222–232.

  • Neto, P. A. d. M. S., do Carmo Machado, I., McGregor, J. D., De Almeida, E. S., & de Lemos Meira, S. R. (2011). A systematic mapping study of software product lines testing. Information and Software Technology 53, 5, 407–423.

  • Nilsson, R., Offutt, J., & Mellin, J. (2006). Test case generation for mutation-based testing of timeliness. Electronic Notes in Theoretical Computer Science, 164(4), 97–114.

    Article  Google Scholar 

  • Noguchi, T., Washizaki, H., Fukazawa, Y., Sato, A., & Ota, K. (2015). History-based test case prioritization for black box testing using ant colony optimization. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST) (2015), IEEE, pp. 1–2.

  • Noor, T. B., & Hemmati, H. (2015). A similarity-based approach for test case prioritization using historical failure data. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), IEEE, pp. 58–68.

  • Papadakis, M., Jia, Y., Harman, M., & Le Traon, Y. (2015). Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique. In Proceedings of the 37th International Conference on Software Engineering-Volume 1, IEEE Press, pp. 936–946.

  • Parejo, J. A., Sánchez, A. B., Segura, S., Ruiz-Cortés, A., Lopez-Herrejon, R. E., & Egyed, A. (2016). Multi-objective test case prioritization in highly configurable systems: A case study. Journal of Systems and Software.

  • Perrouin, G., Sen, S., Klein, J., Baudry, B., & Le Traon, Y. (2010). Automated and scalable t-wise test case generation strategies for software product lines. In 2010 Third International Conference on Software Testing, Verification and Validation, IEEE, pp. 459–468.

  • Perrouin, G., Oster, S., Sen, S., Klein, J., Baudry, B., & Le Traon, Y. (2012). Pairwise testing for software product lines: Comparison of two approaches. Software Quality Journal, 20(3–4), 605–643.

    Article  Google Scholar 

  • Petrović, G., Ivanković, M., Fraser, G., & Just, R. (2021). Does mutation testing improve testing practices? In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE, pp. 910–921.

  • Pohl, K., Böckle, G., & Linden, F. J. v. d. (2005). Software Product Line Engineering: Foundations, Principles and Techniques. Springer-Verlag New York, Inc., Secaucus, NJ, USA.

  • Polzer, A., Merschen, D., Botterweck, G., Pleuss, A., Thomas, J., Hedenetz, B., & Kowalewski, S. (2012). Managing complexity and variability of a model-based embedded software product line. Innovations and Systems and Software Engineering, 8(1), 35–49.

    Article  Google Scholar 

  • Pradhan, D., Wang, S., Ali, S., Yue, T., & Liaaen, M. (2018). Remap: Using rule mining and multi-objective search for dynamic test case prioritization. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST), IEEE, pp. 46–57.

  • Pradhan, D., Wang, S., Ali, S., Yue, T., & Liaaen, M. (2019). Employing rule mining and multi-objective search for dynamic test case prioritization. Journal of Systems and Software, 153, 86–104.

    Article  Google Scholar 

  • Rogstad, E., Briand, L., & Torkar, R. (2013). Test case selection for black-box regression testing of database applications. Information and Software Technology, 55(10), 1781–1795.

    Article  Google Scholar 

  • Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (1999). Test case prioritization: An empirical study. In: Software Maintenance. (ICSM–99) Proceedings. IEEE International Conference on, IEEE, pp 179–188

  • Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001). Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10), 929–948.

    Article  Google Scholar 

  • Sagardui, G., Etxeberria, L., Agirre, J. A., Arrieta, A., Nicolas, C. F., & Martin, J. M. (2017). A configurable validation environment for refactored embedded software: An application to the vertical transport domain. In 2017 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW), IEEE, pp. 16–19.

  • Sahak, M., Jawawi, D. N., & Halim, S. A. (2017). An experiment of different similarity measures on test case prioritization for software product lines. Journal of Telecommunication, Electronic and Computer Engineering (JTEC) 9, 3-4, 177–185.

  • Sánchez, A. B., Segura, S., & Ruiz-Cortés, A. (2014). A comparison of test case prioritization criteria for software product lines. In IEEE International Conference on Software Testing, Verification, and Validation, pp. 41–50.

  • Sánchez, A. B., Segura, S., Parejo, J. A., & Ruiz-Cortés, A. (2015). Variability testing in the wild: the drupal case study. Software & Systems Modeling, 1–22.

  • Sánchez, A. B., Segura, S., Parejo, J. A., & Ruiz-Cortés, A. (2017). Variability testing in the wild: the drupal case study. Software & Systems Modeling, 16(1), 173–194.

    Article  Google Scholar 

  • Sankaranarayanan, S., & Fainekos, G. (2012). Simulating insulin infusion pump risks by in-silico modeling of the insulin-glucose regulatory system. In International Conference on Computational Methods in Systems Biology, Springer, pp. 322–341.

  • Shin, S. Y., Nejati, S., Sabetzadeh, M., Briand, L. C., & Zimmer, F. (2018). Test case prioritization for acceptance testing of cyber physical systems: A multi-objective search-based approach. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (New York, NY, USA), ISSTA, ACM, pp. 49–60.

  • Shokry, H., & Hinchey, M. (2009). Model-based verification of embedded software. Computer, 42(4), 53–59.

    Article  Google Scholar 

  • Spieker, H., Gotlieb, A., Marijan, D., & Mossige, M. (2017). Reinforcement learning for automatic test case prioritization and selection in continuous integration. In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis, ACM, pp. 12–22.

  • Srikanth, H., Cashman, M., & Cohen, M. B. (2016). Test case prioritization of build acceptance tests for an enterprise cloud application: An industrial case study. Journal of Systems and Software, 119, 122–135.

    Article  Google Scholar 

  • Thomas, S. W., Hemmati, H., Hassan, A. E., & Blostein, D. (2014). Static test case prioritization using topic models. Empirical Software Engineering, 19(1), 182–212.

    Article  Google Scholar 

  • Voelter, M., & Groher, I. (2007). Product line implementation using aspect-oriented and model-driven software development. In 11th International Software Product Line Conference (SPLC 2007), IEEE, pp. 233–242.

  • Walcott, K. R., Soffa, M. L., Kapfhammer, G. M., & Roos, R. S. (2006). Time-aware test suite prioritization. In Proceedings of the 2006 International Symposium on Software Testing and Analysis (New York, NY, USA), ISSTA ’06, ACM, pp. 1–12.

  • Wang, S., Ali, S., & Gotlieb, A. (2013). Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the 2013 Genetic and Evolutionary Computation Conference (Amsterdam, Netherlands), pp. 1493–1500.

  • Wang, S., Ali, S., Gotlieb, A., & Liaaen, M. (2016). A systematic test case selection methodology for product lines: results and insights from an industrial case study. Empirical Software Engineering, 1–37.

  • Wang, S., Buchmann, D., Ali, S., Gotlieb, A., Pradhan, D., & Liaaen, M. (2014). Multi-objective test prioritization in software product line testing: An industrial case study. In Proceedings of the 18th International Software Product Line Conference - Volume 1 (New York, NY, USA), SPLC ’14, ACM, pp. 32–41.

  • Wang, S., Ali, S., & Gotlieb, A. (2015). Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software, 103, 370–391.

    Article  Google Scholar 

  • Wang, S., Gotlieb, A., & Ali, S. (2013). Liaaen, M (pp. 237–253). Automated test case selection using feature model: An industrial case study. In MoDELS.

    Google Scholar 

  • Whittaker, J. A., Arbon, J., & Carollo, J. (2012). How Google tests software. Addison-Wesley.

  • Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software Testing, Verification and Reliability, 22(2), 67–120.

    Article  Google Scholar 

  • Zave, P. (1999). Faq sheet on feature interaction.

  • Zhang, L., Hao, D., Zhang, L., Rothermel, G., & Mei, H. (2013). Bridging the gap between the total and additional test-case prioritization strategies. In Proceedings of the 2013 International Conference on Software Engineering (Piscataway, NJ, USA), ICSE ’13, IEEE Press, pp. 192–201.

Download references

Funding

This work has been partially supported by the Basque Government through the Elkartek program under the DIGITAL project (Grant agreement no.KK/2019-00095). The authors are part of the Software and Systems Engineering research group of Mondragon Unibertsitatea (IT1326-19), supported by the Department of Education, Universities and Research of the Basque Government.

Author information

Authors and Affiliations

  1. Mondragon Unibertsitatea, Mondragon, Spain

    Urtzi Markiegi, Aitor Arrieta, Leire Etxeberria & Goiuria Sagardui

Authors
  1. Urtzi Markiegi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aitor Arrieta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leire Etxeberria
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Goiuria Sagardui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Urtzi Markiegi.

Ethics declarations

Conflicts of interest/Competing interests

Not applicable

Appendix

Appendix

Table 8 Correlation between start-up size and APFD based on Spearman’s rank for IT case study
Full size table
Table 9 Correlation between start-up size and APFD based on Spearman’s rank for CW case study
Full size table
Table 10 Correlation between reallocation size and APFD based on Spearman’s rank for IT case study
Full size table
Table 11 Correlation between reallocation size and APFD based on Spearman’s rank for CW case study
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Markiegi, U., Arrieta, A., Etxeberria, L. et al. Dynamic test prioritization of product lines: An application on configurable simulation models. Software Qual J 29, 943–988 (2021). https://doi.org/10.1007/s11219-021-09571-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-021-09571-0

Keywords

Search

Navigation