Skip to main content
SpringerLink
Log in

State-based models in regression test suite prioritization

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

Testing software products is very expensive and time consuming, especially for large software systems with extensive regression testing. During regression testing, a modified system is often re-tested using an existing test suite. Since test suites can be very large, testers are interested in detecting faults in the modified system as early as possible. Test prioritization tries to order test cases for execution in a way that increases the chances of the early detection of faults. Most of the existing test prioritization methods are based on the code of the system under test, but model-based test prioritization has been lately proposed. Most of the existing model-based test prioritization methods can be used only when models are modified during system maintenance. In this paper, we present model-based prioritization for a class of modifications for which models are not modified (only the source code is modified). After identifying the elements of the model related to the modified source code, information collected during the execution of the model is used to prioritize tests for execution. Here, we present and compare existing and new model-based test prioritization methods focused on this class of modifications. The major motivation for presenting these methods is to provide system developers with simple and yet effective test prioritization techniques for early fault detection. Statistical analysis of the empirical study, which compares the effectiveness of the presented methods in terms of early fault detection, show that compared to random ordering of test cases, model-based test prioritization significantly improve the effectiveness of test prioritization with respect to early fault detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  • Androutsopoulos, K., Clark, D., Harman, M., Li, Z., & Tratt, L. (2009a). Control dependence for extended finite state machines. In Proceedings of the Fundamental Approaches to Software Engineering (FASE ‘09) (Vol. 5503, pp. 216–230). LNCS. York: Springer. Accessed 22nd–29th March 2009.

  • Androutsopoulos, K., Gold, N., Harman, M., Li, Z., & Tratt, L. (2009b). A theoretical and empirical study of EFSM dependence. In Proceedings 25th IEEE international conference on software maintenance (ICSM 2009). Edmonton, AB. 23rd–26th Sept 2009. To appear.

  • Basanieri, F., Bertolino, A., & Marchetti, E. (2002). The cow suite approach to planning and deriving test suites in UML projects. International conference on the unified modeling language: The language and its applications (Lecture notes in computer science) (pp. 383–397). Berlin: Springer.

    Google Scholar 

  • Belina, F., & Hogrefe, D. (1989). the CCITT-specification and description language SDL. Computer Networks, 16(3), 11–341.

    Google Scholar 

  • Briand, L. C., Labiche, Y., & He, S. (2009). Automating regression test selection based on UML designs. Information and Software Technology, 51(1), 16–30.

    Article  Google Scholar 

  • Briand, L. C., Labiche, Y., & Soccar, G. (2002). Automating impact analysis and regression test selection based on UML designs. In International conference on software maintenance (ICSM) (pp. 252–261).

  • Catal, C., & Mishra, D. (2013). Test case prioritization: A systematic mapping study. Software Quality Journal, 21, 445–478.

    Article  Google Scholar 

  • Chen, Y., Probert, R. L., & Sims, D. P. (2002). Specification based regression test selection with risk analysis. In Proceedings of conference of the center for advance studies on collaborative research.

  • Chen, Y., Probert, R., & Ural, H. (2007). Model-based regression test suite generation using dependence analysis. In Proceedings of the 3rd ACM workshop on advances in model based testing (A-MOST) (pp. 62–67).

  • Chen, Y.-P., Probert, R., & Ural, H. (2009). Regression test suite reduction based on SDL models of system requirements. Journal of Software Maintenance and Evolution, 21(6), 379–405.

    Article  Google Scholar 

  • Cheng, K., & Krishnakumar, A. (1993). Automatic functional test generation using the extended finite state machine model. In Proceedings ACM/IEEE design automation conference (pp. 86–91).

  • de Winter, J. C. (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research and Evaluation, 18(10), 1–2.

    Google Scholar 

  • Deng, D., Sheu, P. C.-Y., Wang, T., & Onoma, A. K. (2004). Model-based testing and maintenance. In Proceedings of the IEEE sixth international symposium on multimedia software engineering (ISMSE) (pp. 278–285).

  • Dick, J., & Faivre, A. (1992). Automating the generation and sequencing of test case from model-based specification. In Proceedings of the international symposium on formal methods (pp. 268–284).

  • Dignan, L. (2010). GM’s volt: 10 million lines of code. CBS News. http://www.cbsnews.com/news/gms-volt-10-million-lines-of-code/. Accessed 1 April 2016.

  • Do, H., Elbaum, S., & Rothermel, G. (2005). Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. Empirical Software Engineering, 10(4), 405–435.

    Article  Google Scholar 

  • Do, H., & Rothermel, G. (2006a). On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Transactions on Software Engineering, 32(9), 733–752.

    Article  Google Scholar 

  • Do, H., & Rothermel, G. (2006b). On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Transactions on Software Engineering (TSE), 32(9), 733–752.

    Article  Google Scholar 

  • Elbaum, S., Malishevsky, A. G., & Rothermel, G. (2000). Prioritizing test cases for regression testing (Vol. 25, no. 5). ACM.

  • Elbaum, S., Malishevsky, A., & Rothermel, G. (2001). Incorportaing varying test costs and fault severities into test case prioritization. In Proceedings of the 23rd international conference on software engineering, ICSE-01 (pp. 329–338).

  • Elbaum, S., Malishevsky, A., & Rothermel, G. (2002). Test case prioritization: A family of empirical studies. IEEE Transactions on Software Engineering, 28, 159–182.

    Article  Google Scholar 

  • Farooq, Q., Iqbal, M. Z. Z., Malik, Z. I., & Nadeem, A. (2007). An approach for selective state machine based regression testing. In Proceedings of the 3rd international ACM workshop on advances in model-based testing (pp. 44–52). London.

  • Ferrante, J., Ottenstein, K., & Warren, J. (1987). The program dependence graph and its use in optimization. ACM Transactions on Programming Languages and Systems, 9(5), 319–349.

    Article  MATH  Google Scholar 

  • Fraser, G., & Wotawa, F. (2007). Test-case prioritization with model-checkers. In Proceedings of the 25th IASTED international multi-conference: Software engineering (pp. 267–272). Innsbruck.

  • Gupta, R., Harrold, M., & Soffa, M. (1992). An approach to regression testing using slices. In Proceedings of the IEEE international conference on software maintenance (pp. 299–308).

  • Harrold, M. J., & Orso, A. (2008). Retesting software during development and maintenance. In Frontiers of software maintenance (FoSM 2008), international conference on software maintenance (pp. 99–108). Beijing.

  • Hemmati, H., Arcuri, A., & Briand, L. (2013). Achieving scalable model-based testing through test case diversity. ACM Transaction on Software Engineering, 22(1), 6–48.

    Google Scholar 

  • Hong, H., & Ural, H. (2000). A test sequence selection method for statecharts. Software Testing, Verification, and Reliability, 10(4), 203–227.

    Article  Google Scholar 

  • ITU-T Recommendation Standard, Q.931. (1998). Digital subscriber signaling system No. 1: Network layer, ISDN User Network Interface Layer 3 Specifications for Basic Call Control.

  • Jia, Y., & Harman, M. (2011). An analysis and survey of the development of mutation testing. In IEEE transactions on software engineering. in press.

  • Jiang, B., Tse, T. H., Grieskamp, W., Kicillof, N., Cao, Y., Li, X., et al. (2011). Assuring the model evolution of protocol software specifications by regression testing process improvement. Software: Practice and Experience, 41(10), 1073–1103.

    Google Scholar 

  • Jiang, B., Zhang, Z., Chan, W. K., & Tse, T. H. (2009). Adaptive random test case prioritization. In Proceedings of the 2009 IEEE/ACM international conference on automated software engineering (ASE ‘09), IEEE computer society (pp. 233–244). Washington, DC.

  • Kaur, P., Bansal, P., & Sibal, R. (2012). Prioritization of test scenarios derived from UML activity diagram using path complexity. Proceedings of the CUBE international information technology conference (CUBE ‘2012) (pp. 355–359). New York, NY: ACM.

    Chapter  Google Scholar 

  • Khoshgoftaar, T. M., et al. (1997). Application of neural networks to software quality modeling of a very large telecommunications system. IEEE Transactions on Neural Networks, 8(4), 902–909.

    Article  Google Scholar 

  • Kim, J., & Porter, A. (2002). A history-based test prioritization technique for regression testing in resource constraint environments. In Proceedings of the 24th international conference on software engineering (pp. 119–129).

  • Kirby, J. (1987). Example NRL/SCR software requirements for an automobile cruise control and monitoring system. In Technical Report TR-87-07, Wang Institute of Graduate Studies.

  • Korel, B., & Koutsogiannakis, G. (2009). Empirical comparison of code-based and model-based test prioritization. In 5th Workshop on Advances in Model Based Testing, A-MOST 2009, IEEE digital library. Denver.

  • Korel, B., Koutsogiannakis, G., & Tahat, L. (2007). Model-based test prioritization heuristic methods and their evaluation. In Proceedings of the 3rd ACM workshop on advances in model based testing, A-MOST.

  • Korel, B., Koutsogiannakis, G., & Tahat, L. (2008). Application of system models in regression test suite prioritization. In IEEE international conference on software maintenance. ICSM 2008 (pp. 247–256). Beijing.

  • Korel, B., Tahat, L., & Harman, M. (2005). Test prioritization using system models. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM ‘05) (pp. 559–568).

  • Korel, B., Tahat, L., & Vaysburg, B. (2002). Model based regression test reduction using dependence analysis. In Proceedings of the IEEE international conference on software maintenance (pp. 214–223).

  • Kundu, D., Sarma, M., Samanta, D., & Mall, R. (2009). System testing for object-oriented systems with test case prioritization. Software Testesting, Verification and Reliability, 19(1), 297–333. doi:10.1002/stvr.407.

    Article  Google Scholar 

  • Le Traon, Y., Jéron, T., Jézéquel, J.-M., & Morel, P. (2000). Efficient object-oriented integration and regression testing. IEEE Transactions on Reliability, 49(1), 12–25.

    Article  Google Scholar 

  • Li, Z., Harman, M., & Hierons, R. (2007). Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering, 33(4), 225–237.

    Article  Google Scholar 

  • Lilienthal, D. (2011). Top 4 fun Boeing 787 technical facts. NYC Aviation. http://www.nycaviation.com/2011/09/fun-facts-revealed-at-boeings-787-technical-panel/#.Vv4P5OJ97IV. Accessed 1 April 2016.

  • Mei, L., Cai, Y., Jia, C., Jiang, B., Chan, W. K., Zhang, Z., et al. (2014). A subsumption hierarchy of test case prioritization for composite services. IEEE Transactions on Services Computing,. doi:10.1109/TSC.2014.2331683.

    Google Scholar 

  • Mei, L., Chan, W. K., Tse, T. H., & Merkel, R. G. (2011a). XML-manipulating test case prioritization for XML-manipulating services. JSS, 84(4), 603–619.

    Google Scholar 

  • Mei, H., Hao, D., Zhang, L., Zhang, L., Zhou, J., & Rothermel, G. (2011). A static approach to prioritizing JUnit test cases. In IEEE transactions on software engineering.

  • Muccini, H., Dias, M. S., & Richardson, D. J. (2005). Reasoning about software architecture-based regression testing through a case study. In International computer software and applications conference (COMPSAC) (pp. 189–195).

  • Mueller C. (2003). Automated interface propping applied to cost component evaluation. Ph.D. Thesis, Illinois Institute of Technology, Chicago, IL.

  • Offutt, A. J., Lee, A., Rothermel, G., Untch, R., & Zapf, C. (1996). An experimental determination of sufficient mutation operators. ACM Transactions on Software Engineering and Methodology, 5(2), 99–118.

    Article  Google Scholar 

  • Orso, A., Do, H., Rothermel, G., Harrold, M. J., & Rosenblum, D. S. (2007). Using component metadata to regression test component-based software. Software Testing Verification and Reliability (STVR), 17(2), 61–94.

    Article  Google Scholar 

  • Paiva, A. C. R., Faria, J. C. P., Tillmann, N., & Vidal, R. F. A. M. (2005). A model-to-implementation mapping tool for automated model-based GUI testing. In Formal methods and software engineering lecture notes in computer science (vol. 3785, pp. 450–464).

  • Pilskalns, O., Uyan, G., & Andrews, A. (2006). Regression testing UML designs. In Proceedings of the 22nd IEEE international conference on software maintenance (pp. 254–264).

  • Rothermel, G., & Harrold, M. (1994). Selecting tests and identifying test coverage requirements for modified software. In Proceedings of the IEEE international conference on software maintenance (pp. 358–367).

  • Rothermel, G., & Harrold, M. (1997). A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology, 6(2), 173–210.

    Article  Google Scholar 

  • Rothermel, G., Harrold, M., Ostrin, J., & Hong, C. (1998). An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the international conference on software maintenance (pp. 34–43).

  • Rothermel, G., Untch, R., Chu, C., & Harrold, M. (1999). Test case prioritization: An empirical study. In Proceedings of the IEEE international conference on software maintenance (pp. 179–188).

  • Rothermel, G., Untch, R. H., Chu, C., & Harrold, M. J. (2001). Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10), 929–948.

    Article  Google Scholar 

  • Saadatmand, M., & Cichetti, A. (2014). Mapping of state machines to code: potentials and challenges. In The Ninth International Conference on Software Engineering Advances, ICSEA-2014 (pp. 247–252).

  • Sabharwal, S., Sibal, R., & Sharma, C. (2010). Prioritization of test case scenarios derived from activity diagram using genetic algorithm. In Proceeding of the computer and communication technology international conference (pp. 481–485).

  • Shankland, S. (2013). Linux development by the numbers: Big and getting bigger. Cnet. http://www.cnet.com/news/linux-development-by-the-numbers-big-and-getting-bigger/. Accessed 1 April 2016.

  • Srivastava, A., & Thiagarajan, J. (2002). Effectively prioritizing tests in development environment. In Proceedings of the ACM international symposium on software testing and analysis, ISSTA-02 (pp. 97–106).

  • Stevens, W. (2001). TCP/IP Illustration, volume I, the Protocol. Boston, MA: Addison Wesley Reading.

    Google Scholar 

  • Tahat, L., Korel, B., Harman, M., & Ural, H. (2012). Regression test suite prioritization using system models. Journal of Software Testing, Verification, and Reliability, 12(1), 481–506.

    Article  Google Scholar 

  • Thomas, S. W., Hemmati, H., Hassan, A. E., & Blostein, D. (2014). Static test case prioritization using topic models. Empirical Software Engineering, 19(1), 182–212.

    Article  Google Scholar 

  • Ural, H., & Yenigun, H. (2013). Regression test suite selection using dependence analysis. Journal of Software: Evolution and Process, 25(7), 681–709.

    MATH  Google Scholar 

  • Vaysburg, B., Tahat, L.,& Korel, B. (2002). Dependence analysis in reduction of requirement based test suites. In Proceedings of the ACM international symposium on software testing and analysis (pp. 107–111).

  • Wakinshaw, N., Bogdanov, K., Ali, S., & Holcombe, M. (2008). Automated discovery of state transitions and their functions in source code. Journal of Software Testing, Verification and Reliability, 18(2), 99–121.

    Article  Google Scholar 

  • White, L. (1993). Test manager: A regression testing tool. In Proceedings of the IEEE international conference on software maintenance (pp. 338–347).

  • Wong, W., Horgan, J., London, S., & Agrawal, H. (1997b). A study of effective regression testing in practice. In Proceedings of the international symposium on software reliability engineering (pp. 230–238).

  • Wong, W. E., Horgan, J. R., Mathur, A. P., & Pasquini, A. (1997a). Test set size minimization and fault detection effectiveness: A case study in a space application. In Proceedings of 21st annual international computer software and applications conference (pp. 522–528).

  • Wu, Y., & Offutt, J. (2003). Maintaining evolving component-based software with UML. In Seventh European conference on software maintenance and reengineering (CSMR) (pp. 133–142).

  • Yenigun, H. (2014). Identifying the effects of modifications as data dependencies. Software Quality Journal, 22, 701–716.

    Article  Google Scholar 

  • Yoo, S., & Harman, M. (2009). Regression testing minimisation, selection and prioritisation: A survey. In Department of Computer Science, King’s College London, Technical Report: TR-09-09.

  • Yoo, S., & Harman, M. (2012). Regression testing minimization, selection and prioritization: A survey. Software, Testing, Verification, and Reliability, 22(2), 67–120.

    Article  Google Scholar 

  • Yoo, S., Harman, M., Tonella, P., & Susi, A. (2009). Clustering test cases to achieve effective and scalable prioritisation incorporating expert knowledge. In Proceedings of the eighteenth international symposium on Software testing and analysis (pp. 201–212). ACM.

  • Zhai, K., Jiang, B., & Chan, W. K. (2014). Prioritizing test cases for regression testing of location-based services: Metrics, techniques, and case study. IEEE Transactions on Services Computing, 7(1), 54–67.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, Illinois Institute of Technology, Chicago, IL, 60616, USA

    Bogdan Korel & George Koutsogiannakis

  2. Management Information System, Gulf University for Science and Technology, Mishref, Kuwait

    Luay Tahat & Nada Almasri

Authors
  1. Luay Tahat
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bogdan Korel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. George Koutsogiannakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nada Almasri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Luay Tahat.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tahat, L., Korel, B., Koutsogiannakis, G. et al. State-based models in regression test suite prioritization. Software Qual J 25, 703–742 (2017). https://doi.org/10.1007/s11219-016-9330-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9330-x

Keywords

Search

Navigation