Empirical Software Engineering

, Volume 21, Issue 5, pp 1990–2032 | Cite as

Change-based test selection: an empirical evaluation

  • Quinten David Soetens
  • Serge Demeyer
  • Andy Zaidman
  • Javier Pérez


Regression test selection (i.e., selecting a subset of a given regression test suite) is a problem that has been studied intensely over the last decade. However, with the increasing popularity of developer tests as the driver of the test process, more fine-grained solutions that work well within the context of the Integrated Development Environment (IDE) are in order. Consequently, we created two variants of a test selection heuristic which exploit fine-grained changes recorded during actual development inside the IDE. One variant only considers static binding of method invocations while the other variant takes dynamic binding into account. This paper investigates the tradeoffs between these two variants in terms of the reduction (i.e., How many tests could we omit from the test suite, and how much did we gain in runtime execution?) as well as the fault detection ability of the reduced test suite (i.e., Were tests omitted erroneously?). We used our approach on three distinct cases, two open source cases —Cruisecontrol and PMD— and one industrial case — Historia. Our results show that only considering static binding reduces the test suite significantly but occasionally omits a relevant test; considering dynamic binding rarely misses a test yet often boils down to running the complete test suite. Nevertheless, our analysis provides indications on when a given variant is more appropriate.


Fine grained changes Test selection Developer tests Dynamic binding 



We express our gratitude to the SEAL team in the University of Zürich, Switzerland for releasing Changedistiller in the public domain; our ChEOPSJ tool is partly based on this release.

We would also like to express are gratitude to Yves Vandewoude at Qmino, for allowing us to conduct these analyses on the Historia codebase.

This work has been sponsored by (i) the Interuniversity Attraction Poles Programme - Belgian State Belgian Science Policy, project MoVES; (ii) the Institute for the Promotion of Innovation through Science and Technology in Flanders (IWT-Vlaanderen) under project number 120028 entitled “Change-centric Quality Assurance (CHAQ)”; (iii) the NWO TestRoots project, the Netherlands.

Figures 67891011 and 12 were created with matplotlib (Hunter 2007).


  1. Andrews JH, Briand LC, Labiche Y (2005) Is mutation an appropriate tool for testing experiments? In: Proceedings Int’l conference on software engineering (ICSE), ACM, pp 402–411. doi: 10.1145/1062455.1062530
  2. Basili VR, Caldiera G, Rombach HD (1994) The goal question metric approach. Encyclopedia of Software EngineeringGoogle Scholar
  3. Beck K (2002) Test driven development: by example. Addison-WesleyGoogle Scholar
  4. Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how and why developers (do not) test in their ides. In: Proceedings of the 10th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 179–190Google Scholar
  5. Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test? In: Proceedings of the 37th international conference on software engineering (ICSE – vol 2). IEEE, pp 559–562Google Scholar
  6. Bennett KH, Rajlich VT (2000) Software maintenance and evolution: a roadmap. In: Proceedings of the conference on the future of software engineering, ICSE ’00, ACM New York, pp 73–87. doi: 10.1145/336512.336534
  7. Binder R (1999) Testing object-oriented systems: models, patterns, and tools. Addison-WesleyGoogle Scholar
  8. Catal C, Mishra D (2013) Test case prioritization: a systematic mapping study. Softw Quality Control 21(3):445–478. doi: 10.1007/s11219-012-9181-z CrossRefGoogle Scholar
  9. Daniel B, Jagannath V, Dig D, Marinov D (2009) Reassert: suggesting repairs for broken unit testsGoogle Scholar
  10. Demeyer S, Tichelaar S, Steyaert P (1999) FAMIX 2.0 - the FAMOOS information exchange model. Tech. rep., University of BerneGoogle Scholar
  11. Dösinger S, Mordinyi R, Biffl S (2012) Communicating continuous integration servers for increasing effectiveness of automated testing. In: 2012 Proceedings of the 27th IEEE/ACM international conference on Automated Software Engineering (ASE). IEEE, pp 374–377Google Scholar
  12. Ebraert P, Vallejos J, Costanza P, Paesschen EV, D’Hondt T (2007) Change-oriented software engineering. In: Proceedings of the Int’l conference on dynamic languages (ICDL). ACM, pp 3–24. doi: 10.1145/1352678.1352680
  13. Engström E, Skoglund M, Runeson P (2008) Empirical evaluations of regression test selection techniques: a systematic review. In: Proceedings Int’l symposium empirical software engineering and measurement (ESEM). ACM, pp 22–31. doi: 10.1145/1414004.1414011
  14. Engström E, Runeson P, Skoglund M (2010) A systematic review on regression test selection techniques. J Inf Softw Technol 52(1):14–30CrossRefGoogle Scholar
  15. Fluri B, Wuersch M, PInzger M, Gall H (2007) Change distilling: tree differencing for fine-grained source code change extraction. IEEE Trans Softw Eng 33 (11):725–743. doi: 10.1109/TSE.2007.70731 CrossRefGoogle Scholar
  16. Garousi V, Varma T (2010) A replicated survey of software testing practices in the canadian province of alberta: what has changed from 2004 to 2009?. J Syst Softw 83(11):2251–2262CrossRefGoogle Scholar
  17. Graves TL, Harrold MJ, Kim JM, Porter A, Rothermel G (2001) An empirical study of regression test selection techniques. ACM Trans Softw Eng Methodol 10(2):184–208CrossRefzbMATHGoogle Scholar
  18. Hattori L, Lanza M (2010) Syde: a tool for collaborative software development. In: Proceedings of the Int’l conference on software engineering (ICSE). ACM, pp 235–238Google Scholar
  19. Hunter JD (2007) Matplotlib: a 2d graphics environment. Comput Sci Eng 9 (3):90–95CrossRefGoogle Scholar
  20. Hurdugaci V, Zaidman A (2012) Aiding software developers to maintain developer tests. In: Proceedings of the 16th European conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 11–20Google Scholar
  21. Mansour N, Bahsoon R, Baradhi G (2001) Empirical comparison of regression test selection algorithms. J Syst Softw 57(1):79–90CrossRefGoogle Scholar
  22. McGregor J (2007) Test early, test often. J Object Technol 6(4)Google Scholar
  23. Meszaros G (2006) XUnit test patterns: refactoring test code. Prentice Hall PTRGoogle Scholar
  24. Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C (1996) An experimental determination of sufficient mutant operators. ACM Trans Softw Eng Methodol 5(2):99–118. doi: 10.1145/227607.227610 CrossRefGoogle Scholar
  25. Parsai A, Soetens QD, Murgia A, Demeyer S (2014) Considering polymorphism in change-based test suite reduction. In: Agile methods. Large-scale development, refactoring, testing, and estimation - XP 2014 international workshops. Rome, Italy, May 26-30, 2014, Revised Selected Papers, pp 166–181. doi: 10.1007/978-3-319-14358-3_14
  26. Qusef A, Bavota G, Oliveto R, De Lucia A, Binkley D (2011) Scotch: test-to-code traceability using slicing and conceptual coupling. In: Proceedings of the Int’l conference on software maintenance (ICSM). IEEE CS, pp 63–72. doi: 10.1109/ICSM.2011.6080773
  27. Robbes R, Lanza M (2007) A change-based approach to software evolution. Electron Notes Theor Comput Sci 166:93–109. doi: 10.1016/j.entcs.2006.06.015 CrossRefGoogle Scholar
  28. Robbes R, Lanza M (2008) Spyware: a change-aware development toolset. In: Proceedings of the Int’l conference in software engineering (ICSE). ACM Press, pp 847–850Google Scholar
  29. Rothermel G, Untch R, Chu C, Harrold M (2001) Prioritizing test cases for regression testing. IEEE Trans Softw Eng 27(10):929–948. doi: 10.1109/32.962562 CrossRefGoogle Scholar
  30. Runeson P (2006) A survey of unit testing practices. IEEE Softw 23(4):22–29CrossRefGoogle Scholar
  31. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Eng 14(2):131–164CrossRefGoogle Scholar
  32. Saff D, Ernst MD (2004) An experimental evaluation of continuous testing during development. In: Proceedings Int’l symposium software testing and analysis (ISSTA). ACM, pp 76–85Google Scholar
  33. Soetens Q D, Demeyer S (2012) ChEOPSJ: change-based test optimization. In: Proceedings of the European conference on software maintenance and reengineering (CSMR). IEEE CS, pp 535–538. doi: 10.1109/CSMR.2012.70
  34. Soetens QD, Demeyer S, Zaidman A (2013) Change-based test selection in the presence of developer tests. In: Proceedings of the 17th European conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 101–110Google Scholar
  35. Tillmann N, Schulte W (2006) Unit tests reloaded: parameterized unit testing with symbolic execution. IEEE Softw 23(4)Google Scholar
  36. Van Rompaey B, Demeyer S (2009) Establishing traceability links between unit test cases and units under test. In: Proceedigns of the conference on software maintenance and reengineering (CSMR). IEEE CS, pp 209–218. doi: 10.1109/CSMR.2009.39
  37. Venolia G, DeLine R, LaToza T (2005) Software development at microsoft observed. Tech. rep., Microsoft Research.
  38. Weijers J (2012) Extending project lombok to improve junit tests. Master’s thesis, Delft University of Technology,
  39. Yin RK (2002) Case study research: design and methods, 3 edn. Sage PublicationsGoogle Scholar
  40. Yoo S, Harman M (2012) Regression testing minimization, selection and prioritization: a survey. Softw Test Verif Reliab 22(2):67–120. doi: 10.1002/stvr.430 CrossRefGoogle Scholar
  41. Yoo S, Nilsson R, Harman M (2011) Faster fault finding at Google using multi objective regression test optimisation. In: 8th European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE ’11). Szeged, HungaryGoogle Scholar
  42. Zaidman A, Van Rompaey B, van Deursen A, Demeyer S (2011) Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining. Empirical Softw Eng 16(3):325–364CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Quinten David Soetens
    • 1
  • Serge Demeyer
    • 1
  • Andy Zaidman
    • 2
  • Javier Pérez
    • 1
  1. 1.University of AntwerpAntwerpBelgium
  2. 2.Delft University of TechnologyDelftThe Netherlands

Personalised recommendations