Empirical Software Engineering

, Volume 23, Issue 2, pp 835–904 | Cite as

The impact of rapid release cycles on the integration delay of fixed issues

  • Daniel Alencar da Costa
  • Shane McIntosh
  • Christoph Treude
  • Uirá Kulesza
  • Ahmed E. Hassan


The release frequency of software projects has increased in recent years. Adopters of so-called rapid releases—short release cycles, often on the order of weeks, days, or even hours—claim that they can deliver fixed issues (i.e., implemented bug fixes and new features) to users more quickly. However, there is little empirical evidence to support these claims. In fact, our prior work shows that code integration phases may introduce delays for rapidly releasing projects—98% of the fixed issues in the rapidly releasing Firefox project had their integration delayed by at least one release. To better understand the impact that rapid release cycles have on the integration delay of fixed issues, we perform a comparative study of traditional and rapid release cycles. Our comparative study has two parts: (i) a quantitative empirical analysis of 72,114 issue reports from the Firefox project, and a (ii) qualitative study involving 37 participants, who are contributors of the Firefox, Eclipse, and ArgoUML projects. Our study is divided into quantitative and qualitative analyses. Quantitative analyses reveal that, surprisingly, fixed issues take a median of 54% (57 days) longer to be integrated in rapid Firefox releases than the traditional ones. To investigate the factors that are related to integration delay in traditional and rapid release cycles, we train regression models that model whether a fixed issue will have its integration delayed or not. Our explanatory models achieve good discrimination (ROC areas of 0.80–0.84) and calibration scores (Brier scores of 0.05–0.16) for rapid and traditional releases. Our explanatory models indicate that (i) traditional releases prioritize the integration of backlog issues, while (ii) rapid releases prioritize issues that were fixed in the current release cycle. Complementary qualitative analyses reveal that participants’ perception about integration delay is tightly related to activities that involve decision making, risk management, and team collaboration. Moreover, the allure of shipping fixed issues faster is a main motivator for adopting rapid release cycles among participants (although this motivation is not supported by our quantitative analysis). Furthermore, to explain why traditional releases deliver fixed issues more quickly, our participants point out the rush for integration in traditional releases and the increased time that is invested on polishing issues in rapid releases. Our results suggest that rapid release cycles may not be a silver bullet for the rapid delivery of new content to users. Instead, our results suggest that the benefits of rapid releases are increased software stability and user feedback.


Integration delay Rapid releases Mining software repositories Release engineering 



This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the National Institute of Science and Technology for Software Engineering (INES), CNPq grant 465614/2014-0. We also thank all the participants from the Firefox, Eclipse, and ArgoUML projects for giving their time to respond our surveys and participate in our interviews.


  1. Adams B, McIntosh S (2016) Modern release engineering in a nutshell: why researchers should care. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER), pp 78–90Google Scholar
  2. AlGhamdi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: Proceedings of the international conference on software maintenance and evolution. IEEE, Piscataway, pp 279–289Google Scholar
  3. Antoniol G, Ayari K, Penta MD, Khomh F, Guéhéneuc Y (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research (CASCON), pp 23–37Google Scholar
  4. Baskerville R, Pries-Heje J (2004) Short cycle time systems development. Inf Syst J 14:237–264CrossRefGoogle Scholar
  5. Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). ACM, New York, pp 238–241Google Scholar
  6. Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, ReadingGoogle Scholar
  7. Beller M, Gousios G, Zaidman A (2015) How (much) do developers test?. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2. IEEE, Piscataway, pp 559–562Google Scholar
  8. Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72CrossRefGoogle Scholar
  9. Charmaz K (2014) Constructing grounded theory. SAGE, Newbury ParkGoogle Scholar
  10. Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114:494–509CrossRefGoogle Scholar
  11. da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Proceedings of the 30th international conference on software maintenance and evolution (ICSME), pp 281–290Google Scholar
  12. da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th international workshop on mining software repositories. ACM, New York, pp 374–385Google Scholar
  13. Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252CrossRefGoogle Scholar
  14. Efron B (1986) How biased is the apparent error rate of a prediction rule?. In: Journal of the american statistical association, Taylor & Francis, Milton Park, vol 81, pp 461–470Google Scholar
  15. Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New DelhizbMATHGoogle Scholar
  16. Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd international workshop on recommendation systems for software engineering (RSSE). ACM, New York, pp 52–56Google Scholar
  17. Greer D, Ruhe G (2004) Software release planning: an evolutionary and iterative approach. Inf Softw Technol 46:243–253CrossRefGoogle Scholar
  18. Harrell FE (2001) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, New YorkCrossRefzbMATHGoogle Scholar
  19. Herraiz I, German DM, Gonzalez-Barahona JM, Robles G (2008) Towards a simplification of the bug report form in eclipse. In: Proceedings of the 2008 international working conference on mining software repositories (MSR), pp 145–148Google Scholar
  20. Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70MathSciNetzbMATHGoogle Scholar
  21. Howell DC (2005) Median absolute deviation. In: Encyclopedia of statistics in behavioral science. Wiley Online Library, HobokenGoogle Scholar
  22. Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. In: Journal of clinical oncology, american society of clinical oncology, vol 26, pp 1364–1370Google Scholar
  23. Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, New York, pp 111–120Google Scholar
  24. Jiang Y, Adams B (2014) How much does integrating this commit cost? - a position paper. In: 2nd International Workshop on Release Engineering (RELENG)Google Scholar
  25. Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast?: case study on the linux kernel. In: Proceedings of the 10th working conference on mining software repositories (MSR), pp 101–110Google Scholar
  26. Kampstra P et al. (2008) Beanplot: a boxplot alternative for visual comparison of distributions. J Stat Softw 28:1–9CrossRefGoogle Scholar
  27. Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14:67–74CrossRefGoogle Scholar
  28. Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 179–188Google Scholar
  29. Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621CrossRefzbMATHGoogle Scholar
  30. Leys C, Ley C, Klein O, Bernard P, Licata L (2013) Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J Exp Soc Psychol 49:764–766CrossRefGoogle Scholar
  31. Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2014) On rapid releases and software testing: a case study and a semi-systematic literature review. In: Journal of Empirical Software Engineering. Springer, Berlin, pp 1–42Google Scholar
  32. McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189CrossRefGoogle Scholar
  33. Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015a) Characterization and prediction of issue-related risks in software projects. In: 12th international conference on mining software repositories (MSR), pp 280–291Google Scholar
  34. Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015b) Predicting delays in software projects using networked classification. In: 30th international conference on automated software engineering (ASE)Google Scholar
  35. Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering (ICSE). IEEE, Piscataway, pp 284–292Google Scholar
  36. Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises, pp 308–313Google Scholar
  37. Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the 4th international workshop on mining software repositories (MSR), p 29Google Scholar
  38. Rahman MT, Rigby PC (2015) Release stabilization on linux and chrome. In: IEEE software journal. vol 2. IEEE, Piscataway, pp 81–88Google Scholar
  39. Regnell B, Brinkkemper S (2005) Market-driven requirements engineering for software products. In: Engineering and managing software requirements, pp 287–308Google Scholar
  40. Sandelowski M (1995) Sample size in qualitative research. Res Nurs Health 18 (2):179–183CrossRefGoogle Scholar
  41. Schroter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs?. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 118–121Google Scholar
  42. Schwaber K (1997) Scrum development process. In: Business object design and implementation. Springer, Berlin, pp 117–134Google Scholar
  43. Sheehan KB (2001) E-mail survey response rates: a review. J Comput-Mediat Commun 6(2)  10.1111/j.1083-6101.2001.tb00117.x  10.1111/j.1083-6101.2001.tb00117.x
  44. Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K (2010) Predicting re-opened bugs: a case study on the eclipse project. In: Proceedings of 17th working conference on reverse engineering (WCRE). IEEE, Piscataway, pp 249–258Google Scholar
  45. Shimagaki J, Kamei Y, McIntosh S, Pursehouse D, Ubayashi N (2016) Why are commits being reverted?: a comparative study of industrial and open source projects. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, Piscataway, pp 301–311Google Scholar
  46. Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 6th international workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92Google Scholar
  47. Souza R, Chavez C, Bittencourt RA (2014) Do rapid releases affect bug reopening? a case study of firefox. In: Proceedings of the brazilian symposium on software engineering (SBES). IEEE, Piscataway, pp 31–40Google Scholar
  48. Souza R, Chavez C, Bittencourt R (2015) Rapid releases and patch backouts: a software analytics approach. In: IEEE software journal, vol 32. IEEE, Piscataway, pp 89–96Google Scholar
  49. Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101CrossRefGoogle Scholar
  50. Subramaniam C, Sen R, Nelson ML (2009) Determinants of open source software project success: a longitudinal study. In: Journal of decision support systems, vol 46. Elsevier, Amsterdam, pp 576–585Google Scholar
  51. Tian Y, Ali N, Lo D, Hassan AE (2015) On the unreliability of bug severity data. Empir Softw Eng 21:1–26Google Scholar
  52. Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic Press, CambridgeGoogle Scholar
  53. Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories. ACM, New York, pp 93–102Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringQueen’s UniversityKingstonCanada
  2. 2.Department of Electrical and Computer EngineeringMcGill UniversityMontrealCanada
  3. 3.School of Computer ScienceUniversity of AdelaideAdelaideAustralia
  4. 4.Department of Informatics and Applied Mathematics (DIMAp)Federal University of Rio Grande do NorteNatalBrazil
  5. 5.Software Analysis and Intelligence Lab (SAIL)Queen’s UniversityKingstonCanada

Personalised recommendations