Skip to main content
SpringerLink
Log in

The impact of rapid release cycles on the integration delay of fixed issues

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

The release frequency of software projects has increased in recent years. Adopters of so-called rapid releases—short release cycles, often on the order of weeks, days, or even hours—claim that they can deliver fixed issues (i.e., implemented bug fixes and new features) to users more quickly. However, there is little empirical evidence to support these claims. In fact, our prior work shows that code integration phases may introduce delays for rapidly releasing projects—98% of the fixed issues in the rapidly releasing Firefox project had their integration delayed by at least one release. To better understand the impact that rapid release cycles have on the integration delay of fixed issues, we perform a comparative study of traditional and rapid release cycles. Our comparative study has two parts: (i) a quantitative empirical analysis of 72,114 issue reports from the Firefox project, and a (ii) qualitative study involving 37 participants, who are contributors of the Firefox, Eclipse, and ArgoUML projects. Our study is divided into quantitative and qualitative analyses. Quantitative analyses reveal that, surprisingly, fixed issues take a median of 54% (57 days) longer to be integrated in rapid Firefox releases than the traditional ones. To investigate the factors that are related to integration delay in traditional and rapid release cycles, we train regression models that model whether a fixed issue will have its integration delayed or not. Our explanatory models achieve good discrimination (ROC areas of 0.80–0.84) and calibration scores (Brier scores of 0.05–0.16) for rapid and traditional releases. Our explanatory models indicate that (i) traditional releases prioritize the integration of backlog issues, while (ii) rapid releases prioritize issues that were fixed in the current release cycle. Complementary qualitative analyses reveal that participants’ perception about integration delay is tightly related to activities that involve decision making, risk management, and team collaboration. Moreover, the allure of shipping fixed issues faster is a main motivator for adopting rapid release cycles among participants (although this motivation is not supported by our quantitative analysis). Furthermore, to explain why traditional releases deliver fixed issues more quickly, our participants point out the rush for integration in traditional releases and the increased time that is invested on polishing issues in rapid releases. Our results suggest that rapid release cycles may not be a silver bullet for the rapid delivery of new content to users. Instead, our results suggest that the benefits of rapid releases are increased software stability and user feedback.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Notes

  1. http://www.scrumguides.org/.

  2. https://www.bugzilla.org/.

  3. https://bugzilla.readthedocs.org/en/5.0/using/understanding.html.

  4. https://bugzilla.readthedocs.org/en/5.0/using/editing.html#life-cycle-of-a-bug.

  5. https://www.mozilla.org/en-US/firefox/new/.

  6. https://clicky.com/marketshare/global/web-browsers/.

  7. https://en.wikipedia.org/wiki/Firefox_release_history.

  8. https://hg.mozilla.org/mozilla-central/.

  9. http://mozilla.github.io/process-releases/draft/development_overview/.

  10. https://en.wikipedia.org/wiki/History_of_Firefox#Version_27.

  11. https://www.mozilla.org/en-US/firefox/organizations/faq/.

  12. https://en.wikipedia.org/wiki/Firefox_release_history.

  13. http://cvsbook.red-bean.com/cvsbook.html.

  14. https://mercurial.selenic.com/.

  15. https://hg.mozilla.org/mozilla-central/rev/bd0fdb3585c6.

  16. https://hg.mozilla.org/mozilla-central/rev/2e06eade69ce.

  17. https://www.mozilla.org/en-US/firefox/releases/.

  18. http://argouml.tigris.org.

  19. http://argouml.tigris.org/project_bugs.html.

  20. https://eclipse.org/.

  21. https://projects.eclipse.org/projects/eclipse.jdt.

  22. From 2013 to 2016 by the time of this study.

  23. https://www.mozilla.org/en-US/firefox/releases/.

  24. We did not observe a statistically significant difference in integration delays between issues that are fixed by the reporters themselves and issues that are fixed by a different team member.

References

  • Adams B, McIntosh S (2016) Modern release engineering in a nutshell: why researchers should care. In: Proceedings of the 23rd international conference on software analysis, evolution, and reengineering (SANER), pp 78–90

  • AlGhamdi HM, Syer MD, Shang W, Hassan AE (2016) An automated approach for recommending when to stop performance tests. In: Proceedings of the international conference on software maintenance and evolution. IEEE, Piscataway, pp 279–289

  • Antoniol G, Ayari K, Penta MD, Khomh F, Guéhéneuc Y (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. In: Proceedings of the 2008 conference of the centre for advanced studies on collaborative research (CASCON), pp 23–37

  • Baskerville R, Pries-Heje J (2004) Short cycle time systems development. Inf Syst J 14:237–264

    Article  Google Scholar 

  • Baysal O, Davis I, Godfrey MW (2011) A tale of two browsers. In: Proceedings of the 8th working conference on mining software repositories (MSR). ACM, New York, pp 238–241

  • Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional, Reading

    Google Scholar 

  • Beller M, Gousios G, Zaidman A (2015) How (much) do developers test?. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2. IEEE, Piscataway, pp 559–562

  • Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72

    Article  Google Scholar 

  • Charmaz K (2014) Constructing grounded theory. SAGE, Newbury Park

    Google Scholar 

  • Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114:494–509

    Article  Google Scholar 

  • da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Proceedings of the 30th international conference on software maintenance and evolution (ICSME), pp 281–290

  • da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th international workshop on mining software repositories. ACM, New York, pp 374–385

  • Dunn OJ (1964) Multiple comparisons using rank sums. Technometrics 6 (3):241–252

    Article  Google Scholar 

  • Efron B (1986) How biased is the apparent error rate of a prediction rule?. In: Journal of the american statistical association, Taylor & Francis, Milton Park, vol 81, pp 461–470

  • Fisher RA (1925) Statistical methods for research workers. Genesis Publishing Pvt Ltd, New Delhi

    MATH  Google Scholar 

  • Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd international workshop on recommendation systems for software engineering (RSSE). ACM, New York, pp 52–56

  • Greer D, Ruhe G (2004) Software release planning: an evolutionary and iterative approach. Inf Softw Technol 46:243–253

    Article  Google Scholar 

  • Harrell FE (2001) Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer, New York

    Book  MATH  Google Scholar 

  • Herraiz I, German DM, Gonzalez-Barahona JM, Robles G (2008) Towards a simplification of the bug report form in eclipse. In: Proceedings of the 2008 international working conference on mining software repositories (MSR), pp 145–148

  • Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 6:65–70

    MathSciNet  MATH  Google Scholar 

  • Howell DC (2005) Median absolute deviation. In: Encyclopedia of statistics in behavioral science. Wiley Online Library, Hoboken

  • Iasonos A, Schrag D, Raj GV, Panageas KS (2008) How to build and interpret a nomogram for cancer prognosis. In: Journal of clinical oncology, american society of clinical oncology, vol 26, pp 1364–1370

  • Jeong G, Kim S, Zimmermann T (2009) Improving bug triage with bug tossing graphs. In: Proceedings of the 7th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, New York, pp 111–120

  • Jiang Y, Adams B (2014) How much does integrating this commit cost? - a position paper. In: 2nd International Workshop on Release Engineering (RELENG)

  • Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast?: case study on the linux kernel. In: Proceedings of the 10th working conference on mining software repositories (MSR), pp 101–110

  • Kampstra P et al. (2008) Beanplot: a boxplot alternative for visual comparison of distributions. J Stat Softw 28:1–9

    Article  Google Scholar 

  • Karlsson J, Ryan K (1997) A cost-value approach for prioritizing requirements. IEEE Softw 14:67–74

    Article  Google Scholar 

  • Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? an empirical case study of mozilla firefox. In: Proceedings of the 9th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 179–188

  • Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621

    Article  MATH  Google Scholar 

  • Leys C, Ley C, Klein O, Bernard P, Licata L (2013) Detecting outliers: do not use standard deviation around the mean, use absolute deviation around the median. J Exp Soc Psychol 49:764–766

    Article  Google Scholar 

  • Mäntylä MV, Adams B, Khomh F, Engström E, Petersen K (2014) On rapid releases and software testing: a case study and a semi-systematic literature review. In: Journal of Empirical Software Engineering. Springer, Berlin, pp 1–42

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189

    Article  Google Scholar 

  • Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015a) Characterization and prediction of issue-related risks in software projects. In: 12th international conference on mining software repositories (MSR), pp 280–291

  • Morakot C, Hoa Khanh D, Truyen T, Aditya G (2015b) Predicting delays in software projects using networked classification. In: 30th international conference on automated software engineering (ASE)

  • Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering (ICSE). IEEE, Piscataway, pp 284–292

  • Paetsch F, Eberlein A, Maurer F (2003) Requirements engineering and agile software development. In: Twelfth IEEE international workshops on enabling technologies: infrastructure for collaborative enterprises, pp 308–313

  • Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the 4th international workshop on mining software repositories (MSR), p 29

  • Rahman MT, Rigby PC (2015) Release stabilization on linux and chrome. In: IEEE software journal. vol 2. IEEE, Piscataway, pp 81–88

  • Regnell B, Brinkkemper S (2005) Market-driven requirements engineering for software products. In: Engineering and managing software requirements, pp 287–308

  • Sandelowski M (1995) Sample size in qualitative research. Res Nurs Health 18 (2):179–183

    Article  Google Scholar 

  • Schroter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs?. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, Piscataway, pp 118–121

  • Schwaber K (1997) Scrum development process. In: Business object design and implementation. Springer, Berlin, pp 117–134

  • Sheehan KB (2001) E-mail survey response rates: a review. J Comput-Mediat Commun 6(2) 10.1111/j.1083-6101.2001.tb00117.x 10.1111/j.1083-6101.2001.tb00117.x

  • Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto K (2010) Predicting re-opened bugs: a case study on the eclipse project. In: Proceedings of 17th working conference on reverse engineering (WCRE). IEEE, Piscataway, pp 249–258

  • Shimagaki J, Kamei Y, McIntosh S, Pursehouse D, Ubayashi N (2016) Why are commits being reverted?: a comparative study of industrial and open source projects. In: 2016 IEEE international conference on software maintenance and evolution (ICSME). IEEE, Piscataway, pp 301–311

  • Smith E, Loftin R, Murphy-Hill E, Bird C, Zimmermann T (2013) Improving developer participation rates in surveys. In: 6th international workshop on cooperative and human aspects of software engineering (CHASE), pp 89–92

  • Souza R, Chavez C, Bittencourt RA (2014) Do rapid releases affect bug reopening? a case study of firefox. In: Proceedings of the brazilian symposium on software engineering (SBES). IEEE, Piscataway, pp 31–40

  • Souza R, Chavez C, Bittencourt R (2015) Rapid releases and patch backouts: a software analytics approach. In: IEEE software journal, vol 32. IEEE, Piscataway, pp 89–96

  • Spearman C (1904) The proof and measurement of association between two things. Am J Psychol 15(1):72–101

    Article  Google Scholar 

  • Subramaniam C, Sen R, Nelson ML (2009) Determinants of open source software project success: a longitudinal study. In: Journal of decision support systems, vol 46. Elsevier, Amsterdam, pp 576–585

  • Tian Y, Ali N, Lo D, Hassan AE (2015) On the unreliability of bug severity data. Empir Softw Eng 21:1–26

    Google Scholar 

  • Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic Press, Cambridge

    Google Scholar 

  • Zaman S, Adams B, Hassan AE (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories. ACM, New York, pp 93–102

Download references

Acknowledgments

This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the National Institute of Science and Technology for Software Engineering (INES), CNPq grant 465614/2014-0. We also thank all the participants from the Firefox, Eclipse, and ArgoUML projects for giving their time to respond our surveys and participate in our interviews.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Queen’s University, Kingston, ON, Canada

    Daniel Alencar da Costa

  2. Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada

    Shane McIntosh

  3. School of Computer Science, University of Adelaide, Adelaide, South Australia, Australia

    Christoph Treude

  4. Department of Informatics and Applied Mathematics (DIMAp), Federal University of Rio Grande do Norte, Natal, Brazil

    Uirá Kulesza

  5. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, ON, Canada

    Ahmed E. Hassan

Authors
  1. Daniel Alencar da Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shane McIntosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christoph Treude
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Uirá Kulesza
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Alencar da Costa.

Additional information

Communicated by: Romain Robbes, Christian Bird, and Emily Hill

Appendices

Appendix A: Firefox Survey

figure l
figure m
figure n
figure o
figure p
figure q
figure r

Appendix B: ArgoUML Survey

figure s
figure t
figure u
figure v
figure w
figure x
figure y
figure z

Appendix C: Eclipse Survey

figure aa
figure ab
figure ac
figure ad
figure ae
figure af
figure ag
figure ah

Appendix D: Methodology Web Page I

figure ai

Appendix E: Methodology Web Page II

figure aj

Appendix F: Invitation Letter

figure ak

Appendix G: Interview Script

figure al

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Costa, D., McIntosh, S., Treude, C. et al. The impact of rapid release cycles on the integration delay of fixed issues. Empir Software Eng 23, 835–904 (2018). https://doi.org/10.1007/s10664-017-9548-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9548-7

Keywords

Search

Navigation