Skip to main content
SpringerLink
Log in

An empirical study of patch uplift in rapid release development pipelines

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

In rapid release development processes, patches that fix critical issues, or implement high-value features are often promoted directly from the development channel to a stabilization channel, potentially skipping one or more stabilization channels. This practice is called patch uplift. Patch uplift is risky, because patches that are rushed through the stabilization phase can end up introducing regressions in the code. This paper examines patch uplift operations at Mozilla, with the aim to identify the characteristics of the uplifted patches that did not effectively fix the targeted problem and that introduced regressions. Through statistical and manual analyses, a series of problems were investigated, including the reasons behind patch uplift decisions, the root causes of ineffective uplifts, the characteristics of uplifted patches that introduced regressions, and whether these regressions can be prevented. Additionally, three Mozilla release managers were interviewed in order to understand organizational factors that affect patch uplift decisions and outcomes. Results show that most patches are uplifted because of a wrong functionality or a crash. Certain uplifts did not effectively address their problems because they did not completely fix the problems or lead to regressions. Uplifted patches that lead to regressions tend to have larger patch size, and most of the faults are due to semantic or memory errors in the patches. Also, release managers are more inclined to accept patch uplift requests that concern certain specific components, and–or that are submitted by certain specific developers. About 25% to 30% of the regressions due to Beta or Release uplifts could have been prevented as they could be reproduced by developers and were found in widely used feature/website/configuration or via telemetry.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. In this paper, “duplicate” issues indicate different issues that aim to address the same problem, rather than DUPLICATE in the Bugzilla sense, which means identical issues.

  2. There are in total 7,267 studied uplift requests, but some requests are across multiple channels.

  3. Please refer to our data repository to see the whole list of ignored words: https://github.com/swatlab/uplift-analysis

  4. https://bugzilla.mozilla.org/show_bug.cgi?id=1156182

  5. https://bugzilla.mozilla.org/show_bug.cgi?id=1044975

  6. https://bugzilla.mozilla.org/show_bug.cgi?id=1154003

  7. https://bugzilla.mozilla.org/show_bug.cgi?id=1162520

  8. https://bugzilla.mozilla.org/show_bug.cgi?id=1297390

  9. https://bugzilla.mozilla.org/show_bug.cgi?id=1280110

  10. https://bugzilla.mozilla.org/show_bug.cgi?id=1059797

  11. https://bugzilla.mozilla.org/show_bug.cgi?id=1239789

  12. https://bugzilla.mozilla.org/show_bug.cgi?id=1075199

  13. https://bugzilla.mozilla.org/show_bug.cgi?id=1160914

  14. Please refer to the detailed comparison in our data repository: https://github.com/swatlab/uplift-analysis

  15. Please refer to the detailed comparisons in our data repository: https://github.com/swatlab/uplift-analysis

References

  • Adams B, McIntosh S (2016) Modern release engineering in a nutshell–why researchers should care. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER). IEEE, vol 5, pp 78–90

  • Adams B, Bellomo S, Bird C, Marshall-Keim T, Khomh F, Moir K (2015) The practice and future of release engineering: a roundtable with three release engineers. IEEE Softw 32(2):42–49

    Article  Google Scholar 

  • An L, Khomh F (2015) An empirical study of highly-impactful bugs in Mozilla projects. In: Proceedings of the IEEE International Conference on Software Quality Reliability and Security (QRS). IEEE

  • An L, Khomh F, Adams B (2014) Supplementary bug fixes vs. re-opened bugs. In: 2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, pp 205–214

  • Castelluccio M, An L, Khomh F (2017) Is it safe to uplift this patch? an empirical study on mozilla firefox. In: 2017 IEEE International Conference on Software maintenance and evolution (ICSME). IEEE, pp 411–421

  • Cliff N (1993) Dominance statistics: ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494

    Article  Google Scholar 

  • coverity.com (2017) Coverity tool. http://www.coverity.com, online; Accessed March 31st, 2017

  • Csardi G, Nepusz T (2006) The igraph software package for complex network research. Interjournal. Compl Syst 1695(5):1–9

    Google Scholar 

  • Da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on integration delay of addressed issues: an empirical study of the mozilla firefox project. In: Proceedings of the 13th International Conference on Mining Software Repositories (MSR), pp 374–385

  • Dmitrienko A, Molenberghs G, Chuang-Stein C, Offen W (2005) Analysis of Clinical Trials Using SAS: A Practical Guide. SAS Institute, http://www.google.ca/books?id=G5ElnZDDm8gC

  • Fischer M, Pinzger M, Gall H (2003) Populating a release history database from version control and bug tracking systems. In: Proceedings of the 29th International Conference on Software Maintenance (ICSM). IEEE, pp 23–32

  • Hassan S, Shang W, Hassan A E (2017) An empirical study of emergency updates for top android mobile apps. Empir Softw Eng 22(1):505–546

    Article  Google Scholar 

  • Hollander M, Wolfe DA, Chicken E (2013) Nonparametric statistical methods, 3rd edn. Wiley, New York

    MATH  Google Scholar 

  • Islam MR, Zibran MF (2017) Leveraging automated sentiment analysis in software engineering. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), IEEE, pp 203– 214

  • Jalbert N, Weimer W (2008) Automated duplicate detection for bug tracking systems. In: DSN 2008 IEEE International Conference on Dependable Systems and Networks With FTCS and DCC 2008. IEEE, pp 52–61

  • Khomh F, Dhaliwal T, Zou Y, Adams B (2012) Do faster releases improve software quality? An empirical case study of Mozilla Firefox. In: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories (MSR). IEEE, pp 179–188

  • Khomh F, Adams B, Dhaliwal T, Zou Y (2015) Understanding the impact of rapid releases on software quality. Empir Softw Eng 20(2):336–373

    Article  Google Scholar 

  • Kim D, Wang X, Kim S, Zeller A, Cheung S C, Park S (2011) Which crashes should I fix first?: Predicting top crashes at an early stage to prioritize debugging efforts. IEEE Trans Softw Eng 37(3):430–447

    Article  Google Scholar 

  • Lin D, Bezemer C P, Hassan A E (2017) Studying the urgent updates of popular games on the steam platform. Empir Softw Eng 22(4):2095–2126

    Article  Google Scholar 

  • martinfowler.com (2017) Feature toggle. https://martinfowler.com/bliki/FeatureToggle.html, online; Accessed March 22nd, 2017

  • MDN Web Docs (2017) A Bug’s Life. https://developer.mozilla.org/en-US/docs/Mozilla/QA/A_Bugs_Life, online; Accessed May 20th, 2018

  • Mike T, Kevan B, Georgios P, Di C, Arvid K (2010) Sentiment in short strength detection informal text. JASIST 61(12):2544–2558

    Article  Google Scholar 

  • Mozilla wiki (2016) Priority Field. https://wiki.mozilla.org/Bugmasters/Projects/Folk_Knowledge/Priority_Field, online; Accessed May 20th, 2018

  • Mozilla wiki (2018a) Mozilla Modules. https://wiki.mozilla.org/Modules, online; Accessed September 22nd, 2018

  • Mozilla wiki (2018b) Mozilla Release Management Uplift Rules. https://wiki.mozilla.org/Release_Management/Uplift_rules, online; Accessed May 20th, 2018

  • Mozilla wiki (2018c) Mozilla Tree Sheriffs. https://wiki.mozilla.org/Sheriffing, online; Accessed May 20th, 2018

  • Mozilla wiki (2018d) Mozilla Tree Sheriffs - Backouts. https://wiki.mozilla.org/Sheriffing/How_To/Backouts, online; Accessed September 22nd, 2018

  • Park J, Kim M, Ray B, Bae DH (2012) An empirical study of supplementary bug fixes. In: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories. IEEE Press, pp 40–49

  • Rahman M T, Rigby P C (2015) Release stabilization on linux and chrome. IEEE Softw 32(2):81–88

    Article  Google Scholar 

  • Rahman MT, Querel LP, Rigby PC, Adams B (2016) Feature toggles: practitioner practices and a case study. In: Proceedings of the 13th International Conference on Mining Software Repositories. ACM, pp 201–211

  • Romano J, Kromrey JD, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’sd for evaluating group differences on the nsse and other surveys. In: annual meeting of the Florida Association of Institutional Research, pp 1–33

  • Runeson P, Alexandersson M, Nyholm O (2007) Detection of duplicate defect reports using natural language processing. In: Proceedings of the 29th international conference on Software Engineering, IEEE Computer Society, pp 499–510

  • scitools.com (2016) Understand tool. https://scitools.com, online; Accessed March 31st, 2016

  • Shihab E, Ihara A, Kamei Y, Ibrahim W M, Ohira M, Adams B, Hassan A E, Ki Matsumoto (2013) Studying re-opened bugs in open source software. Empir Softw Eng 18(5):1005–1042

    Article  Google Scholar 

  • SlideShare (2016) A. Laforge. Chrome release cycle. Job title: Technical Program Manager (Chrome) at Google. http://www.slideshare.net/Jolicloud/chrome-release-cycle, online; Accessed 06 February 2016

  • Śliwerski J, Zimmermann T, Zeller A (2005) When do changes induce fixes?. In: ACM Sigsoft software engineering notes, ACM, vol 30, pp 1–5

  • Sun C, Lo D, Wang X, Jiang J, Khoo SC (2010) A discriminative model approach for accurate duplicate bug report retrieval. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering. ACM, Vol 1, pp 45–54

  • Sun C, Lo D, Khoo SC, Jiang J (2011) Towards more accurate retrieval of duplicate bug reports. In: Proceedings of the 2011 26th IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, pp 253–262

  • Tan L, Liu C, Li Z, Wang X, Zhou Y, Zhai C (2014) Bug characteristics in open source software. Empir Softw Eng 19(6):1665–1705

    Article  Google Scholar 

  • The Bugzilla Guide (2017) The Bugzilla Guide. https://www.bugzilla.org/docs/2.20/html/bugreports.html, online; Accessed May 20th, 2018

  • Tian Y, Sun C, Lo D (2012) Improved duplicate bug report identification. In: 2012 16th European Conference on Software Maintenance and Reengineering (CSMR). IEEE, pp 385–390

  • Tourani P, Adams B (2016) The impact of human discussions on just-in-time quality assurance: an empirical study on openstack and eclipse. In: 2016 IEEE 23rd international conference on software analysis, evolution, and reengineering (SANER). IEEE, vol 1, pp 189–200

  • VMware (2017) JIRA. https://jira.atlassian.com/, accessed March 30th, 2017

  • Wang X, Zhang L, Xie T, Anvik J, Sun J (2008) An approach to detecting duplicate bug reports using natural language and execution information. In: 2008. ICSE’08. ACM/IEEE 30th International Conference on Software Engineering. IEEE, pp 461–470

  • Wikipedia (2018) Okapi BM25. https://en.wikipedia.org/wiki/Okapi_BM25, online; Accessed May 20th, 2018

  • Yin RK (2002) Case Study Research: Design and Methods, 3rd edn. SAGE Publications, New York

    Google Scholar 

  • YouTube (2014) Keynote of the 2014 Release Engineering conference. https://www.youtube.com/watch?v=Nffzkkdq7GM, online; Accessed March 30th, 2017

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for their detailed feedback and useful suggestions that greatly contributed to improving this paper. This work has been partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

Author information

Authors and Affiliations

  1. Mozilla Corporation, London, UK

    Marco Castelluccio

  2. University of Napoli Federico II, Naples, Italy

    Marco Castelluccio

  3. Polytechnique Montreal, Montreal, Québec, Canada

    Le An & Foutse Khomh

Authors
  1. Marco Castelluccio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Le An
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Foutse Khomh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Le An.

Additional information

Communicated by: Lu Zhang, Thomas Zimmermann, Xin Peng and Hong Mei

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Castelluccio, M., An, L. & Khomh, F. An empirical study of patch uplift in rapid release development pipelines. Empir Software Eng 24, 3008–3044 (2019). https://doi.org/10.1007/s10664-018-9665-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-018-9665-y

Keywords

Search

Navigation