Skip to main content
SpringerLink
Log in

The impact of a continuous integration service on the delivery time of merged pull requests

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Continuous Integration (CI) is a software development practice that builds and tests software frequently (e.g., at every push). One main motivator to adopt CI is the potential to deliver software functionalities more quickly than not using CI. However, there is little empirical evidence to support that CI helps projects deliver software functionalities more quickly. Through the analysis of 162,653 pull requests (PRs) of 87 GitHub projects, we empirically study whether adopting a CI service (TravisCI) can quicken the time to deliver merged PRs. We complement our quantitative study by analyzing 450 survey responses from participants of 73 software projects. Our results reveal that adopting a CI service may not necessarily quicken the delivery of merge PRs. Instead, the pivotal benefit of a CI service is to improve the decision making on PR submissions, without compromising the quality or overloading the project’s reviewers and maintainers. The automation provided by CI and the boost in developers’ confidence are key advantages of adopting a CI service. Furthermore, open-source projects planning to attract and retain developers should consider the use of a CI service in their project, since CI is perceived to lower the contribution barrier while making contributors feel more confident and engaged in the project.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data availability

For replication purposes, we publicize our datasets and results to the interested researcher: https://prdeliverydelay.github.io/#datasets

Notes

  1. https://prdeliverydelay.github.io/#studied-projects

  2. https://travis-ci.org/

  3. http://github.com/haraka/haraka

  4. https://prdeliverydelay.github.io/#rq3-r-squared-and-optimism

  5. https://prdeliverydelay.github.io/#rq3-variables-explanatory-power

  6. https://prdeliverydelay.github.io/#datasets

References

  • Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: 2013 35th International Conference on Software Engineering (ICSE), IEEE, pp 712–721

  • Bavota G, Russo B (2015) Four eyes are better than two: On the impact of code reviews on software quality. In: 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 81–90

  • Beck K (2000) Extreme Programming Explained Embrace Change. Addison-Wesley Professional

  • Bernardo JH, da Costa DA, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), IEEE, pp 131–141

  • Best D, Roberts D (1975) Algorithm as 89: the upper tail probabilities of spearman’s rho. Journal of the Royal Statistical Society Series C (Applied Statistics) 24(3):377–379

    Google Scholar 

  • Braun V, Clarke V (2006) Using thematic analysis in psychology. Qualitative research in psychology 3(2):77–101

    Article  Google Scholar 

  • Cassee N, Vasilescu B, Serebrenik A (2020) The silent helper: the impact of continuous integration on code reviews. 2020 IEEE 27th International Conference on Software Analysis. Evolution and Reengineering (SANER), IEEE, pp 423–434

  • Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological Bulletin 114(3):494

    Article  Google Scholar 

  • Coelho J, Valente MT (2017) Why modern open source projects fail. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pp 186–196

  • da Costa DA, Abebe SL, McIntosh S, Kulesza U, Hassan AE (2014) An empirical study of delays in the integration of addressed issues. In: Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on, IEEE, pp 281–290

  • da Costa DA, McIntosh S, Kulesza U, Hassan AE (2016) The impact of switching to a rapid release cycle on the integration delay of addressed issues An empirical study of the mozilla firefox project. In: Proceedings of the 13th International Conference on Mining Software Repositories, ACM, New York, NY, USA, MSR ’16, pp 374–385

  • da Costa DA, McIntosh S, Treude C, Kulesza U, Hassan AE (2018) The impact of rapid release cycles on the integration delay of fixed issues. Empirical Software Engineering 23(2):835–904

    Article  Google Scholar 

  • Debbiche A, Dienér M, Svensson RB (2014) Challenges when adopting continuous integration: A case study. In: International Conference on Product-Focused Software Process Improvement, Springer, pp 17–32

  • Duvall P, Matyas SM, Glover A (2007) Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional

  • Felidré W, Furtado L, da Costa DA, Cartaxo B, Pinto G (2019) Continuous integration theater. In: 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), IEEE, pp 1–10

  • Fitzgerald B, Stol KJ (2017) Continuous software engineering: A roadmap and agenda. J Syst Softw 123:176–189

    Article  Google Scholar 

  • Fowler M, Foemmel M (2006) Continuous integration. Thought-Works) http://www.thoughtworks.com /Continuous Integration pdf p 122

  • Gallaba K, McIntosh S (2018) Use and misuse of continuous integration features: An empirical study of projects that (mis) use travis ci. IEEE Trans Softw Eng 46(1):33–50

    Article  Google Scholar 

  • Giger E, Pinzger M, Gall H (2010) Predicting the fix time of bugs. In: Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, ACM, pp 52–56

  • Goodman D, Elbaz M (2008) " it’s not the pants, it’s the people in the pants" learnings from the gap agile transformation what worked, how we did it, and what still puzzles us. In: Agile 2008 Conference, IEEE, pp 112–115

  • Gousios G, Zaidman A, Storey MA, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of the 37th International Conference on Software Engineering-Volume 1, IEEE Press, pp 358–368

  • Guo Y, Leitner P (2019) Studying the impact of ci on pull request delivery time in open source projects-a conceptual replication. PeerJ Comput Sci 5

  • Harrell F (2015) Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. Springer

  • Hars A, Shaosong O (2002) Working for free? motivations for participating in open-source projects. Int J Electron Commer 6(3):25–39

    Article  Google Scholar 

  • Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering ASE 2016

  • Humble J, Farley D (2010) Continuous delivery: reliable software releases through build, test, and deployment automation. Pearson Education

  • Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast? case study on the linux kernel. In: Mining Software Repositories (MSR), 2013 10th IEEE Working Conference on, IEEE, pp 101–110

  • Karvonen T, Behutiye W, Oivo M, Kuvaja P (2017) Systematic literature review on the impacts of agile release engineering practices. Inf Softw Technol 86:87–100

    Article  Google Scholar 

  • Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. biometrics pp 159–174

  • Laukkanen E, Paasivaara M, Arvonen T (2015) Stakeholder perceptions of the adoption of continuous integration – a case study. In: Proceedings of the 2015 Agile Conference, IEEE Computer Society, AGILE ’15, pp 11–20

  • Mann HB, Whitney DR (1947) On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Ann Math Stat 18(1):50–60. https://doi.org/10.1214/aoms/1177730491

    Article  MathSciNet  Google Scholar 

  • Michlmayr M, Fitzgerald B, Stol KJ (2015) Why and how should open source projects adopt time-based releases? IEEE Software 32(2):55–63

    Article  Google Scholar 

  • Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Software Engineering, 2005. ICSE 2005. Proceedings. 27th International Conference on, IEEE, pp 284–292

  • Neely S, Stolt S (2013) Continuous delivery? easy! just change everything (well, maybe it is not that easy). In: 2013 Agile Conference, IEEE, pp 21–128

  • Nery GS, da Costa DA, Kulesza U (2019) An empirical study of the relationship between continuous integration and test code evolution. In: 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME), IEEE, pp 426–436

  • Nowell LS, Norris JM, White DE, Moules NJ (2017) Thematic analysis: Striving to meet the trustworthiness criteria. Int J Qual Methods 16(1):1609406917733847

    Article  Google Scholar 

  • Perry DE, Porter AA, Votta LG (2000) Empirical studies of software engineering: A roadmap. In: Proceedings of the Conference on The Future of Software Engineering, ACM, ICSE ’00, pp 345–355

  • Rahman AAU, Helms E, Williams L, Parnin C (2015) Synthesizing continuous deployment practices used in software development. In: 2015 Agile Conference, IEEE, pp 1–10

  • Rahman MM, Roy CK (2017) Impact of continuous integration on code reviews. In: 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), IEEE, pp 499–502

  • Romano J, Kromrey J, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys? In: annual meeting of the Florida Association of Institutional Research, pp 1–3

  • Santos J, Alencar da Costa D, Kulesza U (2022) Investigating the impact of continuous integration practices on the productivity and quality of open-source projects. In: Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp 137–147

  • Schroter A, Schröter A, Bettenburg N, Premraj R (2010) Do stack traces help developers fix bugs? In: Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on, IEEE, pp 118–121

  • Shahin M, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943

    Article  Google Scholar 

  • Sharma P, Savarimuthu T, Stanger N (2021) Influence of roles in decision-making during oss development-a study of python. In: Evaluation and Assessment in Software Engineering, pp 50–59

  • Shihab E, Ihara A, Kamei Y, Ibrahim WM, Ohira M, Adams B, Hassan AE, Matsumoto Ki (2010) Predicting re-opened bugs: A case study on the eclipse project. In: Reverse Engineering (WCRE), 2010 17th Working Conference on, IEEE, pp 249–258

  • da Silva ACBG, de Figueiredo Carneiro G, de Paula ACM, Monteiro MP, e Abreu FB (2016) Agility and quality attributes in open source software projects release practices. In: 2016 10th International Conference on the Quality of Information and Communications Technology (QUATIC), IEEE, pp 107–112

  • Soares DM, de Lima Júnior ML, Plastino A, Murta L (2018) What factors influence the reviewer assignment to pull requests? Inf Softw Technol 98:32–43

    Article  Google Scholar 

  • Soares E, Sizilio G, Santos J, da Costa DA, Kulesza U (2022) The effects of continuous integration on software development: a systematic literature review. Empir Softw Eng 27(3):1–61

    Article  Google Scholar 

  • Ståhl D, Bosch J (2014) Modeling continuous integration practice differences in industry software development. J Syst Softw 87:48–59

    Article  Google Scholar 

  • Vasilescu B, Van Schuylenburg S, Wulms J, Serebrenik A, van den Brand MG (2014) Continuous integration in a social-coding world: Empirical evidence from github. In: Software Maintenance and Evolution (ICSME), 2014 IEEE International Conference on, IEEE, pp 401–405

  • Vasilescu B, Yu Y, Wang H, Devanbu P, Filkov V (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering ESEC/FSE 2015

  • Weißgerber P, Neu D, Diehl S (2008) Small patches get in! In: Proceedings of the 2008 international working conference on Mining software repositories, pp 67–76

  • Wilks DS (2011) Statistical methods in the atmospheric sciences, vol 100. Academic press

  • Williamson DF, Parker RA, Kendrick JS (1989) The box plot: A simple visual method to interpret data. Ann Intern Med 110:916–921

    Article  Google Scholar 

  • Yu Y, Yin G, Wang T, Yang C, Wang H (2016) Determinants of pull-based development in the context of continuous integration. Sci China Inf Sci 59(8)

  • Zampetti F, Bavota G, Canfora G, Di Penta M (2019) A study on the interplay between pull request review and continuous integration builds. 2019 IEEE 26th International Conference on Software Analysis. Evolution and Reengineering (SANER), IEEE, pp 38–48

  • Zhang X, Yu Y, Georgios G, Rastogi A (2022) Pull request decisions explained: An empirical overview. IEEE Trans Softw Eng 49:849–871

    Article  Google Scholar 

  • Zhang X, Yu Y, Wang T, Rastogi A, Wang H (2022) Pull request latency explained: An empirical overview. Empir Softw Eng 27(6):1–38

    Article  Google Scholar 

  • Zhao Y, Serebrenik A, Zhou Y, Filkov V, Vasilescu B (2017) The impact of continuous integration on other software development practices: a large-scale empirical study. In: Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, IEEE Press, pp 60–71

Download references

Acknowledgements

This work is partially supported by INES (www.ines.org.br), CNPq grants 465614/2014-0 and 425211/2018-5, CAPES grant 88887.136410/2017-00, FACEPE grants APQ-0399-1.03/17, and PRONEX APQ/0388-1.03/14.

Author information

Authors and Affiliations

  1. Federal Institute of Rio Grande do Norte (IFRN), Macau, Brazil

    João Helis Bernardo

  2. Federal University of Rio Grande do Norte (UFRN), Natal, Brazil

    João Helis Bernardo & Uirá Kulesza

  3. University of Otago, Dunedin, New Zealand

    Daniel Alencar da Costa

  4. University of Melbourne, Melbourne, Australia

    Christoph Treude

Authors
  1. João Helis Bernardo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Alencar da Costa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Uirá Kulesza
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Treude
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to João Helis Bernardo.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Communicated by: Klaas-Jan Stol.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Project Survey Example

figure k
figure l
figure m
figure n
figure o
figure p
figure q
figure r
figure s
figure t
figure u

Appendix B: Invitation Letter

MAIL SUBJECT: Why do PRs take so long to be delivered? Research survey

Dear $contributor.name,

We are a group of researchers from universities based in Brazil, Australia, and New Zealand. We are studying the impact of Continuous Integration on the time to release merged pull requests to end users of open source projects.

We have collected public data from the project $project.fullName in the period from

$project.creationDate to 2016-11-11. According to our data, you have contributed $contributor.deliveredPRsCount pull-requests to $project.fullName which were effectively merged and delivered to end users.

As you were a contributor of the project $project.fullName, we would appreciate if you shared your experience with us by answering a few questions in the following survey:

Table 12 Number of participants per project and their IDs (PART I)
Full size table

Google Form: Understanding Delivery Time of Pull Requests

The survey has 24 questions (all of them are optional) and will take less than 15 minutes to complete. To compensate you for your time, all participants that answer all questions will be entered into a draw of six $50 Amazon gift cards.

Best Regards,

João Helis Bernardo.

PhD student at the Federal University of Rio Grande do Norte, Brazil.

Appendix C: Number of Participants Per Project

The number of participants per project are distributed in Tables 12 and 13.

Table 13 Number of participants per project and their IDs (PART II)
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bernardo, J.H., da Costa, D.A., Kulesza, U. et al. The impact of a continuous integration service on the delivery time of merged pull requests. Empir Software Eng 28, 97 (2023). https://doi.org/10.1007/s10664-023-10327-6

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10327-6

Keywords

Search

Navigation