Skip to main content
SpringerLink
Log in

On the usage, co-usage and migration of CI/CD tools: A qualitative analysis

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Continuous integration, delivery and deployment (CI/CD) is used to support the collaborative software development process. CI/CD tools automate a wide range of activities in the development workflow such as testing, linting, updating dependencies, creating and deploying releases, and so on. Previous quantitative studies have revealed important changes in the landscape of CI/CD usage, with the increasing popularity of cloud-based services, and many software projects migrating to other CI/CD tools. In order to understand the reasons behind these changes in CI/CD usage, this paper presents a qualitative study based on in-depth interviews with 22 experienced software practitioners reporting on their usage, co-usage and migration of 31 different CI/CD tools. Following an inductive and deductive coding process, we analyse the interviews and found a high amount of competition between CI/CD tools. We observe multiple reasons for co-using different CI/CD tools within the same project, and we identify the main reasons and detractors for migrating to different alternatives. Among all reported migrations, we observe a clear trend of migrations away from Travis and migrations towards GitHub Actions and we identify the main reasons behind them.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data Availability

All data generated or analysed during this study are included in this published article (and its supplementary information files). Except for two interview transcripts, the extracted information is available in the article content but the interview transcripts themselves are not accessible to the public due to the interviewees’ request.

Notes

  1. See https://docs.microsoft.com/en-us/azure/devops/boards/queries/history-and-auditing and https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history

  2. https://docs.microsoft.com/en-us/azure/devops/pipelines/release/?view=azure-devops

  3. https://circleci.com/docs/2.0/workspaces

  4. Azure’s Deployment History feature enables to go back in time to allow to redeploy a release that was for example available one year ago, just by selecting that deployment in the history. See https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/deployment-history?tabs=azure-portal.

  5. Whenever 3+ CI tools are used together, each pair of CI tools is reported individually.

  6. https://saucelabs.com/blog/announcing-travis-ci-for-mac-and-ios-powered-by-sauce-labs

  7. https://blog.travis-ci.com/2018-10-11-windows-early-release

  8. https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta

  9. https://about.gitlab.com/blog/2021/08/23/build-cloud-for-macos-beta

  10. https://nvd.nist.gov/vuln/detail/CVE-2021-41077

  11. https://dev.to/lyncozy/crypto-miners-are-killing-free-ci-4pii

  12. 12https://visualstudio.microsoft.com/services/github-codespaces/

References

  • Abdalkareem R, Mujahid S, Shihab E (2020) A machine learning approach to improve the detection of CI skip commits. IEEE Trans Softw Eng

  • Abdalkareem R, Mujahid S, Shihab E, Rilling J (2019) Which commits can be CI skipped? IEEE Trans Softw Eng 47(3):448–463

    Article  Google Scholar 

  • Ampatzoglou A, Bibi S, Avgeriou P, Verbeek M, Chatzigeorgiou A (2019) Identifying, categorizing and mitigating threats to validity in software engineering secondary studies. Inf Softw Technol 106:201–230

    Article  Google Scholar 

  • Beck K (2000) Extreme programming explained: embrace change. Addison-Wesley Professional

  • Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build an explorative analysis of Travis CI with GitHub. In: International conference on mining software repositories (MSR), pp 356–367

  • Bernardo JH, da Costa DA, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: International conference on mining software repositories (MSR), pp 131–141. IEEE

  • Betz RM, Walker RC (2013) Implementing continuous integration software in an established computational chemistry software package. In: International workshop on software engineering for computational science and engineering (SE-CSE), pp 68–74. IEEE

  • Chen L (2015) Continuous delivery: Huge benefits, but challengs too. IEEE Software - Special Issue on Release Engineering 32(2):50–54

    Google Scholar 

  • Chen L (2017) Continuous delivery: overcoming adoption challenges. J Syst Softw 128:72–86

    Article  Google Scholar 

  • Chen T, Zhang Y, Chen S, Wang T (2021) Let’s supercharge the workflows: an empirical study of GitHub actions. In: International conference on software quality, reliability and security companion (QRS-C), pp 01–10. IEEE

  • Decan A, Mens T, Mazrae PR, Golzadeh M (2022) On the use of GitHub actions in software development repositories. In: International conference on software maintenance and evolution (ICSME)

  • Duvall PM, Matyas S, Glover A (2007) Continuous integration: improving software quality and reducing risk. Addison-Wesley Professional

  • Elazhary O, Werner C, Li ZS, Lowlind D, Ernst NA, Storey M-A (2022) Uncovering the benefits and challenges of continuous integration practices. IEEE Trans Softw Eng 48(7):2570–2583

    Article  Google Scholar 

  • Esfahani H, Fietz J, Ke Q, Kolomiets A, Lan E, Mavrinac E, Schulte W, Sanches N, Kandula S (2016) CloudBuild: Microsoft’s distributed and caching build service. In: International conference on software engineering (ICSE), pp 11–20

  • Foundjem A, Constantinou E, Mens T, Adams B (2022) A mixed-methods analysis of micro-collaborative coding practices in OpenStack. Empir Softw Eng 27(5):120

    Article  Google Scholar 

  • Fowler M, Foemmel M (2006) Continuous integration

  • Fusch PI, Ness LR (2015) Are we there yet? Data saturation in qualitative research. The Qualitative Report, vol 20(9)

  • Gallaba K, Junqueira Y, Ewart J, Mcintosh S (2022) Accelerating continuous integration by caching environments and inferring dependencies. IEEE Trans Softw Eng 48(6):2040–2052

    Article  Google Scholar 

  • Ghaleb TA, Da Costa DA, Zou Y (2019) An empirical study of the long duration of continuous integration builds. Empir Softw Eng 24(4):2102–2139

    Article  Google Scholar 

  • Gmeiner J, Ramler R, Haslinger J (2015) Automated testing in the continuous delivery pipeline: a case study of an online company. In: International conference on software testing, verification and validation workshops (ICSTW), pp 1–6. IEEE

  • Golzadeh M, Decan A, Mens T (2022) On the rise and fall of CI services in GitHub. In: International conference on software analysis, evolution and Reengineering (SANER)

  • Guest G, Bunce A, Johnson L (2006) How many interviews are enough? An experiment with data saturation and variability. Field Methods 18 (1):59–82

    Article  Google Scholar 

  • Gupta Y, Khan Y, Gallaba K, McIntosh S (2017) The impact of the adoption of continuous integration on developer attraction and retention. In: International Conference on Mining Software Repositories (MSR), pages 491–494. IEEE

  • Hilton M, Nelson N, Tunnell T, Marinov D, Dig D (2017) Trade-offs in continuous integration: assurance, security, and flexibility. In: Joint meeting on foundations of software engineering (FSE), pp 197–207

  • Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: International conference on automated software engineering (ASE), pp 426–437. IEEE

  • Holmstrom H, Conchúir EÓ, Agerfalk J, Fitzgerald B (2006) Global software development challenges: A case study on temporal, geographical and socio-cultural distance. In: International conference on global software engineering (ICGSE), pp 3–11. IEEE

  • Jin X, Servant F (2020) A cost-efficient approach to building in continuous integration. In: International conference on software engineering (ICSE), pp 13–25. IEEE

  • Jin X, Servant F (2021) What helped, and what did not? An evaluation of the strategies to improve continuous integration. In: International conference on software engineering (ICSE), pp 213–225. IEEE

  • Jin X, Servant F (2022) Which builds are really safe to skip? Maximizing failure observation for build selection in continuous integration. J Syst Softw 188:111292

    Article  Google Scholar 

  • Kim M, Zimmermann T, DeLine R, Begel A (2016) The emerging role of data scientists on software development teams. In: International conference on software engineering (ICSE), pp 96–107. IEEE

  • Kinsman T, Wessel M, Gerosa MA, Treude C (2021) How do software developers use GitHub actions to automate their workflows?. In: International conference on mining software repositories (MSR)

  • Kulas M, Borelli JL, Gässler W, Peter D, Rabien S, de Xivry GO, Busoni L, Bonaglia M, Mazzoni T, Rahmer G (2014) Practical experience with test-driven development during commissioning of the multi-star AO system ARGOS. In: Software and Cyberinfrastructure for Astronomy III, vol 9152, pp 110–119. SPIE

  • Leppänen M, Mäkinen S, Pagels M, Eloranta V-P, Itkonen J, Mäntylä MV, Männistö T (2015) The highways and country roads to continuous deployment. IEEE Softw 32(2):64–72

    Article  Google Scholar 

  • Li Z, Liu W, Chen H, Wang X, Liao X, Xing L, Zha M, Jin H, Zou D (2022) Robbery on: DevOps understanding and mitigating illicit cryptomining on continuous integration service platforms. In: IEEE symposium on security and privacy (SP), pp 2397–2412. IEEE

  • Lu J, Yang Z, Qian J (2014) Implementation of continuous integration and automated testing in software development of smart grid scheduling support system. In: International conference on power system technology, pp 2441–2446. IEEE

  • Machiraju V (2021) Change in Azure pipelines grant for public projects. https://devblogs.microsoft.com/devops/change-in-azure-pipelines-grant-for-public-projects/. Accessed Oct 14 2022

  • Maxwell J (1992) Understanding and validity in qualitative research. Harvard educational review 62(3):279–301

    Article  Google Scholar 

  • Mendy M, Rios N, Rybinski M (2020) The new pricing model for travis-ci.com. https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing. Accessed Oct 14 2022

  • Meyer AN, Barr ET, Bird C, Zimmermann T (2019) Today was a good day: the daily life of software developers. IEEE Trans Softw Eng 47(5):863–880

    Article  Google Scholar 

  • Rahman A, Agrawal A, Krishna R, Sobran A (2018) Characterizing the influence of continuous integration: empirical results from 250+ open source and proprietary projects. In: ACM SIGSOFT international workshop on software Analytics, pp 8–14

  • Ralph P, Tempero E (2018) Construct validity in software engineering research and software metrics. In: International conference on evaluation and assessment in software engineering, pp 13–23

  • Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software. In: International conference on mining software repositories (MSR), pp 345–355. IEEE

  • Russel Bernard H, Wutich A, Gery WR (2016) Analyzing qualitative data: systematic approaches. SAGE Publications 2nd edn.

  • Savor T, Douglas M, Gentili M, Williams L, Beck K, Stumm M (2016) Continuous deployment at Facebook and OANDA. In: International conference on software engineering (ICSE), pp 21–30. IEEE

  • Shahin A, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943

    Article  Google Scholar 

  • Soares E, Sizilio G, Santos J, da Costa DA, Kulesza U (2022) The effects of continuous integration on software development: a systematic literature review. Empir Softw Eng 27(3):1–61

    Article  Google Scholar 

  • Ståhl D, Bosch J (2013) Experienced benefits of continuous integration in industry software product development: a case study. In: IASTED international conference on software engineering, pp 736–743

  • Valenzuela-Toledo P, Bergel A (2022) Evolution of GitHub Action workflows. In: International conference on software analysis, evolution and Reengineering (SANER). IEEE

  • Vasilescu B, Yu Y, Wang H, Devanbu P, Filkov V (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Joint meeting on foundations of software engineering (FSE), pp 805–816

  • Vassallo C, Palomba F (2018) Continuous refactoring in CI: a preliminary study on the perceived advantages and barriers. In: International conference on software maintenance and evolution (ICSME), pp 564–568. IEEE

  • Vassallo C, Proksch S, Gall HC, Di Penta M (2019) Automated reporting of anti-patterns and decay in continuous integration. In: International conference on software engineering (ICSE), pp 105–115. IEEE

  • Widder DG, Hilton M, Kästner C, Vasilescu B (2019) A conceptual replication of continuous integration pain points in the context of Travis CI. In: Joint meeting on european software engineering conference and symposium on the foundations of software engineering (ESEC/FSE), pp 647–658

  • Widder D, Vasilescu B, Hilton Ml, Kästner C (2018) I’m leaving you, Travis: a continuous integration breakup story. In: International conference on mining software repositories (MSR), pp 165–169. IEEE

  • Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: International conference on mining software repositories (MSR), pp 334–344. IEEE

Download references

Acknowledgements

This work is supported by the ARC-21/25 UMONS3 Action de Recherche Concertée financée par le Ministère de la Communauté française – Direction générale de l’Enseignement non obligatoire et de la Recherche scientifique, as well as by the Fonds de la Recherche Scientifique - FNRS under grant numbers O.0157.18F-RG43, T.0149.22 and F.4515.23.

Author information

Authors and Affiliations

  1. Software Engineering Lab, Université de Mons, Mons, Belgium

    Pooya Rostami Mazrae, Tom Mens, Mehdi Golzadeh & Alexandre Decan

Authors
  1. Pooya Rostami Mazrae
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tom Mens
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mehdi Golzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexandre Decan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pooya Rostami Mazrae.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Communicated by: Alexander Serebrenik

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Alexandre Decan (F.R.S.-FNRS Research Associate)

Appendices

Appendix A: Interview Questionnaire

The interview questions were structured in 6 categories. Some questions were conditional to the responses received on previous questions:

  1. 1.

    General questions about the respondent:

    1. a)

      Please briefly introduce yourself.

    2. b)

      Report on your past and current experience in collaborative software development, and on the kinds of projects you are or have been actively involved in, for which CI/CD tools have been used.

    3. c)

      What is or was your involvement in those projects?

    4. d)

      How many years of experience do you have with CI/CD?

  2. 2.

    General questions about CI/CD usage:

    1. a)

      When did you first start to use a CI/CD tool in those projects and what was the reason at that time?

    2. b)

      What are currently the main reasons for using CI/CD in those projects?

  3. 3

    Questions about specific CI/CD tool usage:

    1. a)

      Which different CI/CD tools have you used in the past, or are you currently using?

    2. b)

      Why did you or the project maintainers decide to use that particular CI/CD tool?

    3. c)

      What are the resources (in terms of budget, hardware, personnel, etc.) and effort that are or were available and required for creating, hosting and maintaining the CI/CD infrastructure for your projects?

    4. d)

      [If one of the reported CI/CD tools was Travis:]

      • Was Travis a kind of default choice, or was it a deliberate choice?

      • Are you aware of Travis’ changes in its free plan for public repositories? Has your project been affected by these changes?

    5. e)

      [If none of the reported CI/CD tools was Travis or GitHub Actions:] Why haven’t you ever used Travis or GitHub Actions?

    6. f)

      What were the main reasons for using these CI/CD tools, and what were/are the advantages and shortcomings of each of them according to your experience?

  4. 4.

    Questions about CI/CD migration: [These questions should be answered for every project that was reported by the respondent.]

    1. a)

      Did the project migrate from some CI/CD tool to another one during its lifetime?

    2. b)

      [In case of negative answer to 4.a:] Even if the project did not migrate its CI/CD tool, did you ever consider migrating to another CI/CD tool? If yes, why didn’t you carry out the migration?

    3. c)

      [In case of positive answer to 4.a:]

      • When did the project perform the migration?

      • From which CI/CD tool to which other CI/CD tool?

      • What drove the decision to migrate, and on which replacement CI/CD tool to adopt? (Was the migration because you disliked something in the existing CI/CD tool? Or because you liked something better in the replacement CI/CD tool?)

      • How much effort and time did it take to do the CI/CD migration and why?

      • What were the main difficulties (if any) in doing the migration?

      • How satisfied were you with the replacement CI/CD tool?

    4. d)

      [In case the respondent did not mention GitHub Actions as a CI/CD migration target:]

      • Are you aware of GitHub Actions and its increasing popularity? Why do you think this is the case?

      • Did you ever consider using GitHub Actions for doing CI/CD?

      • If not, why not? What is missing in GitHub Actions in order for the project to migrate to it?

    5. e)

      To what extent has the acquisition of GitHub by Microsoft in June 2018 affected you? Did it trigger you to migrate from one platform to another, for example from Github to GitLab or vice versa?

  5. 5.

    Questions about CI/CD tool co-usage:

    1. a)

      Did or does the same project use multiple different CI/CD tools simultaneously? Which ones?

    2. b)

      When and for how long have they been used together?

    3. c)

      What is or was the reason for using multiple CI/CD tools within the same project? What is or was the purpose of each CI/CD tool?

  6. 6.

    Closing open-ended question:

    1. a)

      Do you have any other important remarks related to CI/CD tool usage that you would like to share with us?

Appendix B: Mapping Between Respondents and CI/CD Tools

Throughout the article we have used respondent IDs whenever we cited relevant quotes from the interviews conducted with them. In order to put these quotes in the right perspective, the table below provides a mapping between the respondent IDs and the CI/CD tools that these respondents mentioned to have used somewhere during their career.

Table 8 Mapping between CI/CD tools and respondents having reported to use them
Full size table

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rostami Mazrae, P., Mens, T., Golzadeh, M. et al. On the usage, co-usage and migration of CI/CD tools: A qualitative analysis. Empir Software Eng 28, 52 (2023). https://doi.org/10.1007/s10664-022-10285-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-022-10285-5

Keywords

Search

Navigation