Abstract
Context
Continuous integration (CI) is a software engineering technique that proclaims frequent activities to assure the software product health. Researchers and practitioners mention several benefits related to CI. However, no systematic study surveys state of the art regarding such benefits or cons.
Objective
This study aims to identify and interpret empirical evidence regarding how CI impacts software development.
Method
Through a Systematic Literature Review, we search for studies in six digital libraries. Starting from 479 studies, we select 101 empirical studies that evaluate CI in the context of software development. We thoroughly read and extract information regarding (i) the CI environment, (ii) findings related to the effects of CI, and (iii) the employed research methods. We apply a thematic synthesis to group and summarize the findings.
Results
Existing research has explored the positive effects of CI, such as better cooperation, or negative effects, such as adding technical and process challenges. From our thematic synthesis, we identify six themes: development activities, software process, quality assurance, integration patterns, issues & defects, and build patterns.
Conclusions
Empirical research in CI has been increasing over recent years. We found that much of the existing research reveals that CI brings positive effects to software development. However, CI may also bring technical challenges to software development teams. Despite the overall positive outlook regarding CI, we still find room for improvements in the existing empirical research that evaluates the effects of CI.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
All data is available via an online appendix: https://doi.org/10.5281/zenodo.4545623.
References
Beck K, Andres C (2005) Extreme programming explained: embrace change. Addison-Wesley, Boston
Bernardo JH, Costa DAD, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: Proceedings of the 15th international conference on mining software repositories—MSR 18. https://doi.org/10.1145/3196398.3196421
Beller M, Gousios G, Zaidman A (2017) TravisTorrent: synthesizing travis CI and GitHub for full-stack research on continuous integration. In: Proceedings of the 14th working conference on mining software repositories. http://www.st.ewi.tudelft.nl/~mbeller/publications/2017_beller_gousios_zaidman_travistorrent_synthesizing_travis_ci_and_github_for_full-stack_research_on_continuous_integration.pdf Accessed 22 July 2020
Chen L, Babar MA, Zhang H (2010) Towards an evidence-based understanding of electronic data sources. https://doi.org/10.14236/ewic/ease2010.17
CI Theatre (2017) In: ThoughtWorks. https://www.thoughtworks.com/radar/techniques/ci-theatre. Accessed 25 Aug 2020
Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70:213–220. https://doi.org/10.1037/h0026256
Cruzes DS, Dybå T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 International symposium on empirical software engineering and measurement. https://doi.org/10.1109/esem.2011.36
Debbiche A, Dienér M, Svensson RB (2014) Challenges when adopting continuous integration: a case study. In: Product-focused software process improvement lecture notes in computer science. https://doi.org/10.1007/978-3-319-13835-0, pp 17–32
Dikert K, Paasivaara M, Lassenius C (2016) Challenges and success factors for large-scale agile transformations: a systematic literature review. J Syst Softw 119:87–108. https://doi.org/10.1016/j.jss.2016.06.013
Duvall PM, Matyas S, Glover A (2013) Continuous integration improving software quality and reducing risk. Addison-Wesley, Upper Saddle River
Duvall PM (2018) Continuous delivery—patterns and anti-patterns in the software lifecycle. In: dzone.com. https://dzone.com/refcardz/continuous-delivery-patterns. Accessed 7 Feb 2021
Dybå T, Dingsøyr T (2008) Strength of evidence in systematic reviews in software engineering. In: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement—ESEM 08. https://doi.org/10.1145/1414004.1414034
Dybå T, Dingsoyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: an experience report. In: First international symposium on empirical software engineering and measurement (ESEM 2007). https://doi.org/10.1109/esem.2007.59
Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, London, pp 285–311
Embury SM, Page C (2019) Effect of continuous integration on build health in undergraduate team projects. In: Software engineering aspects of continuous development and new paradigms of software production and deployment lecture notes in computer science. https://doi.org/10.1007/978-3-030-06019-0
Felidre W, Furtado L, Costa DAD et al (2019) Continuous integration theater. In: 2019 ACM/IEEE international symposium on empirical software engineering and measurement (ESEM). https://doi.org/10.1109/esem.2019.8870152
Fitzgerald B, Stol K-J (2017) Continuous software engineering: a roadmap and agenda. J Syst Softw 123:176–189. https://doi.org/10.1016/j.jss.2015.06.063
Fowler M (2006) Continuous integration. In: Martinfowler.com. https://martinfowler.com/articles/continuousIntegration.html. Accessed 29 Jun 2020
Fowler M (2013) Extremeprogramming. In: Martinfowler.com. https://martinfowler.com/bliki/ExtremeProgramming.html
Fowler M (2017) Continuousintegrationcertification. In: Martinfowler.com. https://martinfowler.com/bliki/ContinuousIntegrationCertification.html. Accessed: 26 Jun 2020
Ghaleb TA, Costa DAD, Zou Y (2019) An empirical study of the long duration of continuous integration builds. Empir Softw Eng 24:2102–2139. https://doi.org/10.1007/s10664-019-09695-9
Hilton M, Tunnell T, Huang K, et al. (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering—ASE 2016. https://doi.org/10.1145/2970276.2970358
Holmstrom H, Conchuir E, Agerfalk P, Fitzgerald B (2006) Global software development challenges: a case study on temporal, geographical and socio-cultural distance. In: 2006 IEEE international conference on global software engineering (ICGSE06). https://doi.org/10.1109/icgse.2006.261210
Johanssen JO, Kleebaum A, Paech B, Bruegge B (2018) Practitioners eye on continuous software engineering. In: Proceedings of the 2018 international conference on software and system process—ICSSP 18. https://doi.org/10.1145/3202710.3203150
Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.471. Accessed 26 June 2020
Kitchenham BA, Tore D, Magne J (2004) Evidence-based software engineering. https://doi.org/10.1109/ICSE.2004.1317449
Laukkanen E, Itkonen J, Lassenius C (2017) Problems, causes and solutions when adopting continuous delivery—a systematic literature review. Inf Softw Technol 82:55–79. https://doi.org/10.1016/j.infsof.2016.10.001
Leppanen M, Makinen S, Pagels M, et al. (2015) The highways and country roads to continuous deployment. IEEE Softw 32:64–72. https://doi.org/10.1109/ms.2015.50
Meedeniya DA, D I, Perera I (2019) Software artefacts consistency management towards continuous integration: a roadmap. Int J Adva Comput Sci Appl. https://doi.org/10.14569/ijacsa.2019.0100411
Pinto G, Castor F, Bonifacio R, Rebouças M (2018) Work practices and challenges in continuous integration: a survey with Travis CI users. Softw: Pract Exp 48:2223–2236. https://doi.org/10.1002/spe.2637
Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR). https://doi.org/10.1109/msr.2017.54
Robles G (2010) Replicating MSR: a study of the potential replicability of papers published in the mining software repositories proceedings. In: 2010 7th IEEE working conference on mining software repositories (MSR 2010). https://doi.org/10.1109/msr.2010.5463348
Rodríguez-Pérez G, Robles G, González-Barahona JM (2018) Reproducibility and credibility in empirical software engineering: a case study based on a systematic literature review of the use of the SZZ algorithm. Inf Softw Technol 99:164–176. https://doi.org/10.1016/j.infsof.2018.03.009
Rogers RO (2004) Scaling continuous integration. In: Extreme programming and agile processes in software engineering lecture notes in computer science, pp 68–76
Shahin M, Babar MA, Zhu L (2017) Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices. IEEE Access 5:3909–3943. https://doi.org/10.1109/access.2017.2685629
Soares E, Sizilio G, Santos J, Alencar D, Kulesza U (2021) SLR artifacts—continuous integration quality impacts (v.1.0.2) [Data set]. In: Zenodo. https://doi.org/10.5281/zenodo.4545623. Accessed: 10 Aug 2021
Ståhl D, Bosch J (2013) Experienced benefits of continuous integration in industry software product development: a case study. Artificial intelligence and applications/794: modelling, identification and control/795: parallel and distributed computing and networks/796: software engineering/792: web-based education. https://doi.org/10.2316/p.2013.796-012
Ståhl D, Bosch J (2014a) Modeling continuous integration practice differences in industry software development. J Syst Softw 87:48–59. https://doi.org/10.1016/j.jss.2013.08.032
Ståhl D, Bosch J (2014b) Automated software integration flows in industry: a multiple-case study. In: Companion proceedings of the 36th international conference on software engineering—ICSE Companion 2014. https://doi.org/10.1145/2591062.2591186
Vasilescu B, Yu Y, Wang H, et al. (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering—ESEC/FSE 2015. https://doi.org/10.1145/2786805.2786850
Vassallo C, Palomba F, Gall HC (2018) Continuous refactoring in CI: a preliminary study on the perceived advantages and barriers. In: 2018 IEEE international conference on software maintenance and evolution (ICSME). https://doi.org/10.1109/icsme.2018.00068
Vassallo C, Proksch S, Gall HC, Penta MD (2019) Automated reporting of anti-patterns and decay in continuous integration. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE). https://doi.org/10.1109/icse.2019.00028
Viggiato M, Oliveira J, Figueiredo E, et al. (2019) Understanding similarities and differences in software development practices across domains. In: 2019 ACM/IEEE 14th international conference on global software engineering (ICGSE). https://doi.org/10.1109/icgse.2019.00013
Volf Z, Shmueli E (2017) Screening heuristics for project gating systems. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering—ESEC/FSE 2017. https://doi.org/10.1145/3106237.3117766
Wheelan SA, Hochberger JM (1996) Validation studies of the group development questionnaire. Small Group Res 27(1):143–170
Yu Y, Yin G, Wang T et al (2016) Determinants of pull-based development in the context of continuous integration. Sci China Inf Sci. https://doi.org/10.1007/s11432-016-5595-8
Zampetti F, Scalabrino S, Oliveto R et al (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR). https://doi.org/10.1109/msr.2017.2
Zhao Y, Serebrenik A, Zhou Y, et al. (2017) The impact of continuous integration on other software development practices: a large-scale empirical study. In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE). https://doi.org/10.1109/ase.2017.8115619
Acknowledgements
This work is partially supported by INES (www.ines.org.br), CNPq grants 465614/2014-0 and 425211/2018-5, CAPES grant 88887.136410/2017- 00, and FACEPE grants APQ-0399-1.03/17 and PRONEX APQ/0388-1.03/14.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare that they have no conflict of interest.
Additional information
Communicated by: Irit Hadar
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Demographic Attributes
This appendix shows the demographic data of our primary studies. We discuss the evolution of studies over the years and describe the authors’ information next.
Evolution of Studies
CI emerged in the context of eXtreme Programming, a software development methodology that increased and became popular in the late 90s and early 00s (Fowler 2013). Indeed, we identify the first research efforts on CI in 2003. Figure 13 shows an increasing number of publications over the years, especially in the last five years. The majority of papers have been published in conference proceedings (69 papers, i.e., 67.2%), followed by 23 papers published in journals (i.e. 22.5%). 10 other studies have been published in workshops in the last years (i.e. 9.8%).
Publications by year and type of venue
Our primary studies have been published in 29 distinct conferences, 15 journals, and 7 workshops. Figure 14a shows that MSR (IEEE International Working Conference on Mining Software Repositories), ICSE (International Conference on Software Engineering), Agile Conference, and ESEC/FSE (European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering) are the conferences with the highest number of primary studies. Since we aim to collate the most claims possible related to CI, we do not necessarily focus on the goals of a venue (e.g., magazine-based publication). In a later stage, we analyse the rigour of the studies from which we find claims.
Publications in main venues on (a) conferences, (b) Workshops, and (c) Journals
As for workshops, Fig. 14b shows Conference XP (Scientific Workshops Proceedings), SWAN (International Workshop on Software Analytics), and RCoSE (International Workshop on Rapid Continuous Software Engineering) as the most frequent venues. Figure 14c shows that the journals with the highest frequency are Empirical Software Engineering, Information and Software Technology, and IEEE Software.
Paper Authors
The primary studies have 259 different authors involved altogether. Table 21 shows a ranking with those having the highest number of publications included as a primary study in our SLR. Jan Bosch is the most frequent author and all of the top 6 researchers remain active over the last years. Having described the demographic data of our primary studies, we now describe our obtained results.
Appendix B: Selected Studies
Rights and permissions
About this article
Cite this article
Soares, E., Sizilio, G., Santos, J. et al. The effects of continuous integration on software development: a systematic literature review. Empir Software Eng 27, 78 (2022). https://doi.org/10.1007/s10664-021-10114-1
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-021-10114-1