Skip to main content
SpringerLink
Log in

Every build you break: developer-oriented assistance for build failure resolution

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Continuous integration is an agile software development practice. Instead of integrating features right before a release, they are constantly being integrated into an automated build process. This shortens the release cycle, improves software quality, and reduces time to market. However, the whole process will come to a halt when a commit breaks the build, which can happen for several reasons, e.g., compilation errors or test failures, and fixing the build suddenly becomes a top priority. Developers not only have to find the cause of the build break and fix it, but they have to be quick in all of it to avoid a delay for others. Unfortunately, these steps require deep knowledge and are often time-consuming. To support developers in fixing a build break, we propose Bart, a tool that summarizes the reasons for Maven build failures and suggests possible solutions found on the internet. We will show in a case study with 17 participants that developers find Bart useful to understand build breaks and that using Bart substantially reduces the time to fix a build break, on average by 37%. We have also conducted a qualitative study to better understand the workflows and information needs when fixing builds. We found that typical workflows differ substantially between various error categories, and that several uncommon build errors are both very hard to investigate and to fix. These findings will be useful to inform future research in this area.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Active JPA (2018) A simple active record pattern library in Java that makes programming DAL easier. https://github.com/ActiveJpa/activejpa/. Accessed: 2018-02-08

  • BART (2019) Jenkins-Plugin. https://plugins.jenkins.io/bart. Accessed: 2019-07-24

  • Bavota G, Gravino C, Oliveto R, De Lucia A, Tortora G, Genero M, Cruz-Lemus JA (2011) Identifying the weaknesses of uml class diagrams during data model comprehension. In: Proceedings of the 14th international conference on model driven engineering languages and systems, MODELS’11. http://dl.acm.org/citation.cfm?id=2050655.2050673. Springer, Berlin, pp 168–182

  • Beller M, Bholanath R, McIntosh S, Zaidman A (2016) Analyzing the state of static analysis: a large-scale evaluation in open source software. In: IEEE 23rd International conference on software analysis, evolution, and reengineering (SANER), pp 470–481. https://doi.org/10.1109/SANER.2016.105

  • Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build: an explorative analysis of Travis CI with GitHub. In: International conference on mining software repositories

  • Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20(1):37–46

    Article  Google Scholar 

  • Duvall P, Matyas SM, Glover A (2007) Continuous integration: improving software quality and reducing risk. Addison-Wesley

  • Everitt B (2002) The Cambridge dictionary of statistics. Cambridge University Press, Cambridge. http://www.worldcat.org/search?qt=worldcat_org_all&q=052181099X

    MATH  Google Scholar 

  • Fongo (2018) Faked out in-memory Mongo for Java. https://github.com/fakemongo/fongo/. Accessed: 2018-02-08

  • Fraser G, Staats M, McMinn P, Arcuri A, Padberg F (2015) Does automated unit test generation really help software testers? a controlled empirical study. ACM Trans Softw Eng Methodol (TOSEM) 24(4):23

    Article  Google Scholar 

  • Gallaba K, McIntosh S (2018) Use and misuse of continuous integration features: An empirical study of projects that (mis)use travis ci. IEEE Trans Softw Eng, 1–1. https://doi.org/10.1109/TSE.2018.2838131

  • Gallaba K, Macho C, Pinzger M, McIntosh S (2018) Noise and heterogeneity in historical build data: an empirical study of travis CI. In: ASE. ACM, pp 87–97

  • Haiduc S, Aponte J, Marcus A (2010) Supporting program comprehension with source code summarization. In: ICSE (2)

  • Hassan F, Wang X (2018) Hirebuild: an automatic approach to history-driven repair of build scripts. In: ICSE. ACM, pp 1078–1089

  • Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering (ASE), pp 426–437

  • Hilton M, Nelson N, Tunnell T, Marinov D, Dig D (2017) Trade-offs in continuous integration: assurance, security, and flexibility. In: Proceedings of the 25th ACM SIGSOFT international symposium on foundations of software engineering, FSE 2017, p. To Appear

  • Humble J, Farley D (2010) Continuous delivery, reliable software releases through build, test, and deployment automation. Addison-Wesley Professional

  • Kerzazi N, Khomh F, Adams B (2014) Why do automated builds break? An empirical study. In: 30th IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 41–50, DOI https://doi.org/10.1109/ICSME.2014.26, (to appear in print)

  • LaToza TD, Venolia G, DeLine R (2006) Maintaining mental models: a study of developer work habits. In: Proceedings of the 28th international conference on software engineering, ICSE ’06. https://doi.org/10.1145/1134285.1134355. ACM, New York, pp 492–501

  • Likert R (1932) A technique for the measurement of attitudes. Archives of psychology

  • Log parser plugin (2018) https://wiki.jenkins.io/display/JENKINS/Log+Parser+Plugin. Accessed: 2018-02-08

  • Lou Y, Chen J, Zhang L, Hao D, Zhang L (2019) History-driven build failure fixing: how far are we?. In: ISSTA. ACM, pp 43–54

  • Macho C, McIntosh S, Pinzger M (2018) Automatically repairing dependency-related build breakage. In: Proc. of the international conference on software analysis, evolution, and reengineering (SANER), p. To appear

  • Maple S (2016) Java tools and technologies landscape report 2016. ZeroTurnaround post. https://zeroturnaround.com/rebellabs/java-tools-and-technologies-landscape-2016/

  • Maven (2018) http://maven.apache.org/. Accessed: 2018-02-08

  • Miller A (2008) A hundred days of continuous integration. In: Proceedings of the Agile 2008, AGILE ’08, pp 289–293

  • Moreno L, Marcus A (2017) Automatic software summarization: the state of the art. In: ICSE (companion volume). IEEE Computer Society, pp 511–512

  • Moreno L, Aponte J, Sridhara G, Marcus A, Pollock LL, Vijay-Shanker K (2013) Automatic generation of natural language summaries for java classes. In: ICPC. IEEE Computer Society, pp 23–32

  • Moreno L, Bavota G, Penta MD, Oliveto R, Marcus A (2015) How can I use this method?. In: ICSE (1). IEEE Computer Society, pp 880–890

  • Myers GJ (2004) The art of software testing, 2. edn. Wiley

  • Panichella S, Panichella A, Beller M, Zaidman A, Gall HC (2016) The impact of test case summaries on bug fixing performance: an empirical investigation. In: ICSE. ACM, pp 547–558

  • Ponzanelli L, Bavota G, Penta MD, Oliveto R, Lanza M (2014) Mining Stackoverflow to turn the ide into a self-confident programming prompter. In: MSR

  • Rahman A, Parnin C, Williams L (2019) The seven sins: security smells in infrastructure as code scripts. In: 41st International conference on software engineering (ICSE). IEEE/ACM

  • Rastkar S, Murphy GC, Murray G (2010) Summarizing software artifacts: a case study of bug reports. In: ICSE (1). ACM, pp 505–514

  • Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of java-based open-source software. In: Proceedings of the 14th international conference on mining software repositories, MSR’17. ACM, New York, p nn

  • Reddit (2018) https://www.reddit.com/. Accessed: 2018-02-08

  • Robbins NB, Heiberger RM (2011) Plotting Likert and other rating scales. In: Proceedings of the 2011 joint statistical meeting, pp 1058–1066

  • Robinson D (2003) An introduction to abstract algebra. De Gruyter textbook. Walter de Gruyter. https://books.google.it/books?id=Yj3ApD8TeCUC

  • Sentry Java (2018) A sentry SDK for Java and other JVM languages. https://github.com/getsentry/sentry-java/. Accessed: 2018-02-08

  • Seo H, Sadowski C, Elbaum SG, Aftandilian E, Bowdidge RW (2014) Programmers’ build errors: a case study (at Google). In: Proc. Int’l conf on software engineering (ICSE). https://doi.org/10.1145/2568225.2568255, pp 724–734

  • Sorbo AD, Panichella S, Alexandru CV, Shimagaki J, Visaggio CA, Canfora G, Gall HC (2016) What would users change in my app? summarizing app reviews for recommending software changes. In: SIGSOFT FSE. ACM, pp 499–510

  • Spencer D (2009) Card sorting: designing usable categories. Rosenfeld Media

  • StackOverflow (2018) Maven. https://stackoverflow.com/questions/tagged/maven. Accessed: 2018-02-08

  • Treude C, Barzilay O, Storey MA (2011) How do programmers ask and answer questions on the web? (nier track). In: Proceedings of the 33rd international conference on software engineering, ICSE ’11. https://doi.org/10.1145/1985793.1985907. ACM, New York, pp 804–807

  • Urli S, Yu Z, Seinturier L, Monperrus M (2018) How to design a program repair bot? Insights from the repairnator project. arXiv:1811.09852

  • Vasilescu B, Filkov V, Serebrenik A (2013) Stackoverflow and github: associations between software development and crowdsourced knowledge. In: SocialCom. IEEE Computer Society, pp 188–195

  • Vassallo C, Panichella S, Penta MD, Canfora G (2014) CODES: mining source code descriptions from developers discussions. In: ICPC. ACM, pp 106–109

  • Vassallo C, Zampetti F, Romano D, Beller M, Panichella A, Di Penta M, Zaidman A (2016) Continuous delivery practices in a large financial organization. In: 32nd IEEE International conference on software maintenance and evolution (ICSME), pp 41–50

  • Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, Penta MD, Panichella S (2017) A tale of CI build failures: an open source and a financial organization perspective. In: 2017 IEEE International conference on software maintenance and evolution, ICSME 2017, Shanghai, China, September 17-22, 2017. https://doi.org/10.1109/ICSME.2017.67, pp 183–193

  • Vassallo C, Proksch S, Zemp T, Gall HC (2018) Un-break my build: assisting developers with build repair hints. In: International conference on program comprehension

  • Vassallo C, Proksch S, Gall HC, Penta MD (2019a) Automated reporting of anti-patterns and decay in continuous integration. In: ICSE. IEEE / ACM, pp 105–115

  • Vassallo C, Proksch S, Zemp T, Gall HC (2019b) Replication package for “Every build you break: developer-oriented assistance for build failure resolution”. https://doi.org/10.5281/zenodo.3346615

  • Vos TEJ, Tonella P, Prasetya W, Kruse PM, Bagnato A, Harman M, Shehory O (2014) FITTEST: a new continuous and automated testing process for future internet applications. In: CSMR-WCRE. IEEE Computer Society, pp 407–410

  • Wong E, Yang J, Tan L (2013) Autocomment: mining question and answer sites for automatic comment generation. In: ASE. IEEE, pp 562–567

  • Ying ATT, Robillard MP (2013) Code fragment summarization. In: ESEC/SIGSOFT FSE. ACM, pp 655–658

  • Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: Proceedings of the 14th international conference on mining software repositories. IEEE Press, pp 334–344

Download references

Acknowledgements

We would like to thank all the study participants. C. Vassallo and H. Gall acknowledge the support of the Swiss National Science Foundation for their project SURF-MobileAppsData (SNF Project No. 200021-166275).

Author information

Authors and Affiliations

  1. University of Zurich, Zurich, Switzerland

    Carmine Vassallo, Sebastian Proksch, Timothy Zemp & Harald C. Gall

Authors
  1. Carmine Vassallo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Proksch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timothy Zemp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Harald C. Gall
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carmine Vassallo.

Additional information

Communicated by: Chanchal Roy, Janet Siegmund, and David Lo

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vassallo, C., Proksch, S., Zemp, T. et al. Every build you break: developer-oriented assistance for build failure resolution. Empir Software Eng 25, 2218–2257 (2020). https://doi.org/10.1007/s10664-019-09765-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09765-y

Keywords

Search

Navigation