How developers engage with static analysis tools in different contexts

  • Carmine VassalloEmail author
  • Sebastiano Panichella
  • Fabio Palomba
  • Sebastian Proksch
  • Harald C. Gall
  • Andy Zaidman
Part of the following topical collections:
  1. Software Analysis, Evolution and Reengineering (SANER)


Automatic static analysis tools (ASATs) are instruments that support code quality assessment by automatically detecting defects and design issues. Despite their popularity, they are characterized by (i) a high false positive rate and (ii) the low comprehensibility of the generated warnings. However, no prior studies have investigated the usage of ASATs in different development contexts (e.g., code reviews, regular development), nor how open source projects integrate ASATs into their workflows. These perspectives are paramount to improve the prioritization of the identified warnings. To shed light on the actual ASATs usage practices, in this paper we first survey 56 developers (66% from industry and 34% from open source projects) and interview 11 industrial experts leveraging ASATs in their workflow with the aim of understanding how they use ASATs in different contexts. Furthermore, to investigate how ASATs are being used in the workflows of open source projects, we manually inspect the contribution guidelines of 176 open-source systems and extract the ASATs’ configuration and build files from their corresponding GitHub repositories. Our study highlights that (i) 71% of developers do pay attention to different warning categories depending on the development context; (ii) 63% of our respondents rely on specific factors (e.g., team policies and composition) when prioritizing warnings to fix during their programming; and (iii) 66% of the projects define how to use specific ASATs, but only 37% enforce their usage for new contributions. The perceived relevance of ASATs varies between different projects and domains, which is a sign that ASATs use is still not a common practice. In conclusion, this study confirms previous findings on the unwillingness of developers to configure ASATs and it emphasizes the necessity to improve existing strategies for the selection and prioritization of ASATs warnings that are shown to developers.


Static analysis tools Development context Continuous integration Code review Empirical study 



The authors would like to thank all the open-source and industrial developers who responded to our survey, as well as the 11 industrial experts that participated to the semi-structured interviews. We also thank Diego Martin, which acted as external validator of the enforced ASATs and types of code checks. This research was partially supported by the Swiss National Science Foundation through the SNF Projects Nos. 200021-166275 and PP00P2_170529, and the Dutch Science Foundation NWO through the TestRoots project (project number 639.022.314).


  1. Hovemeyer D, Pugh W (2004) Finding Bugs is Easy. In: OOPSLA 2004, ACM, pp 132–136. CrossRefGoogle Scholar
  2. Al Shalabi L, Shaaban Z, Kasasbeh B (2006) Data mining: a preprocessing engine. J Comput Sci 2(9):735–739CrossRefGoogle Scholar
  3. Ayewah N, Pugh W, Hovemeyer D, Morgenthaler JD, Penix J (2008) Using static analysis to find bugs. IEEE Softw 25(5):22–29. CrossRefGoogle Scholar
  4. Ayewah N, Pugh W, Morgenthaler JD, Penix J, Zhou Y (2007) Evaluating static analysis defect warnings on production software. In: Das M, Grossman D (eds) Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on program analysis for software tools and engineering, PASTE’07, San Diego, California, USA, June 13-14, 2007. ACM, pp 1–8
  5. Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 712–721Google Scholar
  6. Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: Proceedings of the international conference on software engineering (ICSE). IEEE, pp 931–940
  7. Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern code reviews in open-source projects: which problems do they fix?. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 202–211Google Scholar
  8. Beller M, Bholanath R, McIntosh S, Zaidman A (2016) Analyzing the state of static analysis: a large-scale evaluation in open source software. In: IEEE 23rd international conference on software analysis, evolution, and reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016., vol 1. IEEE Computer Society, pp 470–481
  9. Beller M, Gousios G, Panichella A, Proksch S, Amann S, Zaidman A (2017) Developer testing in the IDE: patterns, beliefs, and behavior. IEEE Trans Softw Eng (TSE)Google Scholar
  10. Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how and why developers (do not) test in their IDEs. In: Proceedings of the joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 179–190Google Scholar
  11. Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test?. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE Computer Society, pp 559–562Google Scholar
  12. Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build: an explorative analysis of travis ci with github. In: International conference on mining software repositories. IEEE Press, pp 356–367Google Scholar
  13. Beller M, Gousios G, Zaidman A (2017) TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In: Proceedings of the 14th working conference on mining software repositories. IEEE, pp 447–450Google Scholar
  14. Bitbucket (2019) Accessed: 2019-03-10
  15. Bodden E (2018) Self-adaptive static analysis. In: Proceedings of the 40th international conference on software engineering: new ideas and emerging results, ICSE-NIER ’18. ACM, New York, pp 45–48
  16. Buckers T, Cao C, Doesburg M, Gong B, Wang S, Beller M, Zaidman A (2017) UAV: warnings from multiple automated static analysis tools at a glance. In: IEEE 24th international conference on software analysis, evolution and reengineering (SANER). IEEE Computer Society, pp 472–476Google Scholar
  17. Bundler (2019) Accessed: 2019-03-10
  18. Butler S, Wermelinger M, Yu Y, Sharp H (2010) Exploring the influence of identifier names on code quality: an empirical study. In: Proceedings of the European conference on software maintenance and reengineering (CSMR), pp 156–165Google Scholar
  19. Catolino G, Palomba F, De Lucia A, Ferrucci F, Zaidman A (2018) Enhancing change prediction models using developer-related factors. J Syst Softw 143:14–28CrossRefGoogle Scholar
  20. Checkmarx (2019) Accessed: 2019-03-10
  21. CheckStyle (2019) Accessed: 2019-03-10
  22. Chen L (2015) Continuous delivery: huge benefits, but challenges too. IEEE Softw 32(2):50–54CrossRefGoogle Scholar
  23. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46CrossRefGoogle Scholar
  24. Di Penta M, Cerulo L, Aversano L (2009) The life and death of statically detected vulnerabilities: an empirical study. Inf Softw Technol 51(10):1469–1484CrossRefGoogle Scholar
  25. Dias M, Cassou D, Ducasse S (2013) Representing code history with development environment events. In: International workshop on smalltalk technologiesGoogle Scholar
  26. Dillman DA, Smyth JD, Christian LM (2014) Internet, phone, mail, and mixed-mode surveys: the tailored design method. Wiley, New YorkGoogle Scholar
  27. D’silva V, Kroening D, Weissenbacher G (2008) A survey of automated techniques for formal software verification. IEEE Trans Comput Aided Des Integr Circuits Syst 27(7):1165–1178CrossRefGoogle Scholar
  28. Emanuelsson P, Nilsson U (2008) A comparative study of industrial static analysis tools. Electron Notes Theor Comput Sci 217:5–21CrossRefGoogle Scholar
  29. ESLint (2019) Accessed: 2019-03-10
  30. Findbugs (2019) Accessed: 2019-03-10
  31. flake8 (2019) Accessed: 2019-03-10
  32. Flanagan C, Leino KRM, Lillibridge M, Nelson G, Saxe JB, Stata R (2002) Extended static checking for java. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), pp 234–245Google Scholar
  33. Flow (2019) Accessed: 2019-03-10
  34. Gibbs L, Kealy M, Willis K, Green J, Welch N, Daly J (2007) What have sampling and data collection got to do with good qualitative research? Aust N Z J Public Health 31(6):540–544CrossRefGoogle Scholar
  35. Gerrit (2019) Accessed: 2019-03-10
  36. Github (2019) Accessed: 2019-03-10
  37. Gitlab (2019) Accessed: 2019-03-10
  38. Gousios G, Pinzger M, van Deursen A (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 345–355
  39. Gousios G, Zaidman A, Storey MA, van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE computer society, pp 358–368Google Scholar
  40. Gradle (2019) Accessed: 2019-03-10
  41. Heckman SS, Williams LA (2011) A systematic literature review of actionable alert identification techniques for automated static code analysis. Inf Softw Technol 53 (4):363–387. CrossRefGoogle Scholar
  42. Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering (ASE 2016). ACM, pp 426–437Google Scholar
  43. Coverity (2009) Effective management of static analysis vulnerabilities and defects. Accessed 20 Aug 2019
  44. Johnson B, Song Y, Murphy-Hill ER, Bowdidge RW (2013) Why don’t software developers use static analysis tools to find bugs? In: Notkin D, Cheng BHC, Pohl K (eds) 35th international conference on software engineering, ICSE’13, San Francisco, CA, USA, May 18-26, 2013. IEEE Computer Society, pp 672–681
  45. Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26CrossRefGoogle Scholar
  46. Johnson SC (1977) Lint, a C program checker. Bell Telephone Laboratories Murray HillGoogle Scholar
  47. Jørgensen M (2004) A review of studies on expert estimation of software development effort. J Syst Softw 70(1-2):37–60CrossRefGoogle Scholar
  48. JSHint (2019) Accessed: 2019-03-10
  49. Khoshgoftaar TM, Allen EB (1998) Classification of fault-prone software modules: prior probabilities, costs, and model evaluation. Empir Softw Eng 3(3):275–298CrossRefGoogle Scholar
  50. Kim S, Ernst MD (2007) Which warnings should I fix first?. In: Proceedings of the the 6th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, ESEC-FSE ’07. ACM, pp 45–54
  51. Krippendorff K (2004) Content analysis: an introduction to its methodology, 2nd edn. Sage, LondonGoogle Scholar
  52. Lehman MM (1980) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1:213–221. CrossRefGoogle Scholar
  53. Mahmood R, Mahmoud QH (2018) Evaluation of static analysis tools for finding vulnerabilities in java and C/C++ source code. arXiv:1805.09040
  54. Maven (2019) Accessed: 2019-03-10
  55. McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The impact of code review coverage and code review participation on software quality: a case study of the qt, vtk, and ITK projects. In: Proceedings of the working conference on mining software repositories (MSR), pp 192–201Google Scholar
  56. Muske T, Talluri R, Serebrenik A (2018) Repositioning of static analysis alarms. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018. ACM, New York, pp 187–197
  57. Nagappan N, Ball T (2005) Static analysis tools as early indicators of pre-release defect density. In: Proceedings of the international conference on software engineering (ICSE), pp 580–586Google Scholar
  58. Nanda MG, Gupta M, Sinha S, Chandra S, Schmidt D, Balachandran P (2010) Making defect-finding tools work for you. In: Proceedings of the international conference on software engineering (ASE), vol 2, pp 99–108Google Scholar
  59. Novak J, Krajnc A, žontar R (2010) Taxonomy of static code analysis tools. In: The 33rd international convention MIPRO, pp 418–422Google Scholar
  60. Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The role of patch review in software evolution: an analysis of the mozilla firefox. In: Proceedings of the joint international and annual ERCIM workshops on principles of software evolution (IWPSE) and software evolution (Evol) workshops, pp 9–18Google Scholar
  61. Oppenheim B (1992) Questionnaire design, interviewing and attitude measurement. Pinter Publishers, LondonGoogle Scholar
  62. Palomba F, Zanoni M, Fontana FA, De Lucia A, Oliveto R (2017) Toward a smell-aware bug prediction model. IEEE Trans Softw Eng 45(2):194–218CrossRefGoogle Scholar
  63. Panichella S, Arnaoudova V, Di Penta M, Antoniol G (2015) Would static analysis tools help developers with code reviews?. In: 22nd IEEE international conference on software analysis, evolution, and reengineering, SANER 2015, Montreal, QC, Canada, March 2-6, 2015., pp 161–170
  64. Parnas DL, Lawford M (2003) The role of inspection in software quality assurance. IEEE Trans Softw Eng 29(8):674–676CrossRefGoogle Scholar
  65. PEP8 online check (2019) Accessed: 2019-03-10
  66. PMD (2019) Accessed: 2019-03-10
  67. Prettier (2019) Accessed: 2019-03-10
  68. Proksch S, Nadi S, Amann S, Mezini M (2017) Enriching in-ide process information with fine-grained source code history. In: International conference on software analysis, evolution, and reengineeringGoogle Scholar
  69. Pylint (2019) Accessed: 2019-03-10
  70. Rahman F, Khatri S, Barr ET, Devanbu PT (2014) Comparing static bug finders and statistical prediction. In: Proceedings of the international conference on software engineering (ICSE), pp 424–434Google Scholar
  71. Reddit (2019) Accessed: 2019-03-10
  72. Rigby PC (2011) Understanding open source software peer review: review processes, parameters and statistical models, and underlying behaviours and mechanisms. Ph.D. thesis, University of Victoria, BC CanadaGoogle Scholar
  73. Rigby PC, German DM (2006) A preliminary examination of code review processes in open source projects. Tech. Rep. DCS-305-IR, University of VictoriaGoogle Scholar
  74. Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Engg 14(2):131–164. CrossRefGoogle Scholar
  75. Rubocop (2019) Accessed: 2019-03-10
  76. Ruthruff JR, Penix J, Morgenthaler JD, Elbaum S, Rothermel G (2008) Predicting accurate and actionable static analysis warnings: an experimental approach. In: Proceedings of the 30th international conference on software engineering. ACM, pp 341–350Google Scholar
  77. Sadowski C, Aftandilian E, Eagle A, Miller-Cushon L, Jaspan C (2018) Lessons from building static analysis tools at Google. Commun ACM 61(4):58–66. CrossRefGoogle Scholar
  78. Sadowski C, van Gogh J, Jaspan C, Söderberg E, Winter C (2015) Tricorder: building a program analysis ecosystem. In: Bertolino A, Canfora G, Elbaum SG (eds) 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16-24, 2015., vol 1. IEEE Computer Society, pp 598–608
  79. SBT (2019) Accessed: 2019-03-10
  80. SonarQube (2019) Accessed: 2019-03-10
  81. Spencer D (2009) Card sorting: designing usable categories. Rosenfeld MediaGoogle Scholar
  82. Thung F, Lucia L, Lo D, Jiang L, Rahman F, Devanbu PT (2012) To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In: Proceedings of the international conference on automated software engineering (ASE), pp 50–59Google Scholar
  83. Vassallo C, Palomba F, Bacchelli A, Gall HC (2018) Continuous code quality: are we (really) doing that?. In: ASE. ACM, pp 790–795Google Scholar
  84. Vassallo C, Palomba F, Gall HC (2018) Continuous refactoring in ci: a preliminary study on the perceived advantages and barriers. In: 34th IEEE international conference on software maintenance and evolution (ICSME)Google Scholar
  85. Vassallo C, Panichella S, Palomba F, Proksch S, Gall HC, Zaidman A (2019) Replication package for “How developers engage with static analysis tools in different contexts”.
  86. Vassallo C, Panichella S, Palomba F, Proksch S, Zaidman A, Gall HC (2018) Context is king: the developer perspective on the usage of static analysis tools. In: SANER. IEEE Computer Society, pp 38–49Google Scholar
  87. Vassallo C, Proksch S, Zemp T, Gall HC (2018) Un-break my build: assisting developers with build repair hints. In: International conference on program comprehension (ICPC). IEEEGoogle Scholar
  88. Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, Di Penta M, Panichella S (2017) A tale of CI build failures: an open source and a financial organization perspective. In: 2017 IEEE international conference on software maintenance and evolution, ICSME 2017, Shanghai, China, September 17-22, 2017. IEEE Computer Society, pp 183–193
  89. Vassallo C, Zampetti F, Romano D, Beller M, Panichella A, Di Penta M, Zaidman A (2016) Continuous delivery practices in a large financial organization. In: 32nd IEEE international conference on software maintenance and evolution (ICSME), pp 41–50Google Scholar
  90. Wagner S, Jürjens J, Koller C, Trischberger P (2005) Comparing bug finding tools with reviews and tests. In: Proceedings of the 17th IFIP TC6/WG 6.1 international conference on testing of communicating systems, pp 40–55Google Scholar
  91. Wikis (2019) Accessed: 2019-03-10
  92. Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: Proceedings of the 14th international conference on mining software repositories. IEEE Press, pp 334–344Google Scholar
  93. Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J, Vouk M (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng (TSE) 32(4):240–253CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.University of ZurichZurichSwitzerland
  2. 2.Zurich University of Applied ScienceZurichSwitzerland
  3. 3.DelftThe Netherlands

Personalised recommendations