Skip to main content
Log in

How developers engage with static analysis tools in different contexts

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Automatic static analysis tools (ASATs) are instruments that support code quality assessment by automatically detecting defects and design issues. Despite their popularity, they are characterized by (i) a high false positive rate and (ii) the low comprehensibility of the generated warnings. However, no prior studies have investigated the usage of ASATs in different development contexts (e.g., code reviews, regular development), nor how open source projects integrate ASATs into their workflows. These perspectives are paramount to improve the prioritization of the identified warnings. To shed light on the actual ASATs usage practices, in this paper we first survey 56 developers (66% from industry and 34% from open source projects) and interview 11 industrial experts leveraging ASATs in their workflow with the aim of understanding how they use ASATs in different contexts. Furthermore, to investigate how ASATs are being used in the workflows of open source projects, we manually inspect the contribution guidelines of 176 open-source systems and extract the ASATs’ configuration and build files from their corresponding GitHub repositories. Our study highlights that (i) 71% of developers do pay attention to different warning categories depending on the development context; (ii) 63% of our respondents rely on specific factors (e.g., team policies and composition) when prioritizing warnings to fix during their programming; and (iii) 66% of the projects define how to use specific ASATs, but only 37% enforce their usage for new contributions. The perceived relevance of ASATs varies between different projects and domains, which is a sign that ASATs use is still not a common practice. In conclusion, this study confirms previous findings on the unwillingness of developers to configure ASATs and it emphasizes the necessity to improve existing strategies for the selection and prioritization of ASATs warnings that are shown to developers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Compared to our previous work (Vassallo et al. 2018), we collected 14 more participants

  2. In the rest of the paper, we omit the word “usage” while referring to the definition and enforcement of ASATs usage for the sake of better readability.

  3. https://gsuite.google.com/products/forms/

  4. https://github.com/stympy/faker/blob/master/CONTRIBUTING.md

  5. 0.2 < k ≤ 0.4 is considered fair, 0.4 < k ≤ 0.6 moderate, 0.6 < k ≤ 0.8 strong, and k > 0.8 almost perfect (Cohen 1960)

References

  • Hovemeyer D, Pugh W (2004) Finding Bugs is Easy. In: OOPSLA 2004, ACM, pp 132–136. http://doi.acm.org/10.1145/1028664.1028717

    Article  Google Scholar 

  • Al Shalabi L, Shaaban Z, Kasasbeh B (2006) Data mining: a preprocessing engine. J Comput Sci 2(9):735–739

    Article  Google Scholar 

  • Ayewah N, Pugh W, Hovemeyer D, Morgenthaler JD, Penix J (2008) Using static analysis to find bugs. IEEE Softw 25(5):22–29. https://doi.org/10.1109/MS.2008.130

    Article  Google Scholar 

  • Ayewah N, Pugh W, Morgenthaler JD, Penix J, Zhou Y (2007) Evaluating static analysis defect warnings on production software. In: Das M, Grossman D (eds) Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on program analysis for software tools and engineering, PASTE’07, San Diego, California, USA, June 13-14, 2007. https://doi.org/10.1145/1251535.1251536. ACM, pp 1–8

  • Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 712–721

  • Balachandran V (2013) Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: Proceedings of the international conference on software engineering (ICSE). https://doi.org/10.1109/ICSE.2013.6606642. IEEE, pp 931–940

  • Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern code reviews in open-source projects: which problems do they fix?. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 202–211

  • Beller M, Bholanath R, McIntosh S, Zaidman A (2016) Analyzing the state of static analysis: a large-scale evaluation in open source software. In: IEEE 23rd international conference on software analysis, evolution, and reengineering, SANER 2016, Suita, Osaka, Japan, March 14-18, 2016. https://doi.org/10.1109/SANER.2016.105, vol 1. IEEE Computer Society, pp 470–481

  • Beller M, Gousios G, Panichella A, Proksch S, Amann S, Zaidman A (2017) Developer testing in the IDE: patterns, beliefs, and behavior. IEEE Trans Softw Eng (TSE)

  • Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how and why developers (do not) test in their IDEs. In: Proceedings of the joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering (ESEC/FSE). ACM, pp 179–190

  • Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test?. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE Computer Society, pp 559–562

  • Beller M, Gousios G, Zaidman A (2017) Oops, my tests broke the build: an explorative analysis of travis ci with github. In: International conference on mining software repositories. IEEE Press, pp 356–367

  • Beller M, Gousios G, Zaidman A (2017) TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In: Proceedings of the 14th working conference on mining software repositories. IEEE, pp 447–450

  • Bitbucket (2019) https://bitbucket.org/. Accessed: 2019-03-10

  • Bodden E (2018) Self-adaptive static analysis. In: Proceedings of the 40th international conference on software engineering: new ideas and emerging results, ICSE-NIER ’18. https://doi.org/10.1145/3183399.3183401. ACM, New York, pp 45–48

  • Buckers T, Cao C, Doesburg M, Gong B, Wang S, Beller M, Zaidman A (2017) UAV: warnings from multiple automated static analysis tools at a glance. In: IEEE 24th international conference on software analysis, evolution and reengineering (SANER). IEEE Computer Society, pp 472–476

  • Bundler (2019) https://bundler.io/. Accessed: 2019-03-10

  • Butler S, Wermelinger M, Yu Y, Sharp H (2010) Exploring the influence of identifier names on code quality: an empirical study. In: Proceedings of the European conference on software maintenance and reengineering (CSMR), pp 156–165

  • Catolino G, Palomba F, De Lucia A, Ferrucci F, Zaidman A (2018) Enhancing change prediction models using developer-related factors. J Syst Softw 143:14–28

    Article  Google Scholar 

  • Checkmarx (2019) https://www.checkmarx.com/. Accessed: 2019-03-10

  • CheckStyle (2019) http://checkstyle.sourceforge.net. Accessed: 2019-03-10

  • Chen L (2015) Continuous delivery: huge benefits, but challenges too. IEEE Softw 32(2):50–54

    Article  Google Scholar 

  • Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46

    Article  Google Scholar 

  • CodePro (2019) https://www.roseindia.net/eclipse/plugins/tool/CodePro-AnalytiX.shtml. Accessed: 2019-03-10

  • CryptLife (2017) Top ten forums for programmers, https://www.cryptlife.com/designing/programming/10-best-active-forums-for-programmers

  • Di Penta M, Cerulo L, Aversano L (2009) The life and death of statically detected vulnerabilities: an empirical study. Inf Softw Technol 51(10):1469–1484

    Article  Google Scholar 

  • Dias M, Cassou D, Ducasse S (2013) Representing code history with development environment events. In: International workshop on smalltalk technologies

  • Dillman DA, Smyth JD, Christian LM (2014) Internet, phone, mail, and mixed-mode surveys: the tailored design method. Wiley, New York

    Google Scholar 

  • D’silva V, Kroening D, Weissenbacher G (2008) A survey of automated techniques for formal software verification. IEEE Trans Comput Aided Des Integr Circuits Syst 27(7):1165–1178

    Article  Google Scholar 

  • Emanuelsson P, Nilsson U (2008) A comparative study of industrial static analysis tools. Electron Notes Theor Comput Sci 217:5–21

    Article  Google Scholar 

  • ESLint (2019) https://eslint.org/. Accessed: 2019-03-10

  • Findbugs (2019) http://findbugs.sourceforge.net/index.html. Accessed: 2019-03-10

  • flake8 (2019) http://flake8.pycqa.org/en/latest/. Accessed: 2019-03-10

  • Flanagan C, Leino KRM, Lillibridge M, Nelson G, Saxe JB, Stata R (2002) Extended static checking for java. In: Proceedings of the ACM SIGPLAN conference on programming language design and implementation (PLDI), pp 234–245

  • Flow (2019) https://flow.org/ Accessed: 2019-03-10

  • Gibbs L, Kealy M, Willis K, Green J, Welch N, Daly J (2007) What have sampling and data collection got to do with good qualitative research? Aust N Z J Public Health 31(6):540–544

    Article  Google Scholar 

  • Gerrit (2019) https://code.google.com/p/gerrit/. Accessed: 2019-03-10

  • Github (2019) https://github.com/. Accessed: 2019-03-10

  • Gitlab (2019) https://about.gitlab.com/. Accessed: 2019-03-10

  • Gousios G, Pinzger M, van Deursen A (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. https://doi.org/10.1145/2568225.2568260. ACM, New York, pp 345–355

  • Gousios G, Zaidman A, Storey MA, van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: 37th IEEE/ACM international conference on software engineering (ICSE 2015). IEEE computer society, pp 358–368

  • Gradle (2019) https://gradle.org/. Accessed: 2019-03-10

  • Heckman SS, Williams LA (2011) A systematic literature review of actionable alert identification techniques for automated static code analysis. Inf Softw Technol 53 (4):363–387. https://doi.org/10.1016/j.infsof.2010.12.007

    Article  Google Scholar 

  • Hilton M, Tunnell T, Huang K, Marinov D, Dig D (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering (ASE 2016). ACM, pp 426–437

  • Coverity (2009) Effective management of static analysis vulnerabilities and defects. https://pdfs.semanticscholar.org/1970/a4d1746734577a6eb4fdd783668f6b4202ef.pdf. Accessed 20 Aug 2019

  • Johnson B, Song Y, Murphy-Hill ER, Bowdidge RW (2013) Why don’t software developers use static analysis tools to find bugs? In: Notkin D, Cheng BHC, Pohl K (eds) 35th international conference on software engineering, ICSE’13, San Francisco, CA, USA, May 18-26, 2013. https://doi.org/10.1109/ICSE.2013.6606613. IEEE Computer Society, pp 672–681

  • Johnson RB, Onwuegbuzie AJ (2004) Mixed methods research: a research paradigm whose time has come. Educ Res 33(7):14–26

    Article  Google Scholar 

  • Johnson SC (1977) Lint, a C program checker. Bell Telephone Laboratories Murray Hill

  • Jørgensen M (2004) A review of studies on expert estimation of software development effort. J Syst Softw 70(1-2):37–60

    Article  Google Scholar 

  • JSHint (2019) https://jshint.com/. Accessed: 2019-03-10

  • Khoshgoftaar TM, Allen EB (1998) Classification of fault-prone software modules: prior probabilities, costs, and model evaluation. Empir Softw Eng 3(3):275–298

    Article  Google Scholar 

  • Kim S, Ernst MD (2007) Which warnings should I fix first?. In: Proceedings of the the 6th joint meeting of the european software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, ESEC-FSE ’07. https://doi.org/10.1145/1287624.1287633. ACM, pp 45–54

  • Krippendorff K (2004) Content analysis: an introduction to its methodology, 2nd edn. Sage, London

  • Lehman MM (1980) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1:213–221. https://doi.org/10.1016/0164-1212(79)90022-0

    Article  Google Scholar 

  • Mahmood R, Mahmoud QH (2018) Evaluation of static analysis tools for finding vulnerabilities in java and C/C++ source code. arXiv:1805.09040

  • Maven (2019) http://maven.apache.org/plugins/index.html Accessed: 2019-03-10

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The impact of code review coverage and code review participation on software quality: a case study of the qt, vtk, and ITK projects. In: Proceedings of the working conference on mining software repositories (MSR), pp 192–201

  • Muske T, Talluri R, Serebrenik A (2018) Repositioning of static analysis alarms. In: Proceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, ISSTA 2018. https://doi.org/10.1145/3213846.3213850. ACM, New York, pp 187–197

  • Nagappan N, Ball T (2005) Static analysis tools as early indicators of pre-release defect density. In: Proceedings of the international conference on software engineering (ICSE), pp 580–586

  • Nanda MG, Gupta M, Sinha S, Chandra S, Schmidt D, Balachandran P (2010) Making defect-finding tools work for you. In: Proceedings of the international conference on software engineering (ASE), vol 2, pp 99–108

  • Novak J, Krajnc A, žontar R (2010) Taxonomy of static code analysis tools. In: The 33rd international convention MIPRO, pp 418–422

  • Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The role of patch review in software evolution: an analysis of the mozilla firefox. In: Proceedings of the joint international and annual ERCIM workshops on principles of software evolution (IWPSE) and software evolution (Evol) workshops, pp 9–18

  • Oppenheim B (1992) Questionnaire design, interviewing and attitude measurement. Pinter Publishers, London

    Google Scholar 

  • Palomba F, Zanoni M, Fontana FA, De Lucia A, Oliveto R (2017) Toward a smell-aware bug prediction model. IEEE Trans Softw Eng 45(2):194–218

    Article  Google Scholar 

  • Panichella S, Arnaoudova V, Di Penta M, Antoniol G (2015) Would static analysis tools help developers with code reviews?. In: 22nd IEEE international conference on software analysis, evolution, and reengineering, SANER 2015, Montreal, QC, Canada, March 2-6, 2015. https://doi.org/10.1109/SANER.2015.7081826, pp 161–170

  • Parnas DL, Lawford M (2003) The role of inspection in software quality assurance. IEEE Trans Softw Eng 29(8):674–676

    Article  Google Scholar 

  • PEP8 online check (2019) http://pep8online.com/ Accessed: 2019-03-10

  • PMD (2019) http://pmd.sourceforge.net. Accessed: 2019-03-10

  • Prettier (2019) https://prettier.io/ Accessed: 2019-03-10

  • Proksch S, Nadi S, Amann S, Mezini M (2017) Enriching in-ide process information with fine-grained source code history. In: International conference on software analysis, evolution, and reengineering

  • Pylint (2019) https://www.pylint.org/. Accessed: 2019-03-10

  • Rahman F, Khatri S, Barr ET, Devanbu PT (2014) Comparing static bug finders and statistical prediction. In: Proceedings of the international conference on software engineering (ICSE), pp 424–434

  • Reddit (2017a) Php static analysis tools, https://www.reddit.com/r/PHP/comments/5d4ptt/static_code_analysis_tools_veracode/

  • Reddit (2017b) Static analysis tools, https://www.reddit.com/r/programming/comments/3087rz/static_code_analysis/

  • Reddit (2019) https://www.reddit.com/. Accessed: 2019-03-10

  • Rigby PC (2011) Understanding open source software peer review: review processes, parameters and statistical models, and underlying behaviours and mechanisms. Ph.D. thesis, University of Victoria, BC Canada

  • Rigby PC, German DM (2006) A preliminary examination of code review processes in open source projects. Tech. Rep. DCS-305-IR, University of Victoria

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empirical Softw Engg 14(2):131–164. https://doi.org/10.1007/s10664-008-9102-8

    Article  Google Scholar 

  • Rubocop (2019) https://github.com/rubocop-hq/rubocop Accessed: 2019-03-10

  • Ruthruff JR, Penix J, Morgenthaler JD, Elbaum S, Rothermel G (2008) Predicting accurate and actionable static analysis warnings: an experimental approach. In: Proceedings of the 30th international conference on software engineering. ACM, pp 341–350

  • Sadowski C, Aftandilian E, Eagle A, Miller-Cushon L, Jaspan C (2018) Lessons from building static analysis tools at Google. Commun ACM 61(4):58–66. https://doi.org/10.1145/3188720

    Article  Google Scholar 

  • Sadowski C, van Gogh J, Jaspan C, Söderberg E, Winter C (2015) Tricorder: building a program analysis ecosystem. In: Bertolino A, Canfora G, Elbaum SG (eds) 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16-24, 2015. https://doi.org/10.1109/ICSE.2015.76, vol 1. IEEE Computer Society, pp 598–608

  • SBT (2019) https://www.scala-sbt.org/. Accessed: 2019-03-10

  • SonarQube (2019) http://www.sonarqube.org Accessed: 2019-03-10

  • Spencer D (2009) Card sorting: designing usable categories. Rosenfeld Media

  • StackOverflow (2017a) Static analysis tool customatization, https://stackoverflow.com/questions/2825261/static-analysis-tool-customization-for-any-language

  • StackOverflow (2017b) Static analysis tools, https://stackoverflow.com/questions/22617713/whats-the-current-state-of-static-analysis-tools-for-scala

  • Thung F, Lucia L, Lo D, Jiang L, Rahman F, Devanbu PT (2012) To what extent could we detect field defects? an empirical study of false negatives in static bug finding tools. In: Proceedings of the international conference on automated software engineering (ASE), pp 50–59

  • Vassallo C, Palomba F, Bacchelli A, Gall HC (2018) Continuous code quality: are we (really) doing that?. In: ASE. ACM, pp 790–795

  • Vassallo C, Palomba F, Gall HC (2018) Continuous refactoring in ci: a preliminary study on the perceived advantages and barriers. In: 34th IEEE international conference on software maintenance and evolution (ICSME)

  • Vassallo C, Panichella S, Palomba F, Proksch S, Gall HC, Zaidman A (2019) Replication package for “How developers engage with static analysis tools in different contexts”. https://doi.org/10.5281/zenodo.3253223

  • Vassallo C, Panichella S, Palomba F, Proksch S, Zaidman A, Gall HC (2018) Context is king: the developer perspective on the usage of static analysis tools. In: SANER. IEEE Computer Society, pp 38–49

  • Vassallo C, Proksch S, Zemp T, Gall HC (2018) Un-break my build: assisting developers with build repair hints. In: International conference on program comprehension (ICPC). IEEE

  • Vassallo C, Schermann G, Zampetti F, Romano D, Leitner P, Zaidman A, Di Penta M, Panichella S (2017) A tale of CI build failures: an open source and a financial organization perspective. In: 2017 IEEE international conference on software maintenance and evolution, ICSME 2017, Shanghai, China, September 17-22, 2017. https://doi.org/10.1109/ICSME.2017.67. IEEE Computer Society, pp 183–193

  • Vassallo C, Zampetti F, Romano D, Beller M, Panichella A, Di Penta M, Zaidman A (2016) Continuous delivery practices in a large financial organization. In: 32nd IEEE international conference on software maintenance and evolution (ICSME), pp 41–50

  • Wagner S, Jürjens J, Koller C, Trischberger P (2005) Comparing bug finding tools with reviews and tests. In: Proceedings of the 17th IFIP TC6/WG 6.1 international conference on testing of communicating systems, pp 40–55

    Google Scholar 

  • Wikis (2019) https://help.github.com/en/articles/about-wikis Accessed: 2019-03-10

  • Zampetti F, Scalabrino S, Oliveto R, Canfora G, Di Penta M (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: Proceedings of the 14th international conference on mining software repositories. IEEE Press, pp 334–344

  • Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J, Vouk M (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng (TSE) 32(4):240–253

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank all the open-source and industrial developers who responded to our survey, as well as the 11 industrial experts that participated to the semi-structured interviews. We also thank Diego Martin, which acted as external validator of the enforced ASATs and types of code checks. This research was partially supported by the Swiss National Science Foundation through the SNF Projects Nos. 200021-166275 and PP00P2_170529, and the Dutch Science Foundation NWO through the TestRoots project (project number 639.022.314).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Carmine Vassallo.

Additional information

Communicated by: Massimiliano Di Penta

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Software Analysis, Evolution and Reengineering (SANER)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vassallo, C., Panichella, S., Palomba, F. et al. How developers engage with static analysis tools in different contexts. Empir Software Eng 25, 1419–1457 (2020). https://doi.org/10.1007/s10664-019-09750-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09750-5

Keywords

Navigation