Software Quality Journal

, Volume 21, Issue 2, pp 241–257 | Cite as

Static correspondence and correlation between field defects and warnings reported by a bug finding tool

  • Cesar Couto
  • João Eduardo Montandon
  • Christofer Silva
  • Marco Tulio Valente
Article

Abstract

Despite the interest and the increasing number of static analysis tools for detecting defects in software systems, there is still no consensus on the actual gains that such tools introduce in software development projects. Therefore, this article reports a study carried out to evaluate the degree of correspondence and correlation between post-release defects (i.e., field defects) and warnings issued by FindBugs, a bug finding tool widely used in Java systems. The study aimed to evaluate two types of relations: static correspondence (when warnings contribute to find the static program locations changed to remove field defects) and statistical correlation (when warnings serve as early indicators for future field defects). As a result, we have concluded that there is no static correspondence between field defects and warnings. However, statistical tests showed that there is a moderate level of correlation between warnings and such kinds of software defects.

Keywords

Bug finding tools Field defects Software quality assurance tools 

Notes

Acknowledgments

This work was supported by FAPEMIG, CAPES, and CNPq.

References

  1. Araujo, J. E., Souza, S., & Valente, M. T. (2011). Study on the relevance of the warnings reported by Java bug finding tools. IET Software, 5(4), 366–374.CrossRefGoogle Scholar
  2. Ayewah, N., Hovemeyer, D., Morgenthaler, J. D., Penix J., & William, P. (2008). Using static analysis to find bugs. IEEE Software, 25(5).Google Scholar
  3. Bessey, A., Block, K., Chelf, B., Chou, A., Fulton, B., Hallem, S., Henri-Gros, C., Kamsky, A., McPeak, S., Engler, D. (2010). A few billion lines of code later: Using static analysis to find bugs in the real world. Communications of the ACM, 53(2), 66–75.CrossRefGoogle Scholar
  4. Butler, S., Wermelinger, M., Yu, Y., & Sharp, H. (2009). Relating identifier naming flaws and code quality: An empirical study. In 16th working conference on reverse engineering (WCRE), pp. 31–35.Google Scholar
  5. Copeland, T. (2005). PMD applied. Alexandria: Centennial Books.Google Scholar
  6. Dallmeier, V., & Zimmermann, T. (2007). Extraction of bug localization benchmarks from history. In 22th conference on automated software engineering (ASE), pp. 433–436.Google Scholar
  7. D’Ambros, M., Bacchelli, A., & Michele, L. (2010). On the impact of design flaws on software defects. In 10th international conference on quality software (QSIC), pp 23–31.Google Scholar
  8. D’Ambros, M., Lanza, M., & Robbes, R. (2010). An extensive comparison of bug prediction approaches. In 7th working conference on mining software repositories (MSR), pp. 31–41.Google Scholar
  9. Foster, J. S., Hicks, M. W., & Pugh, W. (2007). Improving software quality with static analysis. In 7th workshop on program analysis for software tools and engineering (PASTE), pp. 83–84.Google Scholar
  10. Granger, C. W. J. (1969). Investigating causal relations by econometric models and cross-spectral methods. Econometrica, 37(3), 424–438.CrossRefMathSciNetGoogle Scholar
  11. Holschuh, T., Pauser, M., Herzig, K., Zimmermann, T. P., & Rahul, Z. (2009). Andreas predicting defects in sap java code: An experience report. In 31st international conference on software engineering (ICSE), pp. 172–181.Google Scholar
  12. Hovemeyer, D., & Pugh, W. (2004). Finding bugs is easy. SIGPLAN Notices, 39(12), 92–106.CrossRefGoogle Scholar
  13. Johnson, S. C. (1977). Lint: A C program checker. Technical Report 65, Bell Laboratories.Google Scholar
  14. Kim, S., & Ernst, M. D. (2007). Which warnings should I fix first? In 15th international symposium on foundations of software engineering (FSE), pp. 45–54.Google Scholar
  15. Lanza, M., & Marinescu, R. (2006). Object-oriented metrics in practice. Springer.Google Scholar
  16. Larus, J. R., Ball, T., Das, M., DeLine, R., Fahndrich, M., Pincus, J., Rajamani, S. K., Ramanathan, V. (2004). Righting software. IEEE Software, 21(3), 92–100.CrossRefGoogle Scholar
  17. Louridas, P. (2006). Static code analysis. IEEE Software, 23(4), 58–61.CrossRefGoogle Scholar
  18. Nagappan, N., & Ball, T. (2005). Static analysis tools as early indicators of pre-release defect density. In 27th international conference on software engineering (ICSE), pp. 580–586.Google Scholar
  19. Nagappan, N., Ball, T., & Zeller, A. (2006). Mining metrics to predict component failures. In 28th international conference on software engineering (ICSE), pp. 452–461.Google Scholar
  20. Perry, D. E., Porter, A. A., & Votta, L. G. (1997). A primer on empirical studies (tutorial). In Tutorial presented at 19th international conference on software engineering (ICSE), pp. 657–658.Google Scholar
  21. Pfleeger, S. L. (1995). Experimental design and analysis in software engineering, part 5: Analyzing the data. Software Engineering Notes, 20(5), 14–17.CrossRefGoogle Scholar
  22. Sprent, P., & Smeeton, N. C. (2007). Applied nonparametric statistical methods. Boca Raton: Chapman & Hall.MATHGoogle Scholar
  23. Subramanyam, R., & Krishnan, M. S. (2003). Empirical analysis of CK metrics for object-oriented design complexity: Implications for software defects. IEEE Transaction on Software Engineering, 29(4):297–310.CrossRefGoogle Scholar
  24. Wagner, S., Jürjens, J., Koller, C., & Trischberger, P. (2005). Comparing bug finding tools with reviews and tests. In 17th international conference on testing of communicating systems (TestCom), volume 3502 of LNCS, pp. 40–55. Springer.Google Scholar
  25. Wagner, S., Aichner, M., Wimmer, J., & Schwalb, M. (2008). An evaluation of two bug pattern tools for Java. In 1st international conference on software testing, verification, and validation (ICST), pp. 248–257.Google Scholar
  26. Zheng, J., Williams, L., Nagappan, N., Hudepohl, J. P., & Vouk M. A. (2006). On the value of static analysis for fault detection in software. IEEE Transactions on Software Engineering, 32(4).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Cesar Couto
    • 1
    • 2
  • João Eduardo Montandon
    • 1
  • Christofer Silva
    • 2
  • Marco Tulio Valente
    • 1
  1. 1.Department of Computer ScienceUFMGBelo HorizonteBrazil
  2. 2.Department of ComputingCEFET-MGBelo HorizonteBrazil

Personalised recommendations