Skip to main content

Are vulnerabilities discovered and resolved like other defects?

Abstract

Software defect data has long been used to drive software development process improvement. If security defects (vulnerabilities) are discovered and resolved by different software development practices than non-security defects, the knowledge of that distinction could be applied to drive process improvement. The goal of this research is to support technical leaders in making security-specific software development process improvements by analyzing the differences between the discovery and resolution of defects versus that of vulnerabilities. We extend Orthogonal Defect Classification (ODC), a scheme for classifying software defects to support software development process improvement, to study process-related differences between vulnerabilities and defects, creating ODC + Vulnerabilities (ODC + V). We applied ODC + V to classify 583 vulnerabilities and 583 defects across 133 releases of three open-source projects (Firefox, phpMyAdmin, and Chrome). Compared with defects, vulnerabilities are found later in the development cycle and are more likely to be resolved through changes to conditional logic. In Firefox, vulnerabilities are resolved 33% more quickly than defects. From a process improvement perspective, these results indicate opportunities may exist for more efficient vulnerability detection and resolution.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    https://www.wired.com/story/ether-cryptocurrency-theft/

  2. 2.

    https://www.mozilla.org/en-US/firefox

  3. 3.

    http://www.phpmyadmin.net/

  4. 4.

    https://www.google.com/chrome

  5. 5.

    https://cwe.mitre.org

  6. 6.

    http://www.nvd.com

  7. 7.

    https://sites.google.com/a/ncsu.edu/odc-v/

  8. 8.

    https://bugzilla.mozilla.org/show_bug.cgi?id=394610

  9. 9.

    https://bugzilla.mozilla.org/show_bug.cgi?id=545080

  10. 10.

    https://www.mozilla.org/security/announce/2011/mfsa2011-10.html

  11. 11.

    https://bugzilla.mozilla.org/show_bug.cgi?id=643051

  12. 12.

    https://bugzilla.mozilla.org/show_bug.cgi?id=643927

  13. 13.

    http://hg.mozilla.org/mozilla-central/rev/1ecbcf5cf362

  14. 14.

    https://bugzilla.mozilla.org/show_bug.cgi?id=547608

  15. 15.

    http://hg.mozilla.org/releases/mozilla-1.9.2/rev/1878344aded9

  16. 16.

    http://www.phpmyadmin.net/home_page/security/PMASA-2007-3.php

  17. 17.

    http://hg.mozilla.org/releases/mozilla-1.9.2/rev/6ccaf1dd6553

  18. 18.

    http://www.phpmyadmin.net/home_page/security/PMASA-2007-3.php

  19. 19.

    http://www.bugzilla.org/

  20. 20.

    http://www.mozilla.org/security/announce/

  21. 21.

    https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Development_process_overview

  22. 22.

    https://wiki.mozilla.org/images/e/ed/Analyst_report_Q1_2010.eps

  23. 23.

    https://www.openhub.net/p/firefox/factoids#FactoidTeamSizeVeryLarge

  24. 24.

    http://www.phpmyadmin.net/home_page/docs.php

  25. 25.

    https://www.phpmyadmin.net/contribute/

  26. 26.

    https://www.phpmyadmin.net/15-years/

  27. 27.

    https://bugs.chromium.org/p/chromium/issues/list

  28. 28.

    https://googleprojectzero.blogspot.com/2015/02/feedback-and-data-driven-updates-to.html

  29. 29.

    https://www.netmarketshare.com/browser-market-share.aspx

  30. 30.

    https://www.openhub.net/p/chrome/factoids#FactoidTeamSizeVeryLarge

  31. 31.

    We use a significance level of .05 for applicable statistical tests.

  32. 32.

    https://cran.r-project.org/web/packages/coin/index.html

  33. 33.

    For example, we ran out of memory on an 8G laptop while attempting to compute Fisher’s exact test for Activity by Security Impact for Firefox

  34. 34.

    Computed using assocstats from R package vsd.

  35. 35.

    When the Cramer’s V column is blank, a computed value was not available.

  36. 36.

    https://www.facebook.com/realsearchNCSU

References

  1. Agresti A (2007) An introduction to categorical data analysis, vol 135. Wiley, New York

    Book  Google Scholar 

  2. Alhazmi O H, Malaiya Y K (2005) Modeling the vulnerability discovery process. In: 16th IEEE international symposium on software reliability engineering, 2005. ISSRE 2005. IEEE, p 10

  3. Anbalagan P (2011) A study of software security problem disclosure, correction and patching processes. PhD thesis, North Carolina State University

  4. Basili V R, Rombach H D (1987) Tailoring the software process to project goals and environments. In: Proceedings of ICSE. IEEE, pp 345–357

  5. Bhandari I, Halliday M J, Chaar J, Chillarege R, Jones K, Atkinson J, Lepori-Costello C, Jasper P, Tarver E, Lewis C et al (1994) In-process improvement through defect data interpretation. IBM Syst J 33(1):182–214

    Article  Google Scholar 

  6. Boehm B (1981) Software engineering economics. Prentice Hall PTR, Upper Saddle River

    MATH  Google Scholar 

  7. Bridge N, Miller C (1998) Orthogonal defect classification using defect data to improve software development. Softw Qual 3(1):1–8

    Google Scholar 

  8. Butcher M, Munro H, Kratschmer T (2002) Improving software testing via odc: three case studies. IBM Syst J 41(1):31–44

    Article  Google Scholar 

  9. Camilo F, Meneely A, Nagappan M (2015) Do bugs foreshadow vulnerabilities?: A study of the chromium project. In: Proceedings of the 12th working conference on mining software repositories, MSR ’15. IEEE Press, Piscataway, pp 269–279

  10. Chillarege R (2006) ODC-a 10x for root cause analysis. Available online at: http://www.chillarege.com/articles/odc-10x-root-cause-analysis.html

  11. Chillarege R, Bhandari I S, Chaar J K, Halliday M J, Moebus D S, Ray B K, Wong M-Y (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956

    Article  Google Scholar 

  12. Chowdhury I, Zulkernine M (2011) Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities. J Syst Archit 57(3):294–313

    Article  Google Scholar 

  13. Cochran W G (1954) Some methods for strengthening the common chi-squared tests. Biometrics 10(4):417–451

    MathSciNet  Article  Google Scholar 

  14. Deming W E (1986) Out of the crisis. MIT Press, Cambridge

    Google Scholar 

  15. Gegick M, Williams L, Osborne J, Vouk M (2008) Prioritizing software security fortification throughcode-level metrics. In: Proceedings of the 4th ACM workshop on quality of protection, QoP ’08. ACM. New York, pp 31–38

  16. Howard M, Lipner S (2009) The security development lifecycle. O’Reilly Media, Incorporated, New York

    Google Scholar 

  17. Hunny U, Zulkernine M, Weldemariam K (2013) Osdc: adapting odc for developing more secure software. In: Proceedings of the 28th SAC. ACM, pp 1131–1136

  18. Landis J R, Koch G G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174

    Article  Google Scholar 

  19. Massacci F, Nguyen V H (2014) An empirical methodology to evaluate vulnerability discovery models. IEEE Trans Softw Eng 40(12):1147–1162

    Article  Google Scholar 

  20. Massacci F, Neuhaus S, Nguyen V H (2011) After-life vulnerabilities: a study on firefox evolution, its vulnerabilities, and fixes. In: Engineering secure software and systems, pp 195–208. Springer, Berlin

    Google Scholar 

  21. Mays R, Jones C, Holloway G, Studinski D (1990) Experiences with defect prevention. IBM Syst J 29(1):4–32

    Article  Google Scholar 

  22. McGraw G (2006) Software security: building security in, volume 1. Addison-Wesley Professional

  23. Neuhaus S, Zimmermann T, Holler C, Zeller A (2007) Predicting vulnerable software components. In: Proceedings of the 14th ACM conference on computer and communications security, CCS ’07. ACM, New York, pp 529–540

  24. Nguyen V H, Massacci F (2013) The (un)reliability of nvd vulnerable versions data: an empirical experiment on google chrome vulnerabilities. In: Proceedings of the 8th ACM SIGSAC symposium on information, computer and communications security, ASIA CCS ’13. ACM, New York, pp 493–498

  25. Ott L (1988) An introduction to statistical methods and data analysis. Duxbury Press

  26. Ozment J A (2007) Vulnerability discovery & software security. PhD thesis, Citeseer

  27. Paulk M C, Weber C V, Curtis B, Chrissis M B (1995) The capability maturity model: guidelines for improving the software process. Addison-Wesley, Reading

    Google Scholar 

  28. Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, FSE 2014. ACM, New York, pp 155–165

  29. Riaz M, King J, Slankas J, Williams L (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts. In: Proceedings of the 22nd RE. IEEE, pp 183–192

  30. Robinson B, Francis P, Ekdahl F (2008) A defect-driven process for software quality improvement. In: Proceedings of the 2nd ESEM. ACM, pp 333–335

  31. Shewhart W (1930) Economic quality control of manufactured product. Bell Syst Tech J 9(2):364–389

    Article  Google Scholar 

  32. Shin Y, Meneely A, Williams L, Osborne J A (2011) Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Trans Softw Eng 37(6):772–787

    Article  Google Scholar 

  33. Shostack A (2014) Threat modeling: designing for security. Wiley, New York

    Google Scholar 

  34. Souza R, Silva B (2017) Sentiment analysis of travis ci builds. In: Proceedings of the 14th international conference on mining software repositories, MSR ’17. IEEE Press, Piscataway, pp 459–462

  35. Syed-Mohamad S M, McBride T (2008) A comparison of the reliability growth of open source and in-house software. In: Proceedings of the 15th APSEC. IEEE, pp 229–236

  36. Theisen C, Herzig K, Morrison P, Murphy B, Williams L A (2015) Approximating attack surfaces with stack traces. In: 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16–24, vol 2. IEEE, pp 199–208

  37. Walden J, Stuckman J, Scandariato R (2014) Predicting vulnerable components: software metrics vs text mining. In: 2014 IEEE 25th international symposium on software reliability engineering, pp 23–33

  38. Zaman S, Adams B, Hassan A E (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories, MSR ’11. ACM, New York, pp 93–102

  39. Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J P, Vouk M A (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng 32(4):240–253

    Article  Google Scholar 

Download references

Acknowledgments

This work is supported, in part, by IBM and by the USA National Security Agency (NSA) Science of Security Lablet at NCSU. Any opinions expressed in this report are those of the author(s) and do not necessarily reflect the views of IBM or the NSA. We thank Marc Delisle of the phpMyadmin for providing us with the snapshot of defect repostitories for this study, and for kindly answering many questions and offering his perspective. We also thank Dr. Alyson Wilson for providing helpful feedback on designing the classification assignments for the raters. We are grateful to Dr. Andy Meneely for providing the Chrome database snapshot, and to Dr. Fabio Massacci and the University of Trento for granting access to their curated Chrome vulnerability list. Finally, we thank the RealSearchFootnote 36 research group for providing helpful feedback on this work.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Patrick J. Morrison.

Additional information

Communicated by: Mark Grechanik

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Morrison, P.J., Pandita, R., Xiao, X. et al. Are vulnerabilities discovered and resolved like other defects?. Empir Software Eng 23, 1383–1421 (2018). https://doi.org/10.1007/s10664-017-9541-1

Download citation

Keywords

  • Software development
  • Measurement
  • Process improvement
  • Security
  • Orthogonal Defect Classification (ODC)