Software defect data has long been used to drive software development process improvement. If security defects (vulnerabilities) are discovered and resolved by different software development practices than non-security defects, the knowledge of that distinction could be applied to drive process improvement. The goal of this research is to support technical leaders in making security-specific software development process improvements by analyzing the differences between the discovery and resolution of defects versus that of vulnerabilities. We extend Orthogonal Defect Classification (ODC), a scheme for classifying software defects to support software development process improvement, to study process-related differences between vulnerabilities and defects, creating ODC + Vulnerabilities (ODC + V). We applied ODC + V to classify 583 vulnerabilities and 583 defects across 133 releases of three open-source projects (Firefox, phpMyAdmin, and Chrome). Compared with defects, vulnerabilities are found later in the development cycle and are more likely to be resolved through changes to conditional logic. In Firefox, vulnerabilities are resolved 33% more quickly than defects. From a process improvement perspective, these results indicate opportunities may exist for more efficient vulnerability detection and resolution.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
We use a significance level of .05 for applicable statistical tests.
For example, we ran out of memory on an 8G laptop while attempting to compute Fisher’s exact test for Activity by Security Impact for Firefox
Computed using assocstats from R package vsd.
When the Cramer’s V column is blank, a computed value was not available.
Agresti A (2007) An introduction to categorical data analysis, vol 135. Wiley, New York
Alhazmi O H, Malaiya Y K (2005) Modeling the vulnerability discovery process. In: 16th IEEE international symposium on software reliability engineering, 2005. ISSRE 2005. IEEE, p 10
Anbalagan P (2011) A study of software security problem disclosure, correction and patching processes. PhD thesis, North Carolina State University
Basili V R, Rombach H D (1987) Tailoring the software process to project goals and environments. In: Proceedings of ICSE. IEEE, pp 345–357
Bhandari I, Halliday M J, Chaar J, Chillarege R, Jones K, Atkinson J, Lepori-Costello C, Jasper P, Tarver E, Lewis C et al (1994) In-process improvement through defect data interpretation. IBM Syst J 33(1):182–214
Boehm B (1981) Software engineering economics. Prentice Hall PTR, Upper Saddle River
Bridge N, Miller C (1998) Orthogonal defect classification using defect data to improve software development. Softw Qual 3(1):1–8
Butcher M, Munro H, Kratschmer T (2002) Improving software testing via odc: three case studies. IBM Syst J 41(1):31–44
Camilo F, Meneely A, Nagappan M (2015) Do bugs foreshadow vulnerabilities?: A study of the chromium project. In: Proceedings of the 12th working conference on mining software repositories, MSR ’15. IEEE Press, Piscataway, pp 269–279
Chillarege R (2006) ODC-a 10x for root cause analysis. Available online at: http://www.chillarege.com/articles/odc-10x-root-cause-analysis.html
Chillarege R, Bhandari I S, Chaar J K, Halliday M J, Moebus D S, Ray B K, Wong M-Y (1992) Orthogonal defect classification-a concept for in-process measurements. IEEE Trans Softw Eng 18(11):943–956
Chowdhury I, Zulkernine M (2011) Using complexity, coupling, and cohesion metrics as early indicators of vulnerabilities. J Syst Archit 57(3):294–313
Cochran W G (1954) Some methods for strengthening the common chi-squared tests. Biometrics 10(4):417–451
Deming W E (1986) Out of the crisis. MIT Press, Cambridge
Gegick M, Williams L, Osborne J, Vouk M (2008) Prioritizing software security fortification throughcode-level metrics. In: Proceedings of the 4th ACM workshop on quality of protection, QoP ’08. ACM. New York, pp 31–38
Howard M, Lipner S (2009) The security development lifecycle. O’Reilly Media, Incorporated, New York
Hunny U, Zulkernine M, Weldemariam K (2013) Osdc: adapting odc for developing more secure software. In: Proceedings of the 28th SAC. ACM, pp 1131–1136
Landis J R, Koch G G (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174
Massacci F, Nguyen V H (2014) An empirical methodology to evaluate vulnerability discovery models. IEEE Trans Softw Eng 40(12):1147–1162
Massacci F, Neuhaus S, Nguyen V H (2011) After-life vulnerabilities: a study on firefox evolution, its vulnerabilities, and fixes. In: Engineering secure software and systems, pp 195–208. Springer, Berlin
Mays R, Jones C, Holloway G, Studinski D (1990) Experiences with defect prevention. IBM Syst J 29(1):4–32
McGraw G (2006) Software security: building security in, volume 1. Addison-Wesley Professional
Neuhaus S, Zimmermann T, Holler C, Zeller A (2007) Predicting vulnerable software components. In: Proceedings of the 14th ACM conference on computer and communications security, CCS ’07. ACM, New York, pp 529–540
Nguyen V H, Massacci F (2013) The (un)reliability of nvd vulnerable versions data: an empirical experiment on google chrome vulnerabilities. In: Proceedings of the 8th ACM SIGSAC symposium on information, computer and communications security, ASIA CCS ’13. ACM, New York, pp 493–498
Ott L (1988) An introduction to statistical methods and data analysis. Duxbury Press
Ozment J A (2007) Vulnerability discovery & software security. PhD thesis, Citeseer
Paulk M C, Weber C V, Curtis B, Chrissis M B (1995) The capability maturity model: guidelines for improving the software process. Addison-Wesley, Reading
Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the 22Nd ACM SIGSOFT international symposium on foundations of software engineering, FSE 2014. ACM, New York, pp 155–165
Riaz M, King J, Slankas J, Williams L (2014) Hidden in plain sight: automatically identifying security requirements from natural language artifacts. In: Proceedings of the 22nd RE. IEEE, pp 183–192
Robinson B, Francis P, Ekdahl F (2008) A defect-driven process for software quality improvement. In: Proceedings of the 2nd ESEM. ACM, pp 333–335
Shewhart W (1930) Economic quality control of manufactured product. Bell Syst Tech J 9(2):364–389
Shin Y, Meneely A, Williams L, Osborne J A (2011) Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Trans Softw Eng 37(6):772–787
Shostack A (2014) Threat modeling: designing for security. Wiley, New York
Souza R, Silva B (2017) Sentiment analysis of travis ci builds. In: Proceedings of the 14th international conference on mining software repositories, MSR ’17. IEEE Press, Piscataway, pp 459–462
Syed-Mohamad S M, McBride T (2008) A comparison of the reliability growth of open source and in-house software. In: Proceedings of the 15th APSEC. IEEE, pp 229–236
Theisen C, Herzig K, Morrison P, Murphy B, Williams L A (2015) Approximating attack surfaces with stack traces. In: 37th IEEE/ACM international conference on software engineering, ICSE 2015, Florence, Italy, May 16–24, vol 2. IEEE, pp 199–208
Walden J, Stuckman J, Scandariato R (2014) Predicting vulnerable components: software metrics vs text mining. In: 2014 IEEE 25th international symposium on software reliability engineering, pp 23–33
Zaman S, Adams B, Hassan A E (2011) Security versus performance bugs: a case study on firefox. In: Proceedings of the 8th working conference on mining software repositories, MSR ’11. ACM, New York, pp 93–102
Zheng J, Williams L, Nagappan N, Snipes W, Hudepohl J P, Vouk M A (2006) On the value of static analysis for fault detection in software. IEEE Trans Softw Eng 32(4):240–253
This work is supported, in part, by IBM and by the USA National Security Agency (NSA) Science of Security Lablet at NCSU. Any opinions expressed in this report are those of the author(s) and do not necessarily reflect the views of IBM or the NSA. We thank Marc Delisle of the phpMyadmin for providing us with the snapshot of defect repostitories for this study, and for kindly answering many questions and offering his perspective. We also thank Dr. Alyson Wilson for providing helpful feedback on designing the classification assignments for the raters. We are grateful to Dr. Andy Meneely for providing the Chrome database snapshot, and to Dr. Fabio Massacci and the University of Trento for granting access to their curated Chrome vulnerability list. Finally, we thank the RealSearchFootnote 36 research group for providing helpful feedback on this work.
Communicated by: Mark Grechanik
About this article
Cite this article
Morrison, P.J., Pandita, R., Xiao, X. et al. Are vulnerabilities discovered and resolved like other defects?. Empir Software Eng 23, 1383–1421 (2018). https://doi.org/10.1007/s10664-017-9541-1
- Software development
- Process improvement
- Orthogonal Defect Classification (ODC)