Empirical Software Engineering

, Volume 22, Issue 4, pp 1903–1935 | Cite as

Are delayed issues harder to resolve? Revisiting cost-to-fix of defects throughout the lifecycle

  • Tim Menzies
  • William Nichols
  • Forrest Shull
  • Lucas Layman


Many practitioners and academics believe in a delayed issue effect (DIE); i.e. the longer an issue lingers in the system, the more effort it requires to resolve. This belief is often used to justify major investments in new development processes that promise to retire more issues sooner. This paper tests for the delayed issue effect in 171 software projects conducted around the world in the period from 2006–2014. To the best of our knowledge, this is the largest study yet published on this effect. We found no evidence for the delayed issue effect; i.e. the effort to resolve issues in a later phase was not consistently or substantially greater than when issues were resolved soon after their introduction. This paper documents the above study and explores reasons for this mismatch between this common rule of thumb and empirical data. In summary, DIE is not some constant across all projects. Rather, DIE might be an historical relic that occurs intermittently only in certain kinds of projects. This is a significant result since it predicts that new development processes that promise to faster retire more issues will not have a guaranteed return on investment (depending on the context where applied), and that a long-held truth in software engineering should not be considered a global truism.


Software economics Phase delay Cost to fix 



The authors wish to thank David Tuma and Yasutaka Shirai for their work on the SEI databases that made this analysis possible. In particular, we thank Tuma Solutions for providing the Team Process Data Warehouse software. Also, the authors gratefully acknowledge the careful comments of anonymous reviewers from the FSE and ICSE conferences. This work was partially funded by an National Science Foundation grants NSF-CISE 1302169 and CISE 1506586.

This material is based upon work funded and supported by TSP Licensing under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center sponsored by the United States Department of Defense. This material has been approved for public release and unlimited distribution. DM-0003956

Personal Software Process S M , Team Software Process S M , and TSP S M are service marks of Carnegie Mellon University.


  1. Arcuri A, Briand L (2011) A practical guide for using statistical tests to assess randomized algorithms in software engineering. In: ICSE’11, pp 1–10Google Scholar
  2. Aschwanden C (2010) Convincing the public to accept new medical guidelines. Accessed: 2015-02-10
  3. Aschwanden C (2015) Your brain is primed to reach false conclusions. Accessed: 2015-02-10
  4. Avram A (2014) Idc study: how many software developers are out there?
  5. Bachmann FH, Carballo L, McHale J, Nord RL (2013) Integrate end to end early and often. Softw IEEE 30(4):9–14CrossRefGoogle Scholar
  6. Baziuk W (1995) Bnr/nortel: path to improve product quality, reliability and customer satisfaction. In: Sixth international symposium on software reliability engineering, 1995. Proceedings. IEEE, pp 256–262Google Scholar
  7. Beck K, Beedle M, van Bennekum A, Cockburn A, Cunningham W, Fowler M, Grenning J, Highsmith J, Hunt A, Jeffries R, Kern J, Marick B, Martin RC, Mallor S, Schwaber K, Sutherland J, Thomas D (2001) The agile manifesto.
  8. Beck K (2000) Extreme programming explained: embrace change. Addison WesleyGoogle Scholar
  9. Bettenburg N, Nagappan M, Hassan AE (2012) Think locally, act globally: improving defect and effort prediction models. In: MSR’12Google Scholar
  10. Bettenburg N, Nagappan M, Hassan AE (2014) Towards improving statistical modeling of software engineering data: think locally, act globally!. Emp Softw Eng:1–42Google Scholar
  11. Boehm B.W., Papaccio P.N. (1988) Understanding and controlling software costs. IEEE Trans Softw Eng 14(10):1462–1477CrossRefGoogle Scholar
  12. Boehm B (1976) Software engineering. IEEE Trans Comput C-25(12):1226–1241CrossRefzbMATHGoogle Scholar
  13. Boehm B (1980) Developing small-scale application software products: some experimental results. In: Proceedings of the IFIP congress, pp 321–326Google Scholar
  14. Boehm B (1981) Software engineering economics. Prentice Hall, Englewood CliffszbMATHGoogle Scholar
  15. Boehm B (2010) Architecting: how much and when? In: Oram A, Wilson G (eds) Making software: what really works, and why we believe it. O’Reilly Media, pp 141–186Google Scholar
  16. Boehm B, Basili VR (2001) Software defect reduction top 10 list. IEEE Softw:135–137Google Scholar
  17. Boehm B, Horowitz E, Madachy R, Reifer D, Clark BK, Steece B, Winsor Brown A, Chulani S, Abts C (2000) Software cost estimation with cocomo II. Prentice HallGoogle Scholar
  18. Boehm BW (1988) A spiral model of software development and enhancement. Computer 21(5):61–72CrossRefGoogle Scholar
  19. Boehm BW (2012) Architecting: how much and when. In: Oram A, Wilson G (eds) Making software: what really works, and why we believe it. O’Reilly, pp 161–186Google Scholar
  20. Bright P (2015) What windows as a service and a ’free upgrade’ mean at home and at work.
  21. Budgen D, Brereton P, Kitchenham B (2009) Is evidence based software engineering mature enough for practice & policy? In: 33rd Annual IEEE software engineering workshop 2009 (SEW-33). SkovdeGoogle Scholar
  22. Carlson MDA, Sean Morrison R (2009) Study design, precision, and validity in observational studies. J Palliative Med 12(1):77–82CrossRefGoogle Scholar
  23. Cook TD, Campbell DT (1979) Quasi-experimentation: design & analysis issues for field settings. Houghton Mifflin BostonGoogle Scholar
  24. Daly EB (1977) Management of software development. IEEE Trans Softw Eng SE-3(3):229–242CrossRefGoogle Scholar
  25. Deming WE (1986) Out of the crisis. MIT PressGoogle Scholar
  26. Devanbu P, Zimmermann T, Bird C (2016) Belief & evidence in empirical software engineering. In: Proceedings of the 38th international conference on software engineering, pp 108–119. ACMGoogle Scholar
  27. Dunsmore HE (1988) Evidence supports some truisms, belies others. (some empirical results concerning software development). IEEE Softw:96–99Google Scholar
  28. Efron B, Tibshirani RJ (1993) An introduction to the bootstrap. Mono. Stat. Appl. Probab. Chapman and Hall, LondonCrossRefGoogle Scholar
  29. Elssamadisy A, Schalliol G (2002) Recognizing and responding to ”bad smells” in extreme programming. In: Proceedings of the 24th international conference on software engineering, ICSE ’02. ACM, New York, pp 617–622Google Scholar
  30. Endres A, Rombach D (2003) A handbook of software and systems engineering: empirical observations, laws and theories. Addison WesleyGoogle Scholar
  31. Fagan ME (1976) Design and code inspections to reduce errors in program development. IBM Syst J 15(3):182–211CrossRefGoogle Scholar
  32. Fenton NE, Neil M (2000) Software metrics: a roadmap. In: Finkelstein A (ed) Software metrics: a roadmap. Available from ACM Press, New York
  33. Fenton NE, Ohlsson N (2000) Quantitative analysis of faults and failures in a complex software system. IEEE Trans Softw Eng:797–814Google Scholar
  34. Fenton NE, Pfleeger SL (1997) Software metrics: a rigorous & practical approach. International Thompson PressGoogle Scholar
  35. Ghotra B, McIntosh S, Hassan AE (2015) Revisiting the impact of classification techniques on the performance of defect prediction models. In: Proc. of the international conference on software engineering (ICSE), pp 789–800Google Scholar
  36. Glass RL (2002) Facts and fallacies of software engineering. Addison-Wesley Professional, BostonGoogle Scholar
  37. Gordon P (2016) The cost of requirements errors.
  38. Harter DE, Kemerer CF, Slaughter SA (2012) Does software process improvement reduce the severity of defects? A longitudinal field study. IEEE Trans Softw Eng 38(4):810–827CrossRefGoogle Scholar
  39. Humphrey WS (1995) A discipline for software engineering. Addison-Wesley Longman Publishing Co. Inc.Google Scholar
  40. Humphrey WS (2000) Introduction to the team software process. Addison-Wesley Longman Ltd., EssexGoogle Scholar
  41. Humphrey WS (2005) TSP(SM)-leading a development team (SEI series in software engineering). Addison-Wesley ProfessionalGoogle Scholar
  42. IEEE-1012 (1998) IEEE standard 1012-2004 for software verification and validationGoogle Scholar
  43. IfSQ (2013) Catching defects during testing is 10 times more expensive.
  44. Gartner Inc (2014) Gartner says worldwide software market grew 4.8 percent in 2013.
  45. Jacobson I, Booch G, Rumbaugh J (1999) The unified software development process. Addison-Wesley ReadingGoogle Scholar
  46. Jones C (2007) Estimating software costs, 2nd edn. McGraw-HillGoogle Scholar
  47. Jones C, Bonsignour O (2012) The economics of software quality. Addison WesleyGoogle Scholar
  48. Jørgensen M, Gruschke TM (2009) The impact of lessons-learned sessions on effort estimation and uncertainty assessments. IEEE Trans Softw Eng 35(3):368–383CrossRefGoogle Scholar
  49. Kampenes VBy, Dybå T, Hannay JE, Sjøberg DIK (2007) A systematic review of effect size in software engineering experiments. Inf Softw Technol 49(11-12):1073–1086CrossRefGoogle Scholar
  50. Karg LM, Grottke M, Beckhaus A (2011) A systematic literature review of software quality cost research. J Syst Softw 84(3):415–427CrossRefGoogle Scholar
  51. Kim D (2013) Making agile mandatory at the department of defenseGoogle Scholar
  52. Kitchenham BA, Dyba T, Jørgensen M (2004) Evidence-based software engineering. In: ICSE ’04: Proceedings of the 26th international conference on software engineering. IEEE Computer Society, Washington, pp 273–281Google Scholar
  53. Kocaguneli E, Zimmermann T, Bird C, Nagappan N, Menzies T (2013) Distributed development considered harmful? In: Proceedings - international conference on software engineering, pp 882– 890Google Scholar
  54. Lapham MA, Garcia-Miller S, Nemeth-Adams L, Brown N, Hackemack L, Hammons CB, Levine L, Schenker AR (2011) Agile methods: selected dod management and acquisition concerns. Technical report, Carnegie Mellon University - Software Engineering InstituteGoogle Scholar
  55. Larman C, Basili VR (2003) Iterative and incremental development: a brief history. Computer 36(6):47–56CrossRefGoogle Scholar
  56. Leffingwell D (1996) Calculating your return on investment form more effective requirements management. Rational Software Corporation
  57. Madigan D, Stang PE, Berlin JA, Schuemie M, Overhage MJ, Suchard MA, Dumouchel B, Hartzema AG, Ryan PB (2014) A systematic statistical approach to evaluating evidence from observational studies. Ann Rev Stat Appl 1:11–39CrossRefGoogle Scholar
  58. Maxwell KD (2002) Applied statistics for software managers. Prentice-Hall, Englewood CliffsGoogle Scholar
  59. McConnell S (1996) Software quality at top speed. Softw Develop 4(8):38–42Google Scholar
  60. McConnell S (2001) An ounce of prevention. IEEE Softw 18(3):5–7CrossRefGoogle Scholar
  61. McHale J (2002) Tsp: process costs and benefits. CrosstalkGoogle Scholar
  62. Mead NR, Allen JH, Barnum S, Ellison RJ, McGraw G (2004) Software security engineering: a guide for project managers. Addison-Wesley ProfessionalGoogle Scholar
  63. Menzies T, Benson M, Costello K, Moats C, Northey M, Richarson J (2008) Learning better IV&V practices. Innovations in Systems and Software Engineering. Available from
  64. Menzies T, Butcher A, Cok DR, Marcus A, Layman L, Shull F, Turhan B, Zimmermann T (2013) Local versus global lessons for defect prediction and effort estimation, vol 39. Available from
  65. Menzies T, Butcher A, Marcus A, Zimmermann T, Cok D (2011) Local vs global models for effort estimation and defect prediction. In: IEEE ASE’11. Available from
  66. Minku LL, Yao X (2013) Ensembles and locality: insight on improving software effort estimation. Inf Softw Technol 55(8):1512–1528CrossRefGoogle Scholar
  67. Mittas N, Angelis L (2013) Ranking and clustering software cost estimation models through a multiple comparisons algorithm. IEEE Trans Softw Eng 39(4):537–551CrossRefGoogle Scholar
  68. Paivarinta T, Smolander K (2015) Theorizing about software development practices. Sci Comput Program 101:124–135CrossRefGoogle Scholar
  69. Parker J (2013) Good requirements deliver a high roi.
  70. Passos C, Braun AP, Cruzes DS, Mendonca M (2011) Analyzing the impact of beliefs in software project practices. In: ESEM’11Google Scholar
  71. Paul R Exclusive: a behind-the-scenes look at facebook release engineering. Accessed: 2016-06-14
  72. Popper K (1959) The logic of scientific discovery. Basic Books. New YorkGoogle Scholar
  73. Posnett D, Filkov V, Devanbu P (2011) Ecological inference in empirical software engineering. In: Proceedings of ASE’11Google Scholar
  74. Prasad V, Vandross A, Toomey C, Cheung M, Rho J, Quinn S, Jacob Chacko S, Borkar D, Gall V, Selvaraj S, Ho N, Cifu A (2013) A decade of reversal: an analysis of 146 contradicted medical practices. Mayo Clinic Proc 88(8):790–798CrossRefGoogle Scholar
  75. Pressman RS (2005) Software engineering: a practitioner’s approach. Palgrave MacmillanGoogle Scholar
  76. Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github. In: Proceedings of the ACM SIGSOFT 22nd international symposium on the foundations of software engineering, FSE ’14. ACMGoogle Scholar
  77. Reifer DJ (2007) Profiles of level 5 cmmi organizations. Crosstalk: J Defense Softw Eng:24–28Google Scholar
  78. Royce W (1998) Software project management: a unified framework. Addison-Wesley, ReadingGoogle Scholar
  79. Shepperd MJ, MacDonell SG (2012) Evaluating prediction systems in software project estimation. Inf Softw Technol 54(8):820–827CrossRefGoogle Scholar
  80. Shepperd MJ, Song Q, Sun Z, Mair C (2013) Data quality: some comments on the nasa software defect datasets. IEEE Trans Software Eng 39(9):1208–1215CrossRefGoogle Scholar
  81. Shirai Y, Nichols W, Kasunic M (2014) Initial evaluation of data quality in a tsp software engineering project data repository. In: Proceedings of the 2014 international conference on software and system process, ICSSP 2014. ACM, New York, pp 25–29Google Scholar
  82. Shull F, Basili V, Boehm B, Winsor Brown A, Costa P, Lindvall M, Port D, Rus I, Tesoriero R, Zelkowitz M (2002) What we have learned about fighting defects. In: Eighth IEEE symposium on software metrics, 2002. Proceedings, pp 249–258Google Scholar
  83. Shull F, Feldmann R (2008) Building theories from multiple evidence sources. In: Shull F, Singer J, Sjoberg DIK (eds) Guide to advanced empirical software engineering. Springer-Verlag, pp 337–364Google Scholar
  84. Sjøberg DIK, Dybå T, Anda BCD, Hannay JE (2008) Building theories in software engineering. In: Shull F, Singer J, Sjøberg DIK (eds) Guide to advanced empirical software engineering. Springer-Verlag, pp 312–336Google Scholar
  85. Soni M (2016) Defect prevention: reducing costs and enhancing quality.
  86. Stecklein J, Dabney J, Dick B, Haskins B, Lovell R, Moroney G (2004) Error cost escalation through the project life cycle. In: 14th Annual INCOSE international symposium. ToulouseGoogle Scholar
  87. Stephenson WE (1976) An analysis of the resources used in the safeguard system software development. In: Proceedings of the 2Nd International Conference On Software Engineering, ICSE ’76. IEEE Computer Society Press, Los Alamitos, pp 312–321Google Scholar
  88. Stol K-J, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98CrossRefGoogle Scholar
  89. Tassey G (2002) The economic impacts of inadequate infrastructure for software testing. Technical report, National Institute of Standards and TechnologyGoogle Scholar
  90. Westland CJ (2002) The cost of errors in software development: evidence from industry. J Syst Softw 62(1):1–9CrossRefGoogle Scholar
  91. Willis RR, Rova RM, Scott MD, Johnson MJ, Ryskowski JF, Moon JA, Winfield TO, Shumate KC (1998) Hughes aircrafts widespread deployment of a continuously improving software process. Technical report, Carnegie Mellon University - Software Engineering InstituteGoogle Scholar
  92. Wohlin C (2014) Guidelines for snowballing in systematic literature studies and a replication in software engineering. In: Proceedings of the 18th international conference on evaluation and assessment in software engineering, page Article 38Google Scholar
  93. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business MediaGoogle Scholar
  94. Yang Y, He Z, Mao K, Li Q, Nguyen V, Boehm BW, Valerdi R (2013) Analyzing and handling local bias for calibrating parametric cost estimation models. Inf Softw Technol 55(8):1496–1511CrossRefGoogle Scholar
  95. Ye Y, Xie L, He Z, Qi L, Nguyen V, Boehm BW, Valerdi R (2011) Local bias and its impacts on the performance of parametric estimation models. In: PROMISEGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.CSNorth Carolina State UniversityRaleighUSA
  2. 2.Software Engineering InstituteCarnegie Mellon UniversityPittsburghUSA
  3. 3.Fraunhofer CESECollege ParkUSA

Personalised recommendations