The effectiveness of test-driven development: an industrial case study

Abstract

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an industrial environment. Three comparable medium-sized projects were observed during their development cycle. Two projects were driven without TDD practice, while the third one introduced TDD into the development process. The effectiveness of TDD was expressed in terms of external code quality, productivity, and maintainability. Our results indicate that the TDD developers produced higher quality code that is easier to maintain, although we did observe a reduction in productivity.

This is a preview of subscription content, access via your institution.

Fig. 1

Notes

  1. 1.

    http://www.extremeprogramming.org/rules/unittests.html

  2. 2.

    In incremental development, the releases are defined by beginning with one small, functional subsystem and then adding functionality with each new release (Pflegeer and Atlee 2006).

  3. 3.

    JUnit—A regression testing framework written by Erich Gamma and Kent Beck, http://www.junit.org.

  4. 4.

    CCCC—C and C ++ Code Counter http://cccc.sourceforge.net/, a free software tool for the measurement of source code related metrics by Tim Littlefair.

  5. 5.

    Session Initiation Protocol, Internet Engineering Task Force (IETF), RFC 3261, http://datatracker.ietf.org/doc/rfc3261/).

  6. 6.

    Primavera Progress Reporter is a web-based program that connects project team members across projects, throughout an enterprise.

  7. 7.

    ISO/IEC 9,126. (2001). Software engineering—product quality—part 1: Quality model. ISO 2001.

  8. 8.

    Cyclomatic complexity is a measure of the complexity of code related to the number of ways there are to traverse a piece of code (McCabe 1976).

  9. 9.

    An unacceptable behavior of the software is defined as a failure. A fault is a defect in the program that, when executed with certain input data, causes a failure.

  10. 10.

    See Albrecht and Gaffney (1983) for details.

  11. 11.

    Unit test code was excluded.

  12. 12.

    Sanchez et. al. 2007 reported the average ratio across all ten releases as 0.61.

References

  1. Albrecht, A. J., & Gaffney, J. E., Jr. (1983). Software function, source lines of code, and development effort prediction: A software science validation. IEEE Transactions on Software Engineering, SE-9(6), 639–648.

    Google Scholar 

  2. Beck, K. (2001). Aim, fire. IEEE Software, 18(5), 87–89.

    Article  Google Scholar 

  3. Beck, K. (2003). Test driven development: by example. USA: Addison Wesley Professional.

    Google Scholar 

  4. Bhat, T., Nagappan, N. (2006). Evaluating the efficacy of test-driven development: industrial case studies. Proceedings of International Symposium, Empirical Software Engineering (ISESE 06), ACM Press, 356–363.

  5. Crispin, Lisa. (2006). Driving software quality: how test-driven development impacts software quality. IEEE Software, 23(6), 70–71. doi:10.1109/MS.2006.157.

    Article  Google Scholar 

  6. Cusumano, M. A., & Selby, R. W. (1997). How microsoft builds software. Communications of the ACM, 40(6), 53–61.

    Article  Google Scholar 

  7. Damm, L.-O., & Lundberg, L. (2006). Results from introducing component-level test automation and test-driven development. Journal of Systems and Software, 79, 1001–1014. Elsevier Science Inc.

    Article  Google Scholar 

  8. Edwards, S. H. (2003). Using test-driven development in the classroom: providing students with concrete feedback on performance. Proceedings of International Conference on Education and Information Systems: Technologies and Applications (EISTA ‘03).

  9. Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226–237.

    Article  Google Scholar 

  10. Fenton, N. E., & Ohlsson, N. (2000). Quantitative analysis of faults and failures in a complex software system. IEEE Transactions on Software Engineering, 26(8), 797–814.

    Article  Google Scholar 

  11. Fenton, N. E., & Pfleeger, S. L. (1998). Software metrics: A rigorous & practical approach (2nd ed.). Boston, USA: PWS Publishing Company.

    Google Scholar 

  12. Fowler, M., Beck, K., Brant, J., Opdyke, W., & Roberts, D. (1999). Refactoring: Improving the design of existing code. USA: Addison Wesley.

    Google Scholar 

  13. George, B., Williams, L. (2003). An initial investigation of test driven development in industry. ACM Symposium on Applied Computing (SAC), March.

  14. Janzen, D. S., & Saiedian, H. (2008). Does test-driven development really improve software design quality? IEEE Software, 25(2), 77–84.

    Article  Google Scholar 

  15. Jeffries, R. E. (1999). Extreme Testing. Software Testing and Quality Engineering, 1(2), 23–26, March/April.

  16. Jeffries, R., Melnik, G. (2007). TDD: The art of fearless programming. IEEE Software, May/June, 24–30.

  17. Jovanović, D., Dogša, T. (2003). Comparison of software development models and their usage in computer-telephony systems. In ConTEL 2003: Proceedings of the 7th international conference on telecommunications, University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, Croatia (vol. 2, pp. 587–592), June 11–13.

  18. Kitchenham, B., Pickard, L., & Pfleeger, S. L. (1995). Case studies for method and tool evaluation. IEEE Software, 12(4), 52–62.

    Article  Google Scholar 

  19. Madeyski, L. (2005). Preliminary analysis of the effects of pair programming and test-driven development on the external code quality. Frontiers in Artificial Intelligence and Applications.Proceeding of the 2005 conference on Software Engineering, vol. 130, pp. 113–123.

  20. Madeyski, L., Szała, Ł. (2007). The impact of test-driven development on software development productivity—An empirical study. Software Process Improvement 14th European Conference, EuroSPI 2007, Potsdam, Germany, September 26–28, 2007 in Proceedings lecture notes in computer science, Springer Berlin/Heidelberg, vol. 4764.

  21. Malik, S., Palencia, J. R. (1999). Synchronize and Stabilize vs. Open-source–An analysis of 2 team-organization models when developing consumer level shrink-wrapped software. Computer Science 95.314A Research Report, Carleton University, Ottawa, Ontario, Canada.

  22. Martin, R. C., Newkirk, J. W., Kess, R. S. (2002). Agile software development, principles, patterns, and practices (1st ed.). Prentice Hall, Englewood Cliffs.

  23. Maurer, F., Martel, S. (2002). On the productivity of agile software practices: An industrial case study. Technical report, University of Calgary, Department of Computer Science, Calgary, Alberta, Canada.

  24. Maximilien, M., Williams, L. (2003). Assessing test-driven development at IBM. Proceedings of 25th International Conference on Software Engineering (pp. 564–569).

  25. McCabe, T. J. (1976). A complexity measure. IEEE Transactions on Software Engineering, 2(4), 308–320.

    MathSciNet  Article  Google Scholar 

  26. Mueller, M. M., & Hagner, O. (2002). Experiment about Test-First Programming. IEE Proceedings–Software, 149(5), 131–136.

    Google Scholar 

  27. Newkirk, J., Vorontsov, A. (2004). Test-driven development in microsoft NET. Microsoft Press.

  28. Pfleeger, S. L., & Atlee, J. M. (2006). Software engineering: Theory and practice. New Jersey, USA: Prentice Hall.

    Google Scholar 

  29. Sanchez, J., Williams, L., Maximilien, E. M. (2007). A Longitudinal Study of the Use of a Test-Driven development Practice in Industry. AGILE 2007 Conference (AGILE 2007), 13–17 August, Washington, DC, USA. IEEE Computer Society 2007, ISBN 0-7695-2872-4.

  30. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin Company.

    Google Scholar 

  31. Siniaalto, M., Abrahamsson, P. (2007). A comparative case study on the impact of test-driven development on program design and test coverage. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement, IEEE Computer Society, Washington, pp. 275–284, ISBN ~ ISSN: 1938–6451.

  32. Stephens, M., Rosenberg, D. (2004). The Irony of Extreme Programming. Dr. Dobb’s Journal, May 2004.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to David Batič.

Appendices

Appendix 1

See Table 3.

Table 3 Projects details

Appendix 2

See Table 4.

Table 4 Survey after TDD training

Appendix 3

See Table 5.

Table 5 Final survey

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Dogša, T., Batič, D. The effectiveness of test-driven development: an industrial case study. Software Qual J 19, 643–661 (2011). https://doi.org/10.1007/s11219-011-9130-2

Download citation

Keywords

  • Test-driven development
  • Testing and debugging
  • Testing strategies
  • Productivity
  • Maintainability
  • Software quality/SQA
  • Software engineering process