Software Quality Journal

, Volume 25, Issue 2, pp 343–372 | Cite as

Test-Driven Development in scientific software: a survey

  • Aziz Nanthaamornphong
  • Jeffrey C. CarverEmail author


Scientific software developers are increasingly employing various software engineering practices. Specifically, scientists are beginning to use Test-Driven Development (TDD). Even with this increasing use of TDD, the effect of TDD on scientific software development is not fully understood. To help scientific developers determine whether TDD is appropriate for their scientific projects, we surveyed scientific developers who use TDD to understand: (1) TDDs effectiveness, (2) the benefits and challenges of using TDD, and (3) the use of refactoring practices (an important part of the TDD process). Some key positive results include: (1) TDD helps scientific developers increase software quality, in particular functionality and reliability; and (2) TDD helps scientific developers reduce the number of problems in the early phase of projects. Conversely, some key challenges include: (1) TDD may not be effective for all types of scientific projects; and (2) Writing a good test is the most difficult task in TDD, particularly in a parallel computing environment. To summarize, TDD generally has a positive effect on the quality of scientific software, but it often requires a large effort investment. The results of this survey indicate the need for additional empirical evaluation of the use of TDD for the development of scientific software to help organizations make better decisions.


Software engineering Test-Driven Development Survey 



The authors gratefully thank all participants in the survey for their time and contributions. Jeffrey Carver would like to acknowledge partial support from NSF Grants 1243887 and 1445344.


  1. Abdel-Hamid, A. (2013). Refactoring as a lifeline: Lessons learned from refactoring. In Agile Conference (AGILE), 2013, pp. 129–136Google Scholar
  2. Anselm, L. S., & Juliet, M. C. (1990). Basics of qualitative research: Rounded theory procedures and techniques. Newbury Park, CA: Sage Publications.Google Scholar
  3. Beck, K. (2002). Test driven development: By example. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.Google Scholar
  4. Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Boston, MA: Addison-Wesley Professional.Google Scholar
  5. Carver, J. (2011). Development of a mesh generation code with a graphical front-end: A case study. Journal of End User Computing, 23(4), 1–16.CrossRefGoogle Scholar
  6. Carver, J. C., Kendall, R. P., Squires, S. E., & Post, D. E. (2007). Software development environments for scientific and engineering software: A series of case studies. In The 29th international conference on software engineering (pp. 550–559). MN: Minneapolis.Google Scholar
  7. Causevic, A., Sundmark, D., & Punnekkat, S. (2011). Factors limiting industrial adoption of test driven development: A systematic review. In The 4th international conference on software testing (pp. 337–346). Berlin: Verification and Validation.Google Scholar
  8. Desai, C., Janzen, D., & Savage, K. (2008). A survey of evidence for Test-Driven Development in academia. SIGCSE Bulletin, 40(2), 97–101.CrossRefGoogle Scholar
  9. Eclipse. (2013). Photran—An integrated development environment and refactoring tool for fortran. Accessed December 2013
  10. Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226–237. doi: 10.1109/TSE.2005.37 CrossRefGoogle Scholar
  11. Fowler, M. (1999). Refactoring: Improving the design of existing code. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.zbMATHGoogle Scholar
  12. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.zbMATHGoogle Scholar
  13. ISO IEC. (2011). Systems and software engineering: System and software quality requirements and evaluation (SQuaRE)—System and software quality models. ISO/IEC, 25010, 2011.Google Scholar
  14. Janzen, D., & Saiedian, H. (2005). Test-Driven Development concepts, taxonomy, and future direction. Computer, 38(9), 43–50. doi: 10.1109/MC.2005.314 CrossRefGoogle Scholar
  15. Kollanus, S. (2010). Test-Driven Development—Still a promising approach? In Proceedings of the 7th international conference on the quality of information and communications technology (pp. 403–408). Portugal: Porto.Google Scholar
  16. Koskela, L. (2007). Test driven: Practical TDD and acceptance TDD for java developers. Greenwich, CO: Manning Publications Co.Google Scholar
  17. Mens, T., & Tourwé, T. (2004). A survey of software refactoring. IEEE Transactions on Software Engineering, 30(2), 126–139.CrossRefGoogle Scholar
  18. Nanthaamornphong, A., Morris, K., Rouson, D., & Michelsen, H. (2013). A case study: Agile development in the community laser-induced incandescence modeling environment (CLiiME). In The 5th international workshop on software engineering for computational science and engineering (pp. 9–18).Google Scholar
  19. Nanthaamornphong, A., Carver, J., Morris, K., Michelsen, H., & Rouson, D. (2014). Building cliime via Test-Driven Development: A case study. Computing in Science Engineering, 16(3), 36–46.CrossRefGoogle Scholar
  20. Opdyke, W. F. (1992). Refactoring object-oriented frameworks. PhD thesis, University of Illinois at Urbana-Champaign, Champaign, Illinois, USA.Google Scholar
  21. Orchard, D., & Rice, A. (2013). Upgrading fortran source code using automatic refactoring. Proceedings of the international workshop on refactoring tools (pp. 29–32). Indiana: Indianapolis.Google Scholar
  22. Overbey, J., Xanthos, S., Johnson, R., & Foote, B. (2005). Refactorings for fortran and high-performance computing. In Proceedings of the 2nd international workshop on software engineering for high performance computing system applications (pp. 37–39). Missouri: St. Louis.Google Scholar
  23. Overbey, J. L., Negara, S., & Johnson, R. E. (2009). Refactoring and the evolution of fortran. In Proceedings of the international workshop on software engineering for computational science and engineering (pp. 28–34). British Columbia: Vancouver.Google Scholar
  24. Rafique, Y., & Misic, V. (2013). The effects of Test-Driven Development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835–856.CrossRefGoogle Scholar
  25. Ruparelia, N. B. (2010). Software development lifecycle models. SIGSOFT Software Engineering Notes, 35(3), 8–13.CrossRefGoogle Scholar
  26. Sanchez, J., Williams, L., Maximilien, E. (2007). On the sustained use of a Test-Driven Development practice at ibm. In Agile conference (AGILE), 2007 (pp 5–14)Google Scholar
  27. Sanders, R., & Kelly, D. (2008). Dealing with risk in scientific software development. IEEE Software, 25(4), 21–28.CrossRefGoogle Scholar
  28. Sletholt, M., Hannay, J., Pfahl, D., & Langtangen, H. (2012). What do we know about scientific software development’s agile practices? Computing in Science Engineering, 14(2), 24–37.CrossRefGoogle Scholar
  29. Sletholt, M. T., Hannay, J., Pfahl, D., Benestad, H. C., & Langtangen, H. P. (2011). A literature review of agile practices and their effects in scientific software development. In Proceedings of the 4th international workshop on software engineering for computational science and engineering (pp. 1–9). Hawaii: Honolulu.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Information and Communication TechnologyPrince of Songkla UniversityPhuketThailand
  2. 2.Department of Computer ScienceUniversity of AlabamaTuscaloosaUSA

Personalised recommendations