Skip to main content
SpringerLink
Log in

Analyzing the effects of test driven development in GitHub

  • Original Article
  • Sample SubCategory
  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Testing is an integral part of the software development lifecycle, approached with varying degrees of rigor by different process models. Agile process models recommend Test Driven Development (TDD) as a key practice for reducing costs and improving code quality. The objective of this work is to perform a cost-benefit analysis of this practice. To that end, we have conducted a comparative analysis of GitHub repositories that adopts TDD to a lesser or greater extent, in order to determine how TDD affects software development productivity and software quality. We classified GitHub repositories archived in 2015 in terms of how rigorously they practiced TDD, thus creating a TDD spectrum. We then matched and compared various subsets of these repositories on this TDD spectrum with control sets of equal size. The control sets were samples from all GitHub repositories that matched certain characteristics, and that contained at least one test file. We compared how the TDD sets differed from the control sets on the following characteristics: number of test files, average commit velocity, number of bug-referencing commits, number of issues recorded, usage of continuous integration, number of pull requests, and distribution of commits per author. We found that Java TDD projects were relatively rare. In addition, there were very few significant differences in any of the metrics we used to compare TDD-like and non-TDD projects; therefore, our results do not reveal any observable benefits from using TDD.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. Domain-specific types http://boa.cs.iastate.edu/docs/dsl-types.php.

  2. Here, we consider four time Δ categories (0 minutes, 1 hour, 1 day, 1 week). Repositories are assigned to the category with the shortest time Δ that is greater than all the time Δs found in that repository – hence if the largest Δ for any code in a repository is 5 minute, this entire repository would be in the “1 hour” category, and if the longest was 3 days, it would be in “1 week”.

  3. https://travis-ci.org/.

  4. Modified from Boa examples found at http://boa.cs.iastate.edu/.

  5. http://boa.cs.iastate.edu/boa/?q=boa/job/public/30188.

  6. http://junit.org/junit4/.

  7. http://testng.org/doc/index.html.

  8. http://developer.android.com/tools/testing/index.html.

  9. http://www.javatpoint.com/java-reflection.

References

  • Aickin M, Gensler H (1996) Adjusting for multiple testing when reporting research results: the bonferroni vs holm methods. Amer J Public Health 86(5):726–728

    Article  Google Scholar 

  • Athanasiou D, Nugroho A, Visser J, Zaidman A (2014) Test code quality and its relation to issue handling performance. IEEE Trans Softw Eng 40(11):1100–1125

    Article  Google Scholar 

  • Beck K (2003) Test-driven development: by example. Addison-Wesley Professional

  • Beck K, Andres C (2004) Extreme programming explained: embrace change, 2nd edn. Addison-Wesley Professional

  • Beller M, Gousios G, Panichella A, Zaidman A (2015a) When, how, and why developers (do not) test in their ides. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering. ACM, pp 179–190

  • Beller M, Gousios G, Zaidman A (2015b) How (much) do developers test? In: 2015 IEEE/ACM 37th IEEE International conference on software engineering (ICSE), vol 2. IEEE, pp 559-562

  • Bird C, Rigby PC, Barr ET, Hamilton DJ, German DM, Devanbu P (2009) The promises and perils of mining git. In: 6th IEEE International working conference on mining software repositories, 2009. MSR’09. IEEE, pp 1–10

  • Brack F (2016) Code review and test first development: how to leverage existing tools and processes to make sure an efficient test first approach is being applied. https://medium.com/@fagnerbrack/code-review-and-test-driven-development-4c19b69b5761. Accessed July 2017

  • Budhabhatti M (2008) Test-driven development and continuous integration for mobile applications. https://msdn.microsoft.com/en-us/library/bb985498.aspx. Accessed July 2017

  • DevIQ (2017) Test driven development explained. http://deviq.com/test-driven-development/. Accessed July 2017

Further Reading

  • Dyer R (2013) Bringing ultra-large-scale software repository mining to the masses with boa. Iowa State University, PhD thesis

    Google Scholar 

  • Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: a language and infrastructure for analyzing ultra-large-scale software repositories. In: 35th International conference on software engineering, ICSE, 2013, pp 422-431

  • Elliott E (2016) 5 common misconceptions about tdd and unit tests. https://medium.com/javascript-scene/5-common-misconceptions-about-tdd-unit-tests-863d5beb3ce9. Accessed July 2017

  • Fucci D, Erdogmus H, Turhan B, Oivo M, Juristo N (2016a) A dissection of test-driven development: does it really matter to test-first or to test-last? IEEE Trans Softw Eng

  • Fucci D, Scanniello G, Romano S, Shepperd M, Sigweni B, Uyaguari F, Turhan B, Juristo N, Oivo M (2016b) An external replication on the effects of test-driven development using a multi-site blind analysis approach. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement. ACM, p 3

  • Hindle A, Godfrey MW, Holt RC (2007) Release pattern discovery via partitioning: Methodology and case study. In: Fourth international workshop on mining software repositories, 2007. ICSE Workshops MSR’07. IEEE, pp 19-19

  • Hochberg Y (1988) A sharper bonferroni procedure for multiple tests of significance. Biometrika 75(4):800–802

    Article  MathSciNet  MATH  Google Scholar 

  • Holm S (1979) A simple sequentially rejective multiple test procedure. Scand J Stat 65–70

  • Jeffries R, Melnik G (2007) Guest editors’ introduction: Tdd–the art of fearless programming. IEEE Softw 24(3):24–30

    Article  Google Scholar 

  • Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 92–101

  • Lakeview Labs (2017) Chicago app development process. https://lakeviewlabs.io/process/development-process.html. Accessed July 2017

  • Pugh K (2010) Lean-Agile acceptance test-driven-development. Pearson Education

  • Rousseeuw PJ (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math 20:53–65

    Article  MATH  Google Scholar 

  • Santos EA, Hindle A (2016) Judging a commit by its cover: Correlating commit message entropy with build status on travis-ci. In: Proceedings of the 13th international conference on mining software repositories. ACM, pp 504–507

  • Travis-CI (2017) Learn to code with confidence. https://education.travis-ci.com/. Accessed July 2017

  • Vasa R, Lumpe M, Branch P, Nierstrasz O (2009) Comparative analysis of evolving software systems using the gini coefficient. In: IEEE International conference on software maintenance, 2009. ICSM 2009. IEEE, pp 179–188

  • Vasilescu B, Van Schuylenburg S, Wulms J, Serebrenik A, van den Brand MG (2015) Continuous integration in a social-coding world: empirical evidence from GitHub.** updated version with corrections**. arXiv preprint arXiv:151201862

  • Weiss K (2017) How to do test-driven development with merge control and continuous integration. http://www.izymes.com/how-to-do-test-driven-development-with-merge-control-and-continuous-integration/. Accessed July 2017

  • Winter D (2016) 9 benefits of test driven development. https://www.madetech.com/blog/9-benefits-of-test-driven-development. Accessed July 2017

  • Zaidman A, Van Rompaey B, Demeyer S, Van Deursen A (2008) Mining software repositories to study co-evolution of production & test code. In: 2008 1st International conference on software testing, verification, and validation. IEEE, pp 220–229

Download references

Acknowledgements

The authors wish to acknowledge the support of NSERC, and of AMII.

Author information

Authors and Affiliations

  1. Department of Computing Science, University of Alberta, Edmonton, Alberta, Canada

    Neil C. Borle, Meysam Feghhi, Eleni Stroulia, Russell Greiner & Abram Hindle

  2. Alberta Machine Intelligence Institute, Edmonton, Alberta, Canada

    Russell Greiner

Authors
  1. Neil C. Borle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meysam Feghhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eleni Stroulia
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Russell Greiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abram Hindle
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Neil C. Borle.

Additional information

Communicated by: Jeffrey C. Carver

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Borle, N.C., Feghhi, M., Stroulia, E. et al. Analyzing the effects of test driven development in GitHub. Empir Software Eng 23, 1931–1958 (2018). https://doi.org/10.1007/s10664-017-9576-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-017-9576-3

Keywords

Search

Navigation