Empirical Software Engineering

, Volume 22, Issue 6, pp 3219–3253 | Cite as

Curating GitHub for engineered software projects

  • Nuthan Munaiah
  • Steven Kroh
  • Craig Cabrey
  • Meiyappan Nagappan


Software forges like GitHub host millions of repositories. Software engineering researchers have been able to take advantage of such a large corpora of potential study subjects with the help of tools like GHTorrent and Boa. However, the simplicity in querying comes with a caveat: there are limited means of separating the signal (e.g. repositories containing engineered software projects) from the noise (e.g. repositories containing home work assignments). The proportion of noise in a random sample of repositories could skew the study and may lead to researchers reaching unrealistic, potentially inaccurate, conclusions. We argue that it is imperative to have the ability to sieve out the noise in such large repository forges. We propose a framework, and present a reference implementation of the framework as a tool called reaper, to enable researchers to select GitHub repositories that contain evidence of an engineered software project. We identify software engineering practices (called dimensions) and propose means for validating their existence in a GitHub repository. We used reaper to measure the dimensions of 1,857,423 GitHub repositories. We then used manually classified data sets of repositories to train classifiers capable of predicting if a given GitHub repository contains an engineered software project. The performance of the classifiers was evaluated using a set of 200 repositories with known ground truth classification. We also compared the performance of the classifiers to other approaches to classification (e.g. number of GitHub Stargazers) and found our classifiers to outperform existing approaches. We found stargazers-based classifier (with 10 as the threshold for number of stargazers) to exhibit high precision (97%) but an inversely proportional recall (32%). On the other hand, our best classifier exhibited a high precision (82%) and a high recall (86%). The stargazer-based criteria offers precision but fails to recall a significant portion of the population.


Mining software repositories GitHub Data curation Curation tools 


  1. Allamanis M, Sutton C (2013) Mining source code repositories at massive scale using language modeling Proceedings of the 10th working conference on mining software repositories, IEEE Press, Piscataway, NJ, USA, MSR ’13, pp 207–216. http://dl.acm.org/citation.cfm?id=2487085.2487127
  2. Belady LA, Lehman MM (1976) A model of large program development. IBM Syst J 15(3):225–252. doi:10.1147/sj.153.0225 CrossRefMATHGoogle Scholar
  3. Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t touch my code!: examining the effects of ownership on software quality Proceedings of the 19th ACM SIGSOFT symposium and the 13th european conference on foundations of software engineering, ACM, New York, NY, USA, ESEC/FSE ’11, pp 4–14. doi:10.1145/2025113.2025119
  4. Bissyandé TF, Lo D, Jiang L, Réveillère L, Klein J, Traon YL (2013) Got issues? Who cares about it? A large scale investigation of issue trackers from GitHub 2013 IEEE 24th international symposium on software reliability engineering (ISSRE), pp 188–197. doi:10.1109/ISSRE.2013.6698918
  5. Bissyandé TF, Thung F, Lo D, Jiang L, Réveillère L (2013a) Orion: a software project search engine with integrated diverse software artifacts 2013 18th international conference on engineering of complex computer systems, pp 242–245. doi:10.1109/ICECCS.2013.42
  6. Bissyandé TF, Thung F, Lo D, Jiang L, Réveillère L (2013b) Popularity, interoperability, and impact of programming languages in 100,000 open source projects 2013 IEEE 37th annual computer software and applications conference, pp 303–312. doi:10.1109/COMPSAC.2013.55
  7. Breiman L (2001) Random forests. Mach Learn 45(1):5–32. doi:10.1023/A:1010933404324 CrossRefMATHGoogle Scholar
  8. CA Technologies (2016) Waffle.io - work better on GitHub issues. https://waffle.io/, accessed: 2016-03-11
  9. Carlo Z (2016) Githut - programming languages and GitHub. http://githut.info, accessed: 2016-03-11
  10. Codetree Studios (2016) Codetree - GitHub issues, managed. https://codetree.com/, accessed: 2016-03-11
  11. Danial A (2014) CLOC – Count lines of code. http://cloc.sourceforge.net/, accessed: 2016-03-11, version: 1.62
  12. de Souza CB, Anquetil N, de Oliveira KM (2005) A study of the documentation essential to software maintenance Proceedings of the 23rd annual international conference on design of communication: documenting & designing for pervasive information, ACM, New York, NY, USA, SIGDOC ’05, pp 68–75. doi:10.1145/1085313.1085331
  13. Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: a language and infrastructure for analyzing ultra-large-scale software repositories Proceedings of the 2013 international conference on software engineering, IEEE Press, Piscataway, NJ, USA, ICSE ’13, pp 422–431. http://dl.acm.org/citation.cfm?id=2486788.2486844
  14. Eick SG, Graves TL, Karr AF, Marron JS, Mockus A (2001) Does code decay? Assessing the evidence from change management data. IEEE Trans Softw Eng 27(1):1–12. doi:10.1109/32.895984 CrossRefGoogle Scholar
  15. Emam KE, Benlarbi S, Goel N, Rai SN (2001) The confounding effect of class size on the validity of object-oriented metrics. IEEE Trans Softw Eng 27(7):630–650. doi:10.1109/32.935855 CrossRefGoogle Scholar
  16. GHTorrent (2016a) Hall of fame. http://ghtorrent.org/halloffame.html, accessed: 2016-03-11
  17. GHTorrent (2016b) The relational DB schema. http://ghtorrent.org/relational.html, accessed: 2016-03-11
  18. GitHub Inc (2016a) Github API v3—github developer guide. https://developer.github.com/v3/, accessed: 2016-03-11
  19. GitHub Inc (2016b) Github archive. https://www.githubarchive.org/, accessed: 2016-06-19
  20. GitHub Inc (2016c) No license - choose a license. http://choosealicense.com/no-license/, accessed: 2016-03-11
  21. Gousios G (2013) The GHTorrent dataset and tool suite Proceedings of the 10th working conference on mining software repositories, IEEE Press, Piscataway, NJ, USA, MSR ’13, pp 233–236. http://dl.acm.org/citation.cfm?id=2487085.2487132
  22. Guzman E, Azócar D, Li Y (2014) Sentiment analysis of commit comments in GitHub: an empirical study Proceedings of the 11th working conference on mining software repositories, ACM, New York, NY, USA, MSR 2014, pp 352–355. doi:10.1145/2597073.2597118
  23. HuBoard Inc (2016) Huboard - github issues made awesome. https://huboard.com/, accessed: 2016-03-11
  24. Iowa State University (2016) Publications related to Boa - Boa - Iowa State University. http://boa.cs.iastate.edu/papers/, accessed: 2016-03-11
  25. Jarczyk O, Gruszka B, Jaroszewicz S, Bukowski L, Wierzbicki A (2014) Github projects. Quality analysis of open-source software. Springer International Publishing, Cham, pp 80–94. doi:10.1007/978-3-319-13734-6_6 Google Scholar
  26. Kalliamvakou E, Gousios G, Blincoe K, Singer L, German DM, Damian D (2014) The promises and perils of mining GitHub Proceedings of the 11th working conference on mining software repositories, ACM, New York, NY, USA, MSR 2014, pp 92–101. doi:10.1145/2597073.2597074
  27. Kochhar PS, Bissyandé TF, Lo D, Jiang L (2013) Adoption of software testing in open source projects–a preliminary study on 50,000 projects 2013 17th european conference on software maintenance and reengineering, pp 353–356. doi:10.1109/CSMR.2013.48
  28. Kofink A (2015) Contributions of the under-appreciated: gender bias in an open-source ecology Companion proceedings of the 2015 ACM SIGPLAN international conference on systems, programming, languages and applications: Software for humanity, ACM, New York, NY, USA, SPLASH Companion 2015, pp 83–84. doi:10.1145/2814189.2815369
  29. Laplante P (2007) What every engineer should know about software engineering. What every engineer should know. CRC PressGoogle Scholar
  30. Mockus A, Fielding R T, Herbsleb J (2000) A case study of open source software development: the apache server Proceedings of the 2000 international conference on software engineering. ICSE 2000 the new millennium, pp 263–272. doi:10.1145/337180.337209
  31. Munaiah N, Kroh S, Cabrey C, Nagappan M (2016a) Home of the reporeapers. https://reporeapers.github.io, accessed: 2016-03-11
  32. Munaiah N, Kroh S, Cabrey C, Parikh N (2016b) Reaper - reference implementation. https://github.com/reporeapers/reaper, accessed: 2016-03-11
  33. Nagappan N (2007) Potential of open source systems as project repositories for empirical studies working group results. Springer, Berlin, pp 103–107. doi:10.1007/978-3-540-71301-2_29 Google Scholar
  34. Nagappan N, Williams L, Osborne J, Vouk M, Abrahamsson P (2005) Providing test quality feedback using static source code and automatic test suite metrics 16th IEEE international symposium on software reliability engineering (ISSRE’05), pp 10–94. doi:10.1109/ISSRE.2005.35
  35. Ray B, Posnett D, Filkov V, Devanbu P (2014) A large scale study of programming languages and code quality in github Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, ACM, New York, NY, USA, FSE 2014, pp 155–165. doi:10.1145/2635868.2635922
  36. Rosenberg J (1997) Some misconceptions about lines of code Proceedings fourth international software metrics symposium, pp 137–142. doi:10.1109/METRIC.1997.637174
  37. Ross SM (2003) Peirce’s criterion for the elimination of suspect experimental data. J Eng Technol 20(2):38–41Google Scholar
  38. Sajnani H, Saini V, Ossher J, Lopes CV (2014) Is popularity a measure of quality? an analysis of maven components 2014 IEEE international conference on software maintenance and evolution, pp 231–240. doi:10.1109/ICSME.2014.45
  39. Software Freedom Law Center (2012) Managing copyright information within a free software project - software freedom law center. http://softwarefreedom.org/resources/2012/managingcopyrightinformation.html, accessed: 2015-05-15
  40. Syer MD, Nagappan M, Hassan AE, Adams B (2013) Revisiting prior empirical findings for mobile apps: an empirical case study on the 15 most popular open-source android apps Proceedings of the 2013 conference of the center for advanced studies on collaborative research, IBM corp., riverton, NJ, USA, CASCON ’13, pp 283–297. http://dl.acm.org/citation.cfm?id=2555523.2555553
  41. Tung YH, Chuang CJ, Shan HL (2014) A framework of code reuse in open source software The 16th asia-pacific network operations and management symposium, pp 1–6. doi:10.1109/APNOMS.2014.6996525
  42. Vasilescu B, van Schuylenburg S, Wulms J, Serebrenik A, van den Brand MGJ (2014) Continuous integration in a social-coding world empirical evidence from GitHub 2014 IEEE international conference on software maintenance and evolution, pp 401–405. doi:10.1109/ICSME.2014.62
  43. Vendome C (2015) A large scale study of license usage on GitHub 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2, pp 772–774. doi:10.1109/ICSE.2015.245
  44. Whitehead J, Mistrík I, Grundy J, van der Hoek A (2010) Collaborative software engineering: concepts and techniques. Springer, Berlin, pp 1–30. doi:10.1007/978-3-642-10294-3_1 CrossRefMATHGoogle Scholar
  45. Zaidman A, Rompaey BV, Demeyer S, v Deursen A (2008) Mining software repositories to study co-evolution of production & test code 2008 1st international conference on software testing, verification, and validation, pp 220–229. doi:10.1109/ICST.2008.47
  46. Zenhub (2016) Zenhub - project management for agile teams on GitHub. https://www.zenhub.io/, accessed: 2016-03-11
  47. Zhu H, Hall PAV, May JHR (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366–427. doi:10.1145/267580.267590 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Software EngineeringRochester Institute of TechnologyRochesterUSA
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada

Personalised recommendations