Software provenance tracking at the scale of public source code

Abstract

We study the possibilities to track provenance of software source code artifacts within the largest publicly accessible corpus of publicly available source code, the Software Heritage archive, with over 4 billions unique source code files and 1 billion commits capturing their development histories across 50 million software projects. We perform a systematic and generic estimate of the replication factor across the different layers of this corpus, analysing how much the same artifacts (e.g., SLOC, files or commits) appear in different contexts (e.g., files, commits or source code repositories). We observe a combinatorial explosion in the number of identical source code files across different commits. To discuss the implication of these findings, we benchmark different data models for capturing software provenance information at this scale, and we identify a viable solution, based on the properties of isochrone subgraphs, that is deployable on commodity hardware, is incremental and appears to be maintainable for the foreseeable future. Using these properties, we quantify, at a scale never achieved previously, the growth rate of original, i.e. never-seen-before, source code files and commits, and find it to be exponential over a period of more than 40 years.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Notes

  1. 1.

    For example, hundreds of thousands of projects migrated from GitHub to GitLab.com in the days following the acquisition of GitHub by Microsoft in Summer 2018, see https://about.gitlab.com/2018/06/03/movingtogitlab/.

  2. 2.

    See https://www.gnome.org/news/2018/05/gnome-moves-to-gitlab-2/

  3. 3.

    Each claiming to have the largest knowledge base of software artifacts, see for example https://en.wikipedia.org/wiki/Open_Hub, https://www.theserverside.com/discussions/thread/62521.html

  4. 4.

    Some studies have analyzed up to a few million projects, but this is still a tiny fraction of all public source code.

  5. 5.

    see, e.g., https://hblok.net/blog/storage/

References

  1. Abramatic J-F, Di Cosmo R, Zacchiroli S (2018) Building the universal archive of source code. Commun ACM 61(10):29–31

    Article  Google Scholar 

  2. Albert R, Barabási A (2002) Statistical mechanics of complex networks. Rev Mod Phys 74(1):47

    MathSciNet  Article  Google Scholar 

  3. Alexandru CV, Panichella S, Gall HC (2017) Reducing redundancies in multi-revision code analysis. In: Pinzger M, Bavota G, Marcus A (eds) IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER 2017, Klagenfurt, Austria, February 20-24, 2017, pp 148–159

  4. Alexandru CV, Panichella S, Proksch S, Gall HC (2019) Redundancy-free analysis of multi-revision software artifacts. Empir Softw Eng 24(1):332–380

    Article  Google Scholar 

  5. Allamanis M, Sutton CA (2013) Mining source code repositories at massive scale using language modeling. In: Zimmermann T, Di Penta M, Kim S (eds) Proceedings of the 10th working conference on mining software repositories, MSR ’13, San Francisco, CA, USA, May 18-19, 2013, pp 207–216. IEEE Computer Society

  6. Thomas J., Bergin T (2007) A history of the history of programming languages. Commun ACM 50(5):69–74

    Article  Google Scholar 

  7. Biazzini M, Baudry B (2014) May the fork be with you: novel metrics to analyze collaboration on github. In: Proceedings of the 5th international workshop on emerging trends in software metrics, pp 37–43. ACM

  8. Borges H, Hora A, Valente MT (2016) Understanding the factors that impact the popularity of github repositories. In 2016 IEEE international conference on software maintenance and evolution (ICSME), pp 334–344

  9. Brooks FP Jr (1978) The mythical man-month: essays on software engineering, 1st edn. Addison-Wesley Longman Publishing Co., Inc., Boston

    Google Scholar 

  10. Caneill M, Germȧn DM, Zacchiroli S (2017) The Debsources dataset: Two decades of free and open source software. Empir Softw Eng 22(3):1405–1437

    Article  Google Scholar 

  11. Capraro M, Riehle D (2017) Inner source definition, benefits, and challenges. ACM Comput Surv (CSUR) 49(4):67

    Article  Google Scholar 

  12. Crowston K, Wei K, Howison J, Wiggins A (2008) Free/libre open-source software development: What we know and what we do not know. ACM Comput Surv 44:27:1–7:35

    Google Scholar 

  13. Davies J, Germȧn DM, Godfrey MW, Hindle A (2013) Software bertillonage - determining the provenance of software development artifacts. Empir Softw Eng 18 (6):1195–1237

    Article  Google Scholar 

  14. Di Cosmo R, Zacchiroli S (2017) Software heritage: Why and how to preserve software source code. In: Proceedings of the 14th international conference on digital preservation, iPRES 2017, Kyoto, Japan. Available from https://hal.archives-ouvertes.fr/hal-01590958

  15. Dorogovtsev SN, Mendes JFF (2002) Evolution of networks. Adv Phys 51 (4):1079–1187

    Article  Google Scholar 

  16. Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, pp 422–431. IEEE Press

  17. Germán DM, Di Penta M, Guéhéneuc Y-G, Antoniol G (2009) Code siblings: Technical and legal implications of copying code between applications. In: Godfrey and Whitehead (Godfrey and Godfrey 2009), pp 81–90

  18. Gkortzis A, Mitropoulos D, Spinellis D (2018) Vulinoss: A dataset of security vulnerabilities in open-source systems. In: Zaidman et al. (Zaidman et al 2018), pp 18–21

  19. Godfrey MW (2015) Understanding software artifact provenance. Sci Comput Program 97:86–90

    Article  Google Scholar 

  20. Godfrey MW, German DM, Davies J, Hindle A (2011) Determining the provenance of software artifacts. In: Proceedings of the 5th international workshop on software clones, IWSC ’11. ACM, New York, pp 65–66

  21. Godfrey MW, Godfrey J (eds) (2009) Proceedings of the 6th international working conference on mining software repositories, MSR 2009 (Co-located with ICSE). Proceedings,. IEEE Computer Society, Vancouver

  22. Gousios G, Pinzger M, van Deursen A (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, pp 345–355. ACM

  23. Grieco G, Luis Grinblat G, Uzal L, Rawat S, Feist J, Mounier L (2016) Toward large-scale vulnerability discovery using machine learning. In: Proceedings of the 6th ACM conference on data and application security and privacy, CODASPY ’16. ACM, New York, pp 85–96

  24. Hassan AE (2008) The road ahead for mining software repositories. In: Frontiers of software maintenance FoSM 2008., pp 48–57. IEEE

  25. Hatton L, Spinellis D, van Genuchten M (2017) The long-term growth rate of evolving software: Empirical results and implications. Journal of Software: Evolution and Process, 29(5)

  26. Herraiz I, Rodríguez D, Robles G, Gonzȧlez-Barahona JM (2013) The evolution of the laws of software evolution: A discussion based on a systematic literature review. ACM Comput Surv 46(2):28:1–28:28

    Article  Google Scholar 

  27. Ishio T, Kula RG, Kanda T, German DM, Inoue K (2016) Software ingredients: Detection of Third-Party component reuse in java software release. In: 2016 IEEE/ACM, 13th working conference on mining software repositories (MSR), pp 339–350

  28. Jiang J, Lo D, He J, Xia X, Kochhar PS, Li Z (2017) Why and how developers fork what from whom in github. Empir Softw Eng 22(1):547–578

    Article  Google Scholar 

  29. Lehman MM (1980) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1:213–221

    Article  Google Scholar 

  30. Leskovec J, Sosič R (2016) Snap: A general-purpose network analysis and graph-mining library. ACM Trans Intell Syst Technol (TIST) 8(1):1

    Article  Google Scholar 

  31. Levin DA, Pedersen PM, Shah AC (2009) Resolving license dependencies for aggregations of legally protectable content, June 2009. CIB: H04K1/00; G06Q10/00; G06Q50/00; H04L9/00

  32. Li F, Paxson V (2017) A large-scale empirical study of security patches. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, CCS ’17. ACM, New York, pp 2201–2215

  33. Lopes CV, Maj P, Martins P, Saini V, Yang D, Zitny J, Sajnani H, Vitek J (2017) Dėjȧvu: A map of code duplicates on github. PACMPL 1(OOPSLA) 28:1–84

    Google Scholar 

  34. Ma Y, Bogart C, Amreen S, Zaretzki R, Mockus A (2019) World of code: an infrastructure for mining the universe of open source VCS data. In: Storey et al. (Storey et al 2019), pp 143–154

  35. Markovtsev V, Long W (2018) Public git archive: A big code dataset for all. In: Zaidman et al. (Zaidman et al 2018), pp 34–37

  36. Martinez M, Monperrus M (2015) Mining software repair models for reasoning on the search space of automated program fixing. Empir Softw Eng 20(1):176–205

    Article  Google Scholar 

  37. Merkle RC (1987) A digital signature based on a conventional encryption function. In: Pomerance C (ed) Advances in cryptology - CRYPTO ’87, A conference on the theory and applications of cryptographic techniques, vol 293 of lecture notes in computer science, pp 369–378. Springer

  38. Mockus A (2009) Amassing and indexing a large sample of version control systems: Towards the census of public source code history. In: Godfrey and Whitehead (Godfrey and Godfrey 2009), pp 11–20

  39. Mockus A (2009) Amassing and indexing a large sample of version control systems: Towards the census of public source code history. In: Proceedings of the 2009 6th IEEE international working conference on mining software repositories, MSR ’09. IEEE Computer Society, Washington, pp 11–20

  40. Newman M, Barabasi A-L, Watts DJ (2006) The structure and dynamics of networks: (Princeton studies in complexity). Princeton University Press, Princeton

    Google Scholar 

  41. Pietri A, Spinellis D, Zacchiroli S (2019) The software heritage graph dataset: Public software development under one roof. In Storey et al. (Storey et al 2019), pp 138–142

  42. Rastogi A, Nagappan N (2016) Forking and the sustainability of the developer community participation–an empirical investigation on outcomes and reasons. In: 2016 IEEE 23rd international conference on software analysis, evolution, and Reengineering (SANER), vol 1, pp 102–111. IEEE

  43. Rattan D, Bhatia R, Singh M (2013) Software clone detection: A systematic review. Inf Softw Technol 55(7):1165–1199

    Article  Google Scholar 

  44. Rousseau G, Biais M (2010) Computer tool for managing digital documents. CIB: G06F17/30; G06F21/10; G06F21/64

  45. Roy CK, Cordy JR (2007) A survey on software clone detection research Technical Report 115, Queen’s School of Computing

  46. Semura Y, Yoshida N, Choi E, Inoue K (2017) Ccfindersw: Clone detection tool with flexible multilingual tokenizatio. In: Lv J, Zhang HJ, Hinchey M, Liu X (eds) 24th Asia-Pacific software engineering conference, APSEC 2017. IEEE Computer Society, Nanjing, pp 654–659

  47. Spinellis D (2017) A repository of Unix history and evolution. Empir Softw Eng 22(3):1372–1404

    Article  Google Scholar 

  48. Squire M (2017) The lives and deaths of open source code forges. In: Morgan L (ed) Proceedings of the 13th international symposium on open collaboration, OpenSym Galway, Ireland, August 23-25, 2017, pp 15:1–15:8. ACM

  49. Stol K-J, Fitzgerald B (2014) Inner source–adopting open source development practices in organizations: a tutorial. IEEE Softw 32(4):60–67

    Article  Google Scholar 

  50. Storey M-AD, Adams B, Haiduc S (eds) (2019) Proceedings of the 16th international conference on mining software repositories, MSR 2019, 26-27. IEEE / ACM, Montreal

  51. Svajlenko J, Roy CK (2017) Fast and flexible large-scale clone detection with cloneworks. In: Uchitel S, Orso A, Robillard MP (eds) Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 - companion Volume, pp 27–30. IEEE Computer Society

  52. Thummalapenta S, Cerulo L, Aversano L, Di Penta M (2010) An empirical study on the maintenance of source code clones. Empir Softw Eng 15(1):1–34

    Article  Google Scholar 

  53. Thung F, Bissyande TF, Lo D, Jiang L (2013) Network structure of social coding in github. In: 2013 17th European Conference on Software Maintenance and Reengineering, pp 323–326. IEEE

  54. Tiwari NM, Upadhyaya G, Rajan H (2016) Candoia: A platform and ecosystem for mining software repositories tools. In: Dillon LK, Visser W, Williams L (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, pp 759–764. ACM

  55. Tuunanen T, Koskinen Ji, Kärkkäinen T (2009) Automated software license analysis. Autom Softw Eng 16(3-4):455–490

    Article  Google Scholar 

  56. Vendome C. (2015) A large scale study of license usage on github. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2, pp 772–774

  57. Waldin R, Zhang J (2009) Determining a document similarity metric, July 2009. CIB: G06F17/30

  58. Wu Y, Manabe Y, Kanda T, Germȧn DM, Inoue K (2017) Analysis of license inconsistency in large collections of open source projects. Empir Softw Eng 22 (3):1194–1222

    Article  Google Scholar 

  59. Zaidman A, Kamei Y, Hill E (eds) (2018) Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018. ACM, Gothenburg

  60. Zimmermann T, Premraj R, Zeller A (2007) Predicting defects for eclipse. In: International workshop on predictor models in software engineering, 2007 PROMISE’07: ICSE Workshops 2007, pp 9–9

  61. Zimmermann T, Weißgerber P, Diehl S, Zeller A (2004) Mining version histories to guide software changes. In: Finkelstein A, Estublier J, Rosenblum DS (eds) 26th international conference on software engineering (ICSE 2004), 23-28 May 2004, Edinburgh, pp 563–572

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for precious feedback that allowed us to significantly improve this article.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Guillaume Rousseau.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Miryung Kim

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rousseau, G., Di Cosmo, R. & Zacchiroli, S. Software provenance tracking at the scale of public source code. Empir Software Eng 25, 2930–2959 (2020). https://doi.org/10.1007/s10664-020-09828-5

Download citation

Keywords

  • Software evolution
  • Open source
  • Clone detection
  • Source code tracking
  • Mining software repositories
  • Provenance tracking