Skip to main content
SpringerLink
Log in

Software provenance tracking at the scale of public source code

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

We study the possibilities to track provenance of software source code artifacts within the largest publicly accessible corpus of publicly available source code, the Software Heritage archive, with over 4 billions unique source code files and 1 billion commits capturing their development histories across 50 million software projects. We perform a systematic and generic estimate of the replication factor across the different layers of this corpus, analysing how much the same artifacts (e.g., SLOC, files or commits) appear in different contexts (e.g., files, commits or source code repositories). We observe a combinatorial explosion in the number of identical source code files across different commits. To discuss the implication of these findings, we benchmark different data models for capturing software provenance information at this scale, and we identify a viable solution, based on the properties of isochrone subgraphs, that is deployable on commodity hardware, is incremental and appears to be maintainable for the foreseeable future. Using these properties, we quantify, at a scale never achieved previously, the growth rate of original, i.e. never-seen-before, source code files and commits, and find it to be exponential over a period of more than 40 years.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. For example, hundreds of thousands of projects migrated from GitHub to GitLab.com in the days following the acquisition of GitHub by Microsoft in Summer 2018, see https://about.gitlab.com/2018/06/03/movingtogitlab/.

  2. See https://www.gnome.org/news/2018/05/gnome-moves-to-gitlab-2/

  3. Each claiming to have the largest knowledge base of software artifacts, see for example https://en.wikipedia.org/wiki/Open_Hub, https://www.theserverside.com/discussions/thread/62521.html

  4. Some studies have analyzed up to a few million projects, but this is still a tiny fraction of all public source code.

  5. see, e.g., https://hblok.net/blog/storage/

References

  • Abramatic J-F, Di Cosmo R, Zacchiroli S (2018) Building the universal archive of source code. Commun ACM 61(10):29–31

    Article  Google Scholar 

  • Albert R, Barabási A (2002) Statistical mechanics of complex networks. Rev Mod Phys 74(1):47

    Article  MathSciNet  Google Scholar 

  • Alexandru CV, Panichella S, Gall HC (2017) Reducing redundancies in multi-revision code analysis. In: Pinzger M, Bavota G, Marcus A (eds) IEEE 24th International Conference on Software Analysis, Evolution and Reengineering, SANER 2017, Klagenfurt, Austria, February 20-24, 2017, pp 148–159

  • Alexandru CV, Panichella S, Proksch S, Gall HC (2019) Redundancy-free analysis of multi-revision software artifacts. Empir Softw Eng 24(1):332–380

    Article  Google Scholar 

  • Allamanis M, Sutton CA (2013) Mining source code repositories at massive scale using language modeling. In: Zimmermann T, Di Penta M, Kim S (eds) Proceedings of the 10th working conference on mining software repositories, MSR ’13, San Francisco, CA, USA, May 18-19, 2013, pp 207–216. IEEE Computer Society

  • Thomas J., Bergin T (2007) A history of the history of programming languages. Commun ACM 50(5):69–74

    Article  Google Scholar 

  • Biazzini M, Baudry B (2014) May the fork be with you: novel metrics to analyze collaboration on github. In: Proceedings of the 5th international workshop on emerging trends in software metrics, pp 37–43. ACM

  • Borges H, Hora A, Valente MT (2016) Understanding the factors that impact the popularity of github repositories. In 2016 IEEE international conference on software maintenance and evolution (ICSME), pp 334–344

  • Brooks FP Jr (1978) The mythical man-month: essays on software engineering, 1st edn. Addison-Wesley Longman Publishing Co., Inc., Boston

    Google Scholar 

  • Caneill M, Germȧn DM, Zacchiroli S (2017) The Debsources dataset: Two decades of free and open source software. Empir Softw Eng 22(3):1405–1437

    Article  Google Scholar 

  • Capraro M, Riehle D (2017) Inner source definition, benefits, and challenges. ACM Comput Surv (CSUR) 49(4):67

    Article  Google Scholar 

  • Crowston K, Wei K, Howison J, Wiggins A (2008) Free/libre open-source software development: What we know and what we do not know. ACM Comput Surv 44:27:1–7:35

    Google Scholar 

  • Davies J, Germȧn DM, Godfrey MW, Hindle A (2013) Software bertillonage - determining the provenance of software development artifacts. Empir Softw Eng 18 (6):1195–1237

    Article  Google Scholar 

  • Di Cosmo R, Zacchiroli S (2017) Software heritage: Why and how to preserve software source code. In: Proceedings of the 14th international conference on digital preservation, iPRES 2017, Kyoto, Japan. Available from https://hal.archives-ouvertes.fr/hal-01590958

  • Dorogovtsev SN, Mendes JFF (2002) Evolution of networks. Adv Phys 51 (4):1079–1187

    Article  Google Scholar 

  • Dyer R, Nguyen HA, Rajan H, Nguyen TN (2013) Boa: A language and infrastructure for analyzing ultra-large-scale software repositories. In: Proceedings of the 2013 International Conference on Software Engineering, pp 422–431. IEEE Press

  • Germán DM, Di Penta M, Guéhéneuc Y-G, Antoniol G (2009) Code siblings: Technical and legal implications of copying code between applications. In: Godfrey and Whitehead (Godfrey and Godfrey 2009), pp 81–90

  • Gkortzis A, Mitropoulos D, Spinellis D (2018) Vulinoss: A dataset of security vulnerabilities in open-source systems. In: Zaidman et al. (Zaidman et al 2018), pp 18–21

  • Godfrey MW (2015) Understanding software artifact provenance. Sci Comput Program 97:86–90

    Article  Google Scholar 

  • Godfrey MW, German DM, Davies J, Hindle A (2011) Determining the provenance of software artifacts. In: Proceedings of the 5th international workshop on software clones, IWSC ’11. ACM, New York, pp 65–66

  • Godfrey MW, Godfrey J (eds) (2009) Proceedings of the 6th international working conference on mining software repositories, MSR 2009 (Co-located with ICSE). Proceedings,. IEEE Computer Society, Vancouver

  • Gousios G, Pinzger M, van Deursen A (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering, pp 345–355. ACM

  • Grieco G, Luis Grinblat G, Uzal L, Rawat S, Feist J, Mounier L (2016) Toward large-scale vulnerability discovery using machine learning. In: Proceedings of the 6th ACM conference on data and application security and privacy, CODASPY ’16. ACM, New York, pp 85–96

  • Hassan AE (2008) The road ahead for mining software repositories. In: Frontiers of software maintenance FoSM 2008., pp 48–57. IEEE

  • Hatton L, Spinellis D, van Genuchten M (2017) The long-term growth rate of evolving software: Empirical results and implications. Journal of Software: Evolution and Process, 29(5)

  • Herraiz I, Rodríguez D, Robles G, Gonzȧlez-Barahona JM (2013) The evolution of the laws of software evolution: A discussion based on a systematic literature review. ACM Comput Surv 46(2):28:1–28:28

    Article  Google Scholar 

  • Ishio T, Kula RG, Kanda T, German DM, Inoue K (2016) Software ingredients: Detection of Third-Party component reuse in java software release. In: 2016 IEEE/ACM, 13th working conference on mining software repositories (MSR), pp 339–350

  • Jiang J, Lo D, He J, Xia X, Kochhar PS, Li Z (2017) Why and how developers fork what from whom in github. Empir Softw Eng 22(1):547–578

    Article  Google Scholar 

  • Lehman MM (1980) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1:213–221

    Article  Google Scholar 

  • Leskovec J, Sosič R (2016) Snap: A general-purpose network analysis and graph-mining library. ACM Trans Intell Syst Technol (TIST) 8(1):1

    Article  Google Scholar 

  • Levin DA, Pedersen PM, Shah AC (2009) Resolving license dependencies for aggregations of legally protectable content, June 2009. CIB: H04K1/00; G06Q10/00; G06Q50/00; H04L9/00

  • Li F, Paxson V (2017) A large-scale empirical study of security patches. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, CCS ’17. ACM, New York, pp 2201–2215

  • Lopes CV, Maj P, Martins P, Saini V, Yang D, Zitny J, Sajnani H, Vitek J (2017) Dėjȧvu: A map of code duplicates on github. PACMPL 1(OOPSLA) 28:1–84

    Google Scholar 

  • Ma Y, Bogart C, Amreen S, Zaretzki R, Mockus A (2019) World of code: an infrastructure for mining the universe of open source VCS data. In: Storey et al. (Storey et al 2019), pp 143–154

  • Markovtsev V, Long W (2018) Public git archive: A big code dataset for all. In: Zaidman et al. (Zaidman et al 2018), pp 34–37

  • Martinez M, Monperrus M (2015) Mining software repair models for reasoning on the search space of automated program fixing. Empir Softw Eng 20(1):176–205

    Article  Google Scholar 

  • Merkle RC (1987) A digital signature based on a conventional encryption function. In: Pomerance C (ed) Advances in cryptology - CRYPTO ’87, A conference on the theory and applications of cryptographic techniques, vol 293 of lecture notes in computer science, pp 369–378. Springer

  • Mockus A (2009) Amassing and indexing a large sample of version control systems: Towards the census of public source code history. In: Godfrey and Whitehead (Godfrey and Godfrey 2009), pp 11–20

  • Mockus A (2009) Amassing and indexing a large sample of version control systems: Towards the census of public source code history. In: Proceedings of the 2009 6th IEEE international working conference on mining software repositories, MSR ’09. IEEE Computer Society, Washington, pp 11–20

  • Newman M, Barabasi A-L, Watts DJ (2006) The structure and dynamics of networks: (Princeton studies in complexity). Princeton University Press, Princeton

    Google Scholar 

  • Pietri A, Spinellis D, Zacchiroli S (2019) The software heritage graph dataset: Public software development under one roof. In Storey et al. (Storey et al 2019), pp 138–142

  • Rastogi A, Nagappan N (2016) Forking and the sustainability of the developer community participation–an empirical investigation on outcomes and reasons. In: 2016 IEEE 23rd international conference on software analysis, evolution, and Reengineering (SANER), vol 1, pp 102–111. IEEE

  • Rattan D, Bhatia R, Singh M (2013) Software clone detection: A systematic review. Inf Softw Technol 55(7):1165–1199

    Article  Google Scholar 

  • Rousseau G, Biais M (2010) Computer tool for managing digital documents. CIB: G06F17/30; G06F21/10; G06F21/64

  • Roy CK, Cordy JR (2007) A survey on software clone detection research Technical Report 115, Queen’s School of Computing

  • Semura Y, Yoshida N, Choi E, Inoue K (2017) Ccfindersw: Clone detection tool with flexible multilingual tokenizatio. In: Lv J, Zhang HJ, Hinchey M, Liu X (eds) 24th Asia-Pacific software engineering conference, APSEC 2017. IEEE Computer Society, Nanjing, pp 654–659

  • Spinellis D (2017) A repository of Unix history and evolution. Empir Softw Eng 22(3):1372–1404

    Article  Google Scholar 

  • Squire M (2017) The lives and deaths of open source code forges. In: Morgan L (ed) Proceedings of the 13th international symposium on open collaboration, OpenSym Galway, Ireland, August 23-25, 2017, pp 15:1–15:8. ACM

  • Stol K-J, Fitzgerald B (2014) Inner source–adopting open source development practices in organizations: a tutorial. IEEE Softw 32(4):60–67

    Article  Google Scholar 

  • Storey M-AD, Adams B, Haiduc S (eds) (2019) Proceedings of the 16th international conference on mining software repositories, MSR 2019, 26-27. IEEE / ACM, Montreal

  • Svajlenko J, Roy CK (2017) Fast and flexible large-scale clone detection with cloneworks. In: Uchitel S, Orso A, Robillard MP (eds) Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, Argentina, May 20-28, 2017 - companion Volume, pp 27–30. IEEE Computer Society

  • Thummalapenta S, Cerulo L, Aversano L, Di Penta M (2010) An empirical study on the maintenance of source code clones. Empir Softw Eng 15(1):1–34

    Article  Google Scholar 

  • Thung F, Bissyande TF, Lo D, Jiang L (2013) Network structure of social coding in github. In: 2013 17th European Conference on Software Maintenance and Reengineering, pp 323–326. IEEE

  • Tiwari NM, Upadhyaya G, Rajan H (2016) Candoia: A platform and ecosystem for mining software repositories tools. In: Dillon LK, Visser W, Williams L (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, pp 759–764. ACM

  • Tuunanen T, Koskinen Ji, Kärkkäinen T (2009) Automated software license analysis. Autom Softw Eng 16(3-4):455–490

    Article  Google Scholar 

  • Vendome C. (2015) A large scale study of license usage on github. In: 2015 IEEE/ACM 37th IEEE international conference on software engineering, vol 2, pp 772–774

  • Waldin R, Zhang J (2009) Determining a document similarity metric, July 2009. CIB: G06F17/30

  • Wu Y, Manabe Y, Kanda T, Germȧn DM, Inoue K (2017) Analysis of license inconsistency in large collections of open source projects. Empir Softw Eng 22 (3):1194–1222

    Article  Google Scholar 

  • Zaidman A, Kamei Y, Hill E (eds) (2018) Proceedings of the 15th International Conference on Mining Software Repositories, MSR 2018. ACM, Gothenburg

  • Zimmermann T, Premraj R, Zeller A (2007) Predicting defects for eclipse. In: International workshop on predictor models in software engineering, 2007 PROMISE’07: ICSE Workshops 2007, pp 9–9

  • Zimmermann T, Weißgerber P, Diehl S, Zeller A (2004) Mining version histories to guide software changes. In: Finkelstein A, Estublier J, Rosenblum DS (eds) 26th international conference on software engineering (ICSE 2004), 23-28 May 2004, Edinburgh, pp 563–572

Download references

Acknowledgments

The authors would like to thank the anonymous reviewers for precious feedback that allowed us to significantly improve this article.

Author information

Authors and Affiliations

  1. Université de Paris, Paris, France

    Guillaume Rousseau

  2. Inria and Université de Paris, Paris, France

    Roberto Di Cosmo

  3. Université de Paris and Inria, Paris, France

    Stefano Zacchiroli

Authors
  1. Guillaume Rousseau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roberto Di Cosmo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefano Zacchiroli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guillaume Rousseau.

Additional information

Communicated by: Miryung Kim

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rousseau, G., Di Cosmo, R. & Zacchiroli, S. Software provenance tracking at the scale of public source code. Empir Software Eng 25, 2930–2959 (2020). https://doi.org/10.1007/s10664-020-09828-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-020-09828-5

Keywords

Search

Navigation