Usage and attribution of Stack Overflow code snippets in GitHub projects

Abstract

Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Using those snippets raises maintenance and legal issues. SO’s license (CC BY-SA 3.0) requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO’s license model for code snippets and the required attribution, little is known about the extent to which snippets are copied from SO without proper attribution. We present results of a large-scale empirical study analyzing the usage and attribution of non-trivial Java code snippets from SO answers in public GitHub (GH) projects. We followed three different approaches to triangulate an estimate for the ratio of unattributed usages and conducted two online surveys with software developers to complement our results. For the different sets of projects that we analyzed, the ratio of projects containing files with a reference to SO varied between 3.3% and 11.9%. We found that at most 1.8% of all analyzed repositories containing code from SO used the code in a way compatible with CC BY-SA 3.0. Moreover, we estimate that at most a quarter of the copied code snippets from SO are attributed as required. Of the surveyed developers, almost one half admitted copying code from SO without attribution and about two thirds were not aware of the license of SO code snippets and its implications.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    https://stackoverflow.com/help/licensing

  2. 2.

    http://web.archive.org/

  3. 3.

    http://balusc.omnifaces.org/2007/07/fileservlet.html

References

  1. Abdalkareem R, Shihab E, Rilling J (2017) On code reuse from StackOverflow: An exploratory study on Android apps. Inf Softw Technol 88:148–158

    Article  Google Scholar 

  2. Acar Y, Backes M, Fahl S, Kim D, Mazurek ML, Stransky C (2016) You get where you’re looking for: The impact of information sources on code security. In: Locasto M, Shmatikov V, Erlingsson Ú (eds) 2016 IEEE Symposium on Security and Privacy (S&P 2016), IEEE Computer Society, San Jose, CA, USA, pp 289–305

  3. Achte Z (2016) AZ I-8 O 294/15. Landgericht Bochum http://www.justiz.nrw.de/nrwe/lgs/bochum/lg_bochum/j2016/I_8_O_294_15_Urteil_20160303.html

  4. Agresti A (2007) An introduction to categorical data analysis, 2nd edn. Wiley, Hoboken

    Google Scholar 

  5. Aioobe (2010) How to convert byte size into human readable format in java? http://stackoverflow.com/a/3758880

  6. Allamanis M, Sutton C (2013) Why, when, and what: Analyzing Stack Overflow questions by topic, type, and code. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 53–56

  7. Almeida DA, Murphy GC, Wilson G, Hoye M (2018) Investigating whether and how software developers understand open source software licensing. Empir Softw Eng 11(11):730

    Google Scholar 

  8. Alsup W (2012) Oracle America, Inc v. Google, Inc. United States District Court for the Northern District of California

  9. An L, Mlouki O, Khomh F, Antoniol G (2017) Stack Overflow: A Code Laundering Platform? In: Pinzger M, Bavota G, Marcus A (eds) 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2017) IEEE Computer Society, Klagenfurt, Austria, pp 283–293

  10. Badashian AS, Esteki A, Gholipour A, Hindle A, Stroulia E (2014) Involvement, Contribution and Influence in GitHub and Stack Overflow. In: Ng J, Li J, Wong K (eds) 24th international conference on computer science and software engineering (CASCON 2014), IBM / ACM, Markham, ON, Canada, pp 19–33

  11. Baltes S (2017). sbaltes/api-retriever on GitHub. https://doi.org/10.5281/zenodo.1049419

  12. Baltes S (2018) Usage and Attribution of Stack Overflow Code Snippets in GitHub Projects — Supplementary Material. https://doi.org/10.5281/zenodo.1148069

  13. Baltes S, Dumani L (2018) SOTorrent Dataset. https://doi.org/10.5281/zenodo.1135262

  14. Baltes S, Dumani L, Treude C, Diehl S (2018) SOTOrrent: Reconstructing and analyzing the evolution stack overflow posts. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on mining software repositories (MSR 2018) ACM, Gothenburg, Sweden, pp 1–12

  15. Bartlett JEII, Kotrlik JW, Higgins CC (2001) Organizational research: Determining appropriate sample size in survey research. Information technology Learning, and Performance Journal 19(1):43–50

    Google Scholar 

  16. Bosu A, Corley CS, Heaton D, Chatterji D, Carver JC, Kraft NA (2013) Building reputation in StackOverflow: An empirical investigation. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 89–92

  17. Brandt J, Dontcheva M, Weskamp M, Klemmer SR (2010) Example-centric programming: Integrating web search into the development environment. In: Mynatt E, Edwards K, Rodden T (eds) 2010 Conference on human factors in computing systems (CHI 2010), ACM, Atlanta, GA, USA, pp 513–522

  18. Burrows S, Tahaghoghi SMM, Zobel J (2007) Efficient plagiarism detection for large code repositories. Software—Practice and Experience 37(2):151–176

    Article  Google Scholar 

  19. Campbell BA, Treude C (2017) NLP2Code: Code snippet content assist via natural language tasks. In: Mei H, Zhang L, Zimmermann T (eds) 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME 2017) IEEE Computer Society, Shanghai, China, pp 628–632

  20. Cavaretta MJ (2015) Open source issues in mergers & acquisitions. http://www.mbbp.com/news/open-source-issues

  21. Cochran WG (1977) Sampling techniques, 3rd edn. Wiley, Hoboken

    Google Scholar 

  22. Corley JS (2017) Artifex Software Inc v. Hancom, Inc. United States District Court for the Northern District of California

  23. Creative Commons Corporation (2007) Attribution-ShareAlike 3.0 Unported. https://creativecommons.org/licenses/by-sa/3.0/legalcode

  24. Creative Commons Corporation (2017a) Compatible Licenses. https://creativecommons.org/share-your-work/licensing-considerations/compatible-licenses/

  25. Creative Commons Corporation (2017b) Frequently Asked Questions. https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software

  26. Duijn M, Kucera A, Bacchelli A (2015) Quality questions need quality code: Classifying code fragments on stack overflow. In: Di Penta M, Pinzger M, Robbes R (eds) 12th working conference on mining software repositories (MSR 2015) IEEE Computer Society, Florence, Italy, pp 410–413

  27. Electronic Frontier Foundation (2018) Oracle v. Google. https://www.eff.org/cases/oracle-v-google

  28. Engelfriet A (2016) What is the license status of StackOverflow code snippets? https://legalict.com/software/what-is-the-license-status-of-stackoverflow-code-snippets/

  29. Fazunenko D (2016) Get rid of the humanReadableByteCount() method in openjdk/hotspot. https://bugs.openjdk.java.net/browse/JDK-8170860

  30. Fischer F, Böttinger K, Xiao H, Stransky C, Acar Y, Backes M, Fahl S (2017) Stack overflow considered harmful? the impact of Copy&Paste on android application security. In: Butler KRB, Erlingsson Ú, Parno B (eds) 2017 IEEE Symposium on security and privacy (S&P 2017) IEEE Computer Society, San Jose, CA, USA, pp 121–136

  31. German DM, Hassan AE (2009) License integration patterns: Addressing license mismatches in component-based development. In: Fickas S, Atlee J M, Inverardi P (eds) 31St international conference on software engineering (ICSE 2009) IEEE Computer Society, Vancouver, BC, Canada, pp 188–198

  32. German DM, Di Penta M, Gueheneuc YG, Antoniol G (2009) Code siblings: Technical and legal implications of copying code between applications. In: Godfrey M W, Whitehead J (eds) 6th international working conference on mining software repositories (MSR 2009), IEEE Computer Society, Vancouver, BC, Canada, pp 81–90

  33. Gharehyazie M, Ray B, Filkov V (2017) Some From Here, Some From There: Cross-project Code Reuse in GitHub. In: Gonzalez-Barahona JM, Hindle A, Tan L (eds) 14th international conference on mining software repositories (MSR 2017), IEEE Computer Society, Buenos Aires, Argentina, pp 291–301

  34. GitHub Inc (2017a) Choosealicense.com: No License. https://choosealicense.com/no-license/

  35. GitHub Inc (2017b) GitHub Developer – API. https://developer.github.com/v3/

  36. GitHub Inc (2018) The State of the Octoverse 2017. https://octoverse.github.com/

  37. Google Cloud Platform (2017a) GitHub Data. https://cloud.google.com/bigquery/public-data/github

  38. Google Cloud Platform (2017b) Stack Overflow Data. https://cloud.google.com/bigquery/public-data/stackoverflow

  39. Gousios G (2013) The GHTorrent dataset and tool suite. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013), IEEE, San Francisco, CA, USA, pp 233–236

  40. Gousios G (2017) GHTorrent on the Google cloud. http://ghtorrent.org/gcloud.html

  41. Kaess J, Müller J, Rieger J (2004) Welte v. Sitecom Deutschland GmbH. District Court of Munich I

  42. Kalliamvakou E, Gousios G, Blincoe K, Singer L, Germán DM, Damian D (2014) The promises and perils of mining GitHub. In: Devanbu PT, Kim S, Pinzger M (eds) 11th working conference on mining software repositories (MSR 2014), ACM, Hyderabad, India, pp 92–101

  43. Lancaster T, Culwin F (2004) A comparison of source code plagiarism detection engines. Comput Sci Educ 14(2):101–112

    Article  Google Scholar 

  44. Lopes CV, Maj P, Martins P, Saini V, Di Yang, Zitny J, Sajnani H, Vitek J (2017) DéJàvu: A map of code duplicates on GitHub. Proc ACM Program Lang 1(OOPSLA):84:1–84:28

    Article  Google Scholar 

  45. Martins VT, Fonte D, Henriques PR, Dd Cruz (2014) Plagiarism detection: a tool survey and comparison. In: Pereira MJV, Leal JP, Simoes A (eds) 3rd symposium on languages, applications and technologies (SLATE 2014), Schloss Dagstuhl–Leibniz-Zentrum Fuer Informatik, Bragança, Portugal, Openaccess Series In Informatics (OASIcs), vol 38, pp 143–158

  46. Meloca R, Pinto G, Baiser L, Mattos M, Polato I, Wiese IS, German D (2018) Understanding the usage, impact, and adoption of non-OSI approved licenses. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on mining software repositories (MSR 2018) ACM, Gothenburg, Sweden, pp 1–11

  47. Morrison P, Murphy-Hill E (2013) Is programming knowledge related to age? An exploration of Stack Overflow. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 69–72

  48. Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating GitHub for engineered software projects. Empir Softw Eng 22(6):3219–3253

    Article  Google Scholar 

  49. Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example? A study of programming Q&A in StackOverflow. In: Tonella P, Di Penta M, Maletic JI (eds) 28th IEEE International Conference on Software Maintenance (ICSM 2012) IEEE Computer Society, Trento, Italy, pp 25–34

  50. Nederhof AJ (1985) Methods of coping with social desirability bias: a review. Eur J Soc Psychol 15(3):263–280

    Article  Google Scholar 

  51. PMD (2016) Finding duplicated code. http://pmd.github.io/pmd-5.5.1/usage/cpd-usage.html

  52. Ponzanelli L, Bacchelli A, Lanza M (2013) Seahawk: Stack overflow in the IDEa. In: Notkin D, Cheng BHC, Pohl K (eds) 35th international conference on software engineering (ICSE 2013) IEEE Computer Society, San Francisco, CA, USA, pp 1295–1298

  53. Ponzanelli L, Mocci A, Bacchelli A, Lanza M (2014) Understanding and classifying the quality of technical forum questions. In: Wong WE, McMillin B (eds) 14th international conference on quality software (QSIC 2014) IEEE, Allen, TX, USA, pp 343–352

  54. Poteat H (2016) GitHub’s 2015 Transparency Report. https://github.com/blog/2202-github-s-2015-transparency-report

  55. Prechelt L, Malpohl G, Philippsen M (2002) Finding plagiarisms among a set of programs with JPlag. Journal of Universal Computer Science 8(11):1016–1038

    Google Scholar 

  56. Ragkhitwetsagul C (2016) Measuring code similarity in Large-Scaled code corpora. In: Kraft NA, Menzies T, Adams B, Poshyvanyk D (eds) 2016 IEEE International conference on software maintenance and evolution (ICSME 2016) IEEE Computer Society, Raleigh, NC, USA, pp 626–630

  57. Roy CK, Cordy JR, Koschke R (2009) Comparison and evaluation of code clone detection techniques and tools: a qualitative approach. Sci Comput Program 74 (7):470–495

    MathSciNet  Article  MATH  Google Scholar 

  58. Sajnani H, Saini V, Svajlenko J, Roy CK, Lopes CV (2016) SourcererCC: Scaling code clone detection to big-code. In: Dillon L, Visser W, Williams L (eds) 38th international conference on software engineering (ICSE 2016), ACM, Austin, TX, USA, pp 1157–1168

  59. Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvany D, Oliveto R (2017) Automatically assessing code understandability: How far are we? In: Rosu G, Penta MD, Nguyen TN (eds) 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017) IEEE Computer Society, Urbana, IL, USA, pp 417–427

  60. Schenk D, Lungu M (2013) Geo-locating the knowledge transfer in StackOverflow. In: Ali R, Begel A, Maalej W (eds) 2013 International workshop on social software engineering (SSE 2013) ACM, Saint Petersburg, Russian Federation, pp 21–24

  61. Silvestri G, Yang J, Bozzon A, Tagarelli A (2015) Linking Accounts across Social Networks: The Case of StackOverflow, GitHub and Twitter. In: Armano G, Bozzon A, Giuliani A (eds) 1st International Workshop on Knowledge Discovery on the WEB (KDWeb 2015) CEUR-WS.org, Cagliari, Italy, CEUR Workshop Proceedings, pp 41–52

  62. Software Freedom Law Center (2008) Free Software Foundation Inc v. Cisco Systems, Inc. United States District Court for the Southern District of New York

  63. Sojer M, Henkel J (2011) License risks from ad hoc reuse of code from the internet. Commun ACM 54(12):74–81

    Article  Google Scholar 

  64. St Laurent AM (2004) Understanding Open Source and Free Software Licensing. O’Reilly Media, Sebastopol

    Google Scholar 

  65. Stack Exchange Inc (2015) Stack Exchange Data Dump: August 18, 2015. https://archive.org/details/stackexchange/

  66. Stack Exchange Inc (2016) Stack Exchange API v2.2. https://api.stackexchange.com/docs

  67. Stack Exchange Inc (2017a) Stack Exchange Data Dump 2017-12-01. https://archive.org/details/stackexchange/

  68. Stack Exchange Inc (2017b) Stack Exchange Data Dump: March 14, 2017. https://archive.org/details/stackexchange/

  69. Stack Exchange Inc (2018a) Stack exchange network terms of service. https://web.archive.org/web/20180228075555/http://stackexchange.com/legal

  70. Stack Exchange Inc (2018b) Stack exchange network terms of service. http://stackexchange.com/legal

  71. Stack Exchange Meta (2009) What is up with the source code license on Stack Overflow? http://meta.stackexchange.com/q/25956

  72. Stack Exchange Meta (2013) Do I have to worry about copyright issues for code posted on Stack Overflow? http://meta.stackexchange.com/q/12527

  73. Stack Exchange Meta (2015) Can we get some explicit clarification on the *intended* legal usage of code from SO answers? http://meta.stackoverflow.com/q/286582

  74. Stack Exchange Meta (2016) A new code license: The MIT, this time with attribution required. http://meta.stackexchange.com/q/272956

  75. Subramanian S, Holmes R (2013) Making sense of online code snippets. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 85–88

  76. Tim Post (2018) A new (2018) update to our terms of service is here. https://meta.stackexchange.com/questions/309746/a-new-2018-update-to-our-terms-of-service-is-here

  77. TIOBE software BV (2017) TIOBE Index for February 2017. http://www.tiobe.com/tiobe-index/

  78. Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: Dillon L, Visser W, Williams L (eds) 38th international conference on software engineering (ICSE 2016), ACM, Austin, TX, USA, pp 392–403

  79. Treude C, Robillard MP (2017) Understanding stack overflow code fragments. In: Mei H, Zhang L, Zimmermann T (eds) 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME 2017) IEEE Computer Society, Shanghai, China, pp 509–513

  80. Treude C, Barzilay O, Storey MAD (2011) How do programmers ask and answer questions on the web? In: Taylor RN, Gall HC, Medvidovic N (eds) 33rd international conference on software engineering (ICSE 2011) ACM, Waikiki, Honolulu, pp 804–807

  81. Vasilescu B, Capiluppi A, Serebrenik A (2012) Gender, representation and online participation: A quantitative study of stackoverflow. In: Aberer K, Flache A, Jager W, Liu L, Tang J, Gueret C (eds) 4th International Conference on Social Informatics (SocInfo 2012) Springer, Lausanne, Switzerland, Lecture Notes in Computer Science, pp 332–338

  82. Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge. In: Chang LW, Srivastava J, Zhan J (eds) 2013 International conference on social computing (SocialCom 2013) IEEE Computer Society, Washington, DC, USA, pp 188–195

  83. Vendome C (2015) A large scale study of license usage on GitHub. In: Bertolino A, Canfora G, Elbaum S (eds) 37th international conference on software engineering (ICSE 2015) IEEE, Florence, Italy, pp 772–774

  84. Wang S, Lo D, Jiang L (2013) An empirical study on developer interactions in StackOverflow. In: Shin SY, Maldonado JC (eds) 28th annual ACM symposium on applied computing (SAC 2013) ACM, Coimbra, Portugal, pp 1019–1024

  85. White JS (2008) Jacobsen v. Katzer, 535 F.3d 1373, 1379. United States Court of Appeals for the Federal Circuit

  86. Wikipedia (2017) Free Software Foundation, Inc v. Cisco Systems, Inc. https://en.wikipedia.org/wiki/Free_Software_Foundation,_Inc._v._Cisco_Systems,_Inc.

  87. Xia X, Bao L, Lo D, Kochhar PS, Hassan AE, Xing Z (2017) What do developers search for on the web? Empir Softw Eng 22(6):3149–3185

    Article  Google Scholar 

  88. Yang D, Hussain A, Lopes CV (2016) From query to usable code: an analysis of stack overflow code snippets. In: Kim M, Robbes R, Bird C (eds) 13th international conference on mining software repositories (MSR 2016), ACM, Austin, TX, USA, pp 391–402

  89. Yang D, Martins P, Saini V, Lopes CV (2017) Stack Overflow in GitHub: Any snippets there? In: Gonzalez-Barahona JM, Hindle A, Tan L (eds) 14th international conference on mining software repositories (MSR 2017) IEEE Computer Society, Buenos Aires, Argentina, pp 280–290

  90. Yang J, Hauff C, Bozzon A, Houben GJ (2014) Asking the right question in collaborative Q&A systems. In: Ferres L, Rossi G, Almeida VAF, Herder E (eds) 25th ACM conference on hypertext and social media (HT 2014) ACM, Santiago, Chile, pp 179–189

  91. Zagalsky A, Barzilay O, Yehudai A (2012) Example Overflow: Using social media for code recommendation. In: Maalej W, Robillard MP, Walker RJ, Zimmermann T (eds) 3rd international workshop on recommendation systems for software engineering (RSSE 2012) IEEE, Zurich, Switzerland, pp 38–42

  92. Zhang T, Upadhyaya G, Reinhardt A, Rajan H, Kim M (2018) Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow. In: Crnkovic I, Chechik M, Harman M (eds) 40th international conference on software engineering (ICSE 2018) ACM, Gothenburg, Sweden, pp 1–11

Download references

Acknowledgements

The authors would like to thank the participants of the online surveys, the anonymous reviewers, and Bernhard Baltes-Götz for their valuable feedback. Moreover, we thank Richard Kiefer for his help with the calibration of CPD and the extraction of the snippet sets and Florian Reitz for his help with database-related issues.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Sebastian Baltes.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by: Ahmed E. Hassan

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Baltes, S., Diehl, S. Usage and attribution of Stack Overflow code snippets in GitHub projects. Empir Software Eng 24, 1259–1295 (2019). https://doi.org/10.1007/s10664-018-9650-5

Download citation

Keywords

  • Code snippets
  • Licensing
  • Stack Overflow
  • GitHub
  • Online survey
  • Mining software repositories