Advertisement

Usage and attribution of Stack Overflow code snippets in GitHub projects

  • Sebastian Baltes
  • Stephan Diehl
Article

Abstract

Stack Overflow (SO) is the most popular question-and-answer website for software developers, providing a large amount of copyable code snippets. Using those snippets raises maintenance and legal issues. SO’s license (CC BY-SA 3.0) requires attribution, i.e., referencing the original question or answer, and requires derived work to adopt a compatible license. While there is a heated debate on SO’s license model for code snippets and the required attribution, little is known about the extent to which snippets are copied from SO without proper attribution. We present results of a large-scale empirical study analyzing the usage and attribution of non-trivial Java code snippets from SO answers in public GitHub (GH) projects. We followed three different approaches to triangulate an estimate for the ratio of unattributed usages and conducted two online surveys with software developers to complement our results. For the different sets of projects that we analyzed, the ratio of projects containing files with a reference to SO varied between 3.3% and 11.9%. We found that at most 1.8% of all analyzed repositories containing code from SO used the code in a way compatible with CC BY-SA 3.0. Moreover, we estimate that at most a quarter of the copied code snippets from SO are attributed as required. Of the surveyed developers, almost one half admitted copying code from SO without attribution and about two thirds were not aware of the license of SO code snippets and its implications.

Keywords

Code snippets Licensing Stack Overflow GitHub Online survey Mining software repositories 

Notes

Acknowledgements

The authors would like to thank the participants of the online surveys, the anonymous reviewers, and Bernhard Baltes-Götz for their valuable feedback. Moreover, we thank Richard Kiefer for his help with the calibration of CPD and the extraction of the snippet sets and Florian Reitz for his help with database-related issues.

References

  1. Abdalkareem R, Shihab E, Rilling J (2017) On code reuse from StackOverflow: An exploratory study on Android apps. Inf Softw Technol 88:148–158CrossRefGoogle Scholar
  2. Acar Y, Backes M, Fahl S, Kim D, Mazurek ML, Stransky C (2016) You get where you’re looking for: The impact of information sources on code security. In: Locasto M, Shmatikov V, Erlingsson Ú (eds) 2016 IEEE Symposium on Security and Privacy (S&P 2016), IEEE Computer Society, San Jose, CA, USA, pp 289–305Google Scholar
  3. Agresti A (2007) An introduction to categorical data analysis, 2nd edn. Wiley, HobokenCrossRefGoogle Scholar
  4. Aioobe (2010) How to convert byte size into human readable format in java? http://stackoverflow.com/a/3758880
  5. Allamanis M, Sutton C (2013) Why, when, and what: Analyzing Stack Overflow questions by topic, type, and code. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 53–56Google Scholar
  6. Almeida DA, Murphy GC, Wilson G, Hoye M (2018) Investigating whether and how software developers understand open source software licensing. Empir Softw Eng 11(11):730Google Scholar
  7. Alsup W (2012) Oracle America, Inc v. Google, Inc. United States District Court for the Northern District of CaliforniaGoogle Scholar
  8. An L, Mlouki O, Khomh F, Antoniol G (2017) Stack Overflow: A Code Laundering Platform? In: Pinzger M, Bavota G, Marcus A (eds) 24th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER 2017) IEEE Computer Society, Klagenfurt, Austria, pp 283–293Google Scholar
  9. Badashian AS, Esteki A, Gholipour A, Hindle A, Stroulia E (2014) Involvement, Contribution and Influence in GitHub and Stack Overflow. In: Ng J, Li J, Wong K (eds) 24th international conference on computer science and software engineering (CASCON 2014), IBM / ACM, Markham, ON, Canada, pp 19–33Google Scholar
  10. Baltes S (2017). sbaltes/api-retriever on GitHub.  https://doi.org/10.5281/zenodo.1049419
  11. Baltes S (2018) Usage and Attribution of Stack Overflow Code Snippets in GitHub Projects — Supplementary Material.  https://doi.org/10.5281/zenodo.1148069
  12. Baltes S, Dumani L (2018) SOTorrent Dataset.  https://doi.org/10.5281/zenodo.1135262
  13. Baltes S, Dumani L, Treude C, Diehl S (2018) SOTOrrent: Reconstructing and analyzing the evolution stack overflow posts. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on mining software repositories (MSR 2018) ACM, Gothenburg, Sweden, pp 1–12Google Scholar
  14. Bartlett JEII, Kotrlik JW, Higgins CC (2001) Organizational research: Determining appropriate sample size in survey research. Information technology Learning, and Performance Journal 19(1):43–50Google Scholar
  15. Bosu A, Corley CS, Heaton D, Chatterji D, Carver JC, Kraft NA (2013) Building reputation in StackOverflow: An empirical investigation. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 89–92Google Scholar
  16. Brandt J, Dontcheva M, Weskamp M, Klemmer SR (2010) Example-centric programming: Integrating web search into the development environment. In: Mynatt E, Edwards K, Rodden T (eds) 2010 Conference on human factors in computing systems (CHI 2010), ACM, Atlanta, GA, USA, pp 513–522Google Scholar
  17. Burrows S, Tahaghoghi SMM, Zobel J (2007) Efficient plagiarism detection for large code repositories. Software—Practice and Experience 37(2):151–176CrossRefGoogle Scholar
  18. Campbell BA, Treude C (2017) NLP2Code: Code snippet content assist via natural language tasks. In: Mei H, Zhang L, Zimmermann T (eds) 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME 2017) IEEE Computer Society, Shanghai, China, pp 628–632Google Scholar
  19. Cavaretta MJ (2015) Open source issues in mergers & acquisitions. http://www.mbbp.com/news/open-source-issues
  20. Cochran WG (1977) Sampling techniques, 3rd edn. Wiley, HobokenzbMATHGoogle Scholar
  21. Corley JS (2017) Artifex Software Inc v. Hancom, Inc. United States District Court for the Northern District of CaliforniaGoogle Scholar
  22. Creative Commons Corporation (2007) Attribution-ShareAlike 3.0 Unported. https://creativecommons.org/licenses/by-sa/3.0/legalcode
  23. Creative Commons Corporation (2017b) Frequently Asked Questions. https://creativecommons.org/faq/#can-i-apply-a-creative-commons-license-to-software
  24. Duijn M, Kucera A, Bacchelli A (2015) Quality questions need quality code: Classifying code fragments on stack overflow. In: Di Penta M, Pinzger M, Robbes R (eds) 12th working conference on mining software repositories (MSR 2015) IEEE Computer Society, Florence, Italy, pp 410–413Google Scholar
  25. Electronic Frontier Foundation (2018) Oracle v. Google. https://www.eff.org/cases/oracle-v-google
  26. Engelfriet A (2016) What is the license status of StackOverflow code snippets? https://legalict.com/software/what-is-the-license-status-of-stackoverflow-code-snippets/
  27. Fazunenko D (2016) Get rid of the humanReadableByteCount() method in openjdk/hotspot. https://bugs.openjdk.java.net/browse/JDK-8170860
  28. Fischer F, Böttinger K, Xiao H, Stransky C, Acar Y, Backes M, Fahl S (2017) Stack overflow considered harmful? the impact of Copy&Paste on android application security. In: Butler KRB, Erlingsson Ú, Parno B (eds) 2017 IEEE Symposium on security and privacy (S&P 2017) IEEE Computer Society, San Jose, CA, USA, pp 121–136Google Scholar
  29. German DM, Hassan AE (2009) License integration patterns: Addressing license mismatches in component-based development. In: Fickas S, Atlee J M, Inverardi P (eds) 31St international conference on software engineering (ICSE 2009) IEEE Computer Society, Vancouver, BC, Canada, pp 188–198Google Scholar
  30. German DM, Di Penta M, Gueheneuc YG, Antoniol G (2009) Code siblings: Technical and legal implications of copying code between applications. In: Godfrey M W, Whitehead J (eds) 6th international working conference on mining software repositories (MSR 2009), IEEE Computer Society, Vancouver, BC, Canada, pp 81–90Google Scholar
  31. Gharehyazie M, Ray B, Filkov V (2017) Some From Here, Some From There: Cross-project Code Reuse in GitHub. In: Gonzalez-Barahona JM, Hindle A, Tan L (eds) 14th international conference on mining software repositories (MSR 2017), IEEE Computer Society, Buenos Aires, Argentina, pp 291–301Google Scholar
  32. GitHub Inc (2017a) Choosealicense.com: No License. https://choosealicense.com/no-license/
  33. GitHub Inc (2017b) GitHub Developer – API. https://developer.github.com/v3/
  34. GitHub Inc (2018) The State of the Octoverse 2017. https://octoverse.github.com/
  35. Google Cloud Platform (2017a) GitHub Data. https://cloud.google.com/bigquery/public-data/github
  36. Google Cloud Platform (2017b) Stack Overflow Data. https://cloud.google.com/bigquery/public-data/stackoverflow
  37. Gousios G (2013) The GHTorrent dataset and tool suite. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013), IEEE, San Francisco, CA, USA, pp 233–236Google Scholar
  38. Gousios G (2017) GHTorrent on the Google cloud. http://ghtorrent.org/gcloud.html
  39. Kaess J, Müller J, Rieger J (2004) Welte v. Sitecom Deutschland GmbH. District Court of Munich IGoogle Scholar
  40. Kalliamvakou E, Gousios G, Blincoe K, Singer L, Germán DM, Damian D (2014) The promises and perils of mining GitHub. In: Devanbu PT, Kim S, Pinzger M (eds) 11th working conference on mining software repositories (MSR 2014), ACM, Hyderabad, India, pp 92–101Google Scholar
  41. Lancaster T, Culwin F (2004) A comparison of source code plagiarism detection engines. Comput Sci Educ 14(2):101–112CrossRefGoogle Scholar
  42. Lopes CV, Maj P, Martins P, Saini V, Di Yang, Zitny J, Sajnani H, Vitek J (2017) DéJàvu: A map of code duplicates on GitHub. Proc ACM Program Lang 1(OOPSLA):84:1–84:28CrossRefGoogle Scholar
  43. Martins VT, Fonte D, Henriques PR, Dd Cruz (2014) Plagiarism detection: a tool survey and comparison. In: Pereira MJV, Leal JP, Simoes A (eds) 3rd symposium on languages, applications and technologies (SLATE 2014), Schloss Dagstuhl–Leibniz-Zentrum Fuer Informatik, Bragança, Portugal, Openaccess Series In Informatics (OASIcs), vol 38, pp 143–158Google Scholar
  44. Meloca R, Pinto G, Baiser L, Mattos M, Polato I, Wiese IS, German D (2018) Understanding the usage, impact, and adoption of non-OSI approved licenses. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on mining software repositories (MSR 2018) ACM, Gothenburg, Sweden, pp 1–11Google Scholar
  45. Morrison P, Murphy-Hill E (2013) Is programming knowledge related to age? An exploration of Stack Overflow. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 69–72Google Scholar
  46. Munaiah N, Kroh S, Cabrey C, Nagappan M (2017) Curating GitHub for engineered software projects. Empir Softw Eng 22(6):3219–3253CrossRefGoogle Scholar
  47. Nasehi SM, Sillito J, Maurer F, Burns C (2012) What makes a good code example? A study of programming Q&A in StackOverflow. In: Tonella P, Di Penta M, Maletic JI (eds) 28th IEEE International Conference on Software Maintenance (ICSM 2012) IEEE Computer Society, Trento, Italy, pp 25–34Google Scholar
  48. Nederhof AJ (1985) Methods of coping with social desirability bias: a review. Eur J Soc Psychol 15(3):263–280CrossRefGoogle Scholar
  49. PMD (2016) Finding duplicated code. http://pmd.github.io/pmd-5.5.1/usage/cpd-usage.html
  50. Ponzanelli L, Bacchelli A, Lanza M (2013) Seahawk: Stack overflow in the IDEa. In: Notkin D, Cheng BHC, Pohl K (eds) 35th international conference on software engineering (ICSE 2013) IEEE Computer Society, San Francisco, CA, USA, pp 1295–1298Google Scholar
  51. Ponzanelli L, Mocci A, Bacchelli A, Lanza M (2014) Understanding and classifying the quality of technical forum questions. In: Wong WE, McMillin B (eds) 14th international conference on quality software (QSIC 2014) IEEE, Allen, TX, USA, pp 343–352Google Scholar
  52. Poteat H (2016) GitHub’s 2015 Transparency Report. https://github.com/blog/2202-github-s-2015-transparency-report
  53. Prechelt L, Malpohl G, Philippsen M (2002) Finding plagiarisms among a set of programs with JPlag. Journal of Universal Computer Science 8(11):1016–1038Google Scholar
  54. Ragkhitwetsagul C (2016) Measuring code similarity in Large-Scaled code corpora. In: Kraft NA, Menzies T, Adams B, Poshyvanyk D (eds) 2016 IEEE International conference on software maintenance and evolution (ICSME 2016) IEEE Computer Society, Raleigh, NC, USA, pp 626–630Google Scholar
  55. Roy CK, Cordy JR, Koschke R (2009) Comparison and evaluation of code clone detection techniques and tools: a qualitative approach. Sci Comput Program 74 (7):470–495MathSciNetCrossRefGoogle Scholar
  56. Sajnani H, Saini V, Svajlenko J, Roy CK, Lopes CV (2016) SourcererCC: Scaling code clone detection to big-code. In: Dillon L, Visser W, Williams L (eds) 38th international conference on software engineering (ICSE 2016), ACM, Austin, TX, USA, pp 1157–1168Google Scholar
  57. Scalabrino S, Bavota G, Vendome C, Linares-Vásquez M, Poshyvany D, Oliveto R (2017) Automatically assessing code understandability: How far are we? In: Rosu G, Penta MD, Nguyen TN (eds) 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2017) IEEE Computer Society, Urbana, IL, USA, pp 417–427Google Scholar
  58. Schenk D, Lungu M (2013) Geo-locating the knowledge transfer in StackOverflow. In: Ali R, Begel A, Maalej W (eds) 2013 International workshop on social software engineering (SSE 2013) ACM, Saint Petersburg, Russian Federation, pp 21–24Google Scholar
  59. Silvestri G, Yang J, Bozzon A, Tagarelli A (2015) Linking Accounts across Social Networks: The Case of StackOverflow, GitHub and Twitter. In: Armano G, Bozzon A, Giuliani A (eds) 1st International Workshop on Knowledge Discovery on the WEB (KDWeb 2015) CEUR-WS.org, Cagliari, Italy, CEUR Workshop Proceedings, pp 41–52Google Scholar
  60. Software Freedom Law Center (2008) Free Software Foundation Inc v. Cisco Systems, Inc. United States District Court for the Southern District of New YorkGoogle Scholar
  61. Sojer M, Henkel J (2011) License risks from ad hoc reuse of code from the internet. Commun ACM 54(12):74–81CrossRefGoogle Scholar
  62. St Laurent AM (2004) Understanding Open Source and Free Software Licensing. O’Reilly Media, SebastopolGoogle Scholar
  63. Stack Exchange Inc (2015) Stack Exchange Data Dump: August 18, 2015. https://archive.org/details/stackexchange/
  64. Stack Exchange Inc (2016) Stack Exchange API v2.2. https://api.stackexchange.com/docs
  65. Stack Exchange Inc (2017a) Stack Exchange Data Dump 2017-12-01. https://archive.org/details/stackexchange/
  66. Stack Exchange Inc (2017b) Stack Exchange Data Dump: March 14, 2017. https://archive.org/details/stackexchange/
  67. Stack Exchange Inc (2018a) Stack exchange network terms of service. https://web.archive.org/web/20180228075555/http://stackexchange.com/legal
  68. Stack Exchange Inc (2018b) Stack exchange network terms of service. http://stackexchange.com/legal
  69. Stack Exchange Meta (2009) What is up with the source code license on Stack Overflow? http://meta.stackexchange.com/q/25956
  70. Stack Exchange Meta (2013) Do I have to worry about copyright issues for code posted on Stack Overflow? http://meta.stackexchange.com/q/12527
  71. Stack Exchange Meta (2015) Can we get some explicit clarification on the *intended* legal usage of code from SO answers? http://meta.stackoverflow.com/q/286582
  72. Stack Exchange Meta (2016) A new code license: The MIT, this time with attribution required. http://meta.stackexchange.com/q/272956
  73. Subramanian S, Holmes R (2013) Making sense of online code snippets. In: Zimmermann T, Di Penta M, Kim S (eds) 10th international working conference on mining software repositories (MSR 2013) IEEE, San Francisco, CA, USA, pp 85–88Google Scholar
  74. Tim Post (2018) A new (2018) update to our terms of service is here. https://meta.stackexchange.com/questions/309746/a-new-2018-update-to-our-terms-of-service-is-here
  75. TIOBE software BV (2017) TIOBE Index for February 2017. http://www.tiobe.com/tiobe-index/
  76. Treude C, Robillard MP (2016) Augmenting API documentation with insights from stack overflow. In: Dillon L, Visser W, Williams L (eds) 38th international conference on software engineering (ICSE 2016), ACM, Austin, TX, USA, pp 392–403Google Scholar
  77. Treude C, Robillard MP (2017) Understanding stack overflow code fragments. In: Mei H, Zhang L, Zimmermann T (eds) 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME 2017) IEEE Computer Society, Shanghai, China, pp 509–513Google Scholar
  78. Treude C, Barzilay O, Storey MAD (2011) How do programmers ask and answer questions on the web? In: Taylor RN, Gall HC, Medvidovic N (eds) 33rd international conference on software engineering (ICSE 2011) ACM, Waikiki, Honolulu, pp 804–807Google Scholar
  79. Vasilescu B, Capiluppi A, Serebrenik A (2012) Gender, representation and online participation: A quantitative study of stackoverflow. In: Aberer K, Flache A, Jager W, Liu L, Tang J, Gueret C (eds) 4th International Conference on Social Informatics (SocInfo 2012) Springer, Lausanne, Switzerland, Lecture Notes in Computer Science, pp 332–338Google Scholar
  80. Vasilescu B, Filkov V, Serebrenik A (2013) StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge. In: Chang LW, Srivastava J, Zhan J (eds) 2013 International conference on social computing (SocialCom 2013) IEEE Computer Society, Washington, DC, USA, pp 188–195Google Scholar
  81. Vendome C (2015) A large scale study of license usage on GitHub. In: Bertolino A, Canfora G, Elbaum S (eds) 37th international conference on software engineering (ICSE 2015) IEEE, Florence, Italy, pp 772–774Google Scholar
  82. Wang S, Lo D, Jiang L (2013) An empirical study on developer interactions in StackOverflow. In: Shin SY, Maldonado JC (eds) 28th annual ACM symposium on applied computing (SAC 2013) ACM, Coimbra, Portugal, pp 1019–1024Google Scholar
  83. White JS (2008) Jacobsen v. Katzer, 535 F.3d 1373, 1379. United States Court of Appeals for the Federal CircuitGoogle Scholar
  84. Wikipedia (2017) Free Software Foundation, Inc v. Cisco Systems, Inc. https://en.wikipedia.org/wiki/Free_Software_Foundation,_Inc._v._Cisco_Systems,_Inc.
  85. Xia X, Bao L, Lo D, Kochhar PS, Hassan AE, Xing Z (2017) What do developers search for on the web? Empir Softw Eng 22(6):3149–3185CrossRefGoogle Scholar
  86. Yang D, Hussain A, Lopes CV (2016) From query to usable code: an analysis of stack overflow code snippets. In: Kim M, Robbes R, Bird C (eds) 13th international conference on mining software repositories (MSR 2016), ACM, Austin, TX, USA, pp 391–402Google Scholar
  87. Yang D, Martins P, Saini V, Lopes CV (2017) Stack Overflow in GitHub: Any snippets there? In: Gonzalez-Barahona JM, Hindle A, Tan L (eds) 14th international conference on mining software repositories (MSR 2017) IEEE Computer Society, Buenos Aires, Argentina, pp 280–290Google Scholar
  88. Yang J, Hauff C, Bozzon A, Houben GJ (2014) Asking the right question in collaborative Q&A systems. In: Ferres L, Rossi G, Almeida VAF, Herder E (eds) 25th ACM conference on hypertext and social media (HT 2014) ACM, Santiago, Chile, pp 179–189Google Scholar
  89. Zagalsky A, Barzilay O, Yehudai A (2012) Example Overflow: Using social media for code recommendation. In: Maalej W, Robillard MP, Walker RJ, Zimmermann T (eds) 3rd international workshop on recommendation systems for software engineering (RSSE 2012) IEEE, Zurich, Switzerland, pp 38–42Google Scholar
  90. Zhang T, Upadhyaya G, Reinhardt A, Rajan H, Kim M (2018) Are Code Examples on an Online Q&A Forum Reliable? A Study of API Misuse on Stack Overflow. In: Crnkovic I, Chechik M, Harman M (eds) 40th international conference on software engineering (ICSE 2018) ACM, Gothenburg, Sweden, pp 1–11Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of TrierTrierGermany

Personalised recommendations