Studying backers and hunters in bounty issue addressing process of open source projects

Abstract

Issue addressing is a vital task in the evolution of software projects. However, in practice, not all issues can be addressed on time. To facilitate the issue addressing process, monetary incentives (e.g., bounties) are used to attract developers to address issues. There are two types of core roles who are involved in this process: bounty backers, who propose bounties for an issue report via bounty platforms (e.g., Bountysource), and bounty hunters, who address the bounty issues and win the bounties. We wish to study the process of bounty issue addressing from the angle of two important roles (i.e., backers and hunters) and their related behaviors. With a better understanding of how they address bounty issues, stakeholders (e.g., operators and developers) of open source projects may have a reasonable estimation of what they can expect from backers and hunters. In this study, we investigate 2,955 bounty backers and 882 bounty hunters, and their associated 3,579 GitHub issue reports with 5,589 bounties that were proposed on Bountysource. We find that: 1) Overall, the value of a bounty is small (median bounty value of $20). Both individual and corporate backers prefer to support implementing new features rather than fixing bugs. Corporate backers tend to propose larger bounties and propose bounties more frequently than individual backers. 2) 85.0% of the bounty hunters addressed less than 3 bounty issues. The income of 56.7% of the bounty hunters is no more than $100 and only 2.7% of the hunters have earned more than $2,000. In addition, most of the regular hunters and big hunters are developers that made at least one commit before addressing a bounty issue. 3) The value of a bounty issue is not a statistically significant factor that attracts developers that have never made any commit before to address an issue. Based on our findings, we provide several suggestions for stakeholders of open source projects and hunters.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Notes

  1. 1.

    https://www.bountysource.com

  2. 2.

    https://github.com/austinpray/asset-builder/issues?q=label%3Abounty

  3. 3.

    https://www.google.com/about/appsecurity/chrome-rewards/

  4. 4.

    https://bountysource.github.io/

  5. 5.

    https://developer.github.com/v3/

  6. 6.

    https://github.com/SAILResearch/wip-18-jiayuan-bountysource-SupportMaterials

  7. 7.

    https://cran.r-project.org/web/packages/rms/rms.pdf

  8. 8.

    https://github.com/webmproject/libvpx/

  9. 9.

    https://medium.com/@luc.trudeau/video-compression-bounty-hunters-c8edf43d440

  10. 10.

    https://github.com/OpenRA

  11. 11.

    https://www.bountysource.com/issues/25924774-enable-implement-ppc64-le-linux-lj_gc64-interpreter-and-jit

  12. 12.

    https://www.bountysource.com/issues/5413688-repeat-every-x-days-monthlies

  13. 13.

    https://github.com/HabitRPG/habitica/issues/4173

References

  1. Akobeng AK (2007) Understanding diagnostic tests 3: receiver operating characteristic curves. Acta Paediat 96(5):644–647

    Article  Google Scholar 

  2. Androutsellis-Theotokis S, Spinellis D, Kechagia M, Gousios G, et al. (2011) Open source software: A survey from 10,000 feet. Found Trends Technol Inf Oper Manag 4(3–4):187–347

    Google Scholar 

  3. Apple Inc (2020) Apple Security Bounty). https://developer.apple.com/security-bounty/,. (last visited: Dec 12, 2020)

  4. Atiq A, Tripathi A (2016) Impact of financial benefits on open source software sustainability. In: International conference on information systems (ICIS), pp 1–10

  5. Avelino G, Passos L, Hora A, Valente MT (2016) A novel approach for estimating truck factors. In: IEEE 24th international conference on program comprehension (ICPC), pp 1–10

  6. Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13(1):281–305

    MathSciNet  MATH  Google Scholar 

  7. Bissyandé TF, Thung F, Lo D, Jiang L, Réveillère L (2013) Popularity, interoperability, and impact of programming languages in 100,000 open source projects. In: IEEE 37th annual computer software and applications conference. IEEE, pp 303–312

  8. Canfora G, Di Penta M, Oliveto R, Panichella S (2012) Who is going to mentor newcomers in open source projects?. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering (FSE), pp 1–11

  9. Coelho J, Valente MT, Silva LL, Hora A (2018) Why we engage in floss: Answers from core developers. In: Proceedings of the 11th international workshop on cooperative and human aspects of software engineering, pp 114–121

  10. Comino S, Manenti FM, Parisi ML (2007) From planning to mature: on the success of open source projects. Res Policy 36(10):1575–1586

    Article  Google Scholar 

  11. Dagenais B, Ossher H, Bellamy RKE, Robillard MP, de Vries JP (2010) Moving into a new software project landscape. In: Proceedings of the 32nd ACM/IEEE international conference on software engineering - Volume 1, ICSE ’10, pp 275–284

  12. Dinnie M (2019) How to prioritize feature requests for software development. https://zenkit.com/en/blog/how-to-prioritize-feature-requests-for-software-development,. (last visited: November 8, 2019)

  13. Duebendorfer T, Frei S (2009) Why silent updates boost security. TIK, ETH Zurich, Tech Rep 302

  14. Eghbal N (2016) Roads and bridges: The unseen labor behind our digital infrastructure. Ford Foundation

    Google Scholar 

  15. Eghbal N (2019) A handy guide to financial support for open source

  16. Finifter M, Akhawe D, Wagner D (2013) An empirical study of vulnerability rewards programs. In: USENIX Security Symp., pp 273–288

  17. Frei S, Duebendorfer T, Plattner B (2008) Firefox (in) security update dynamics exposed. ACM SIGCOMM Comput Commun Rev 39(1):16–22

    Article  Google Scholar 

  18. Frey BS, Goette L (1999) Does pay motivate volunteers? Working paper/Inst Empir Res Econ 7

  19. HackerOne (2018) 118 fascinating facts from hackerone’s hacker-powered security report 2018. https://www.hackerone.com/blog/118-Fascinating-Facts-HackerOnes-Hacker-Powered-Security-Report-2018. (last visited: August 27, 2018)

  20. Harhoff D, Henkel J, Von Hippel E (2003) Profiting from voluntary information spillovers: how users benefit by freely revealing their innovations. Res Pol 32(10):1753–1769

    Article  Google Scholar 

  21. Hata H, Guo M, Babar MA (2017) Understanding the heterogeneity of contributors in bug bounty programs. In: Proc. of the ACM/IEEE int’l symp. on empirical software engineering and measurement, pp 223–228

  22. Izquierdo JLC, Cabot J (2018) The role of foundations in open source projects. In: Proceedings of the 40th international conference on software engineering: software engineering in society, pp 3–12

  23. Kanda T, Guo M, Hata H, Matsumoto K (2017) Towards understanding an open-source bounty: Analysis of Bountysource. In: Int’l conf. on software analysis, evolution and reengineering. IEEE, pp 577–578

  24. Kochhar PS, Thung F, Lo D (2014) Automatic fine-grained issue report reclassification. In: 2014 19th international conference on engineering of complex computer systems. IEEE, pp 126–135

  25. Krishnamurthy S, Tripathi AK (2006) Bounty programs in free/libre/open source software. In: The economics of open source software development. Elsevier, pp 165–183

  26. Krishnamurthy S, Ou S, Tripathi AK (2014) Acceptance of monetary rewards in open source software development. Res Policy 43(4):632–644

    Article  Google Scholar 

  27. Kuhn M, et al. (2008) Building predictive models in r using the caret package. J Stat Softw 28(5):1–26

    Article  Google Scholar 

  28. Lakhani KR, Wolf RG (2003) Why hackers do what they do: Understanding motivation and effort in free/open source software projects

  29. Lee A, Carver JC, Bosu A (2017) Understanding the impressions, motivations, and barriers of one time code contributors to floss projects: a survey. In: IEEE/ACM 39th international conference on software engineering (ICSE), pp 187–197

  30. Maillart T, Zhao M, Grossklags J, Chuang J (2017) Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs. J Cybersec 3(2):81–90

    Article  Google Scholar 

  31. Mandrekar JN (2010) Receiver operating characteristic curve in diagnostic test assessment. J Thorac Oncol 5(9):1315–1316

    Article  Google Scholar 

  32. Matt A (2020) Bug bounties won’t make you rich (but you should participate anyway). https://www.techrepublic.com/article/bug-bounties-wont-make-you-rich-but-you-should-participate-anyway/,. (last visited: January 21, 2020)

  33. Mirko Z (2020) Full-time bug hunting:, Pros and cons of an emerging career. https://www.helpnetsecurity.com/2020/04/07/bug-hunting-career/. (April 7, 2020)

  34. Mockus A, Fielding RT, Herbsleb JD (2002) Two case studies of open source software development: Apache and mozilla. ACM Trans Softw Eng Methodol (TOSEM) 11(3):309–346

    Article  Google Scholar 

  35. Moore DS, Kirkland S (2007) The basic practice of statistics, vol 2. WH Freeman New York

    Google Scholar 

  36. Nakasai K, Hata H, Matsumoto K (2018) Are donation badges appealing?: a case study of developer responses to eclipse bug reports. IEEE Softw 36 (3):22–27

    Article  Google Scholar 

  37. Rajbahadur GK, Wang S, Kamei Y, Hassan AE (2019) Impact of discretization noise of the dependent variable on machine learning classifiers in software engineering. IEEE Trans Softw Eng

  38. Robert L (2019) Bug bounties continue to rise, but market has its own 1% problem). https://www.darkreading.com/vulnerabilities---threats/vulnerability-management/bug-bounties-continue-to-rise-but-market-has-its-own-1--problem/d/d-id/1335689

  39. Roberts JA, Hann I-H, Slaughter SA (2006) Understanding the motivations, participation, and performance of open source software developers: a longitudinal study of the apache projects. Manag Sci 52(7):984–999

    Article  Google Scholar 

  40. Robles G, Gonzalez-Barahona JM, Herraiz I (2009) Evolution of the core team of developers in libre software projects. In: 2009 6th IEEE international working conference on mining software repositories. IEEE, pp 167–170

  41. Romano J, Kromrey JD, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’s d for evaluating group differences on the nsse and other surveys. In: Annual meeting of the Florida association of institutional research, pp 1–33

  42. Shah SK (2006) Motivation, governance, and the viability of hybrid forms in open source software development. Manag Sci 52(7):1000–1014

    Article  Google Scholar 

  43. Steinmacher I, Silva MAG, Gerosa MA (2014) Barriers faced by newcomers to open source projects: a systematic review. In: IFIP international conference on open source systems. Springer, pp 153–163

  44. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2016) An empirical comparison of model validation techniques for defect prediction models. IEEE Trans Softw Eng 43(1):1–18

    Article  Google Scholar 

  45. Tantithamthavorn C, McIntosh S, Hassan AE, Matsumoto K (2018) The impact of automated parameter optimization on defect prediction models. IEEE Trans Softw Eng 45(7):683–711

    Article  Google Scholar 

  46. Tom R (2020) Firefox’s bug bounty in 2019 and into the future. https://blog.mozilla.org/security/2020/04/23/bug-bounty-2019-and-future/,. (last visited: April 23, 2020)

  47. Vasilescu B, Posnett D, Ray B, van den Brand MG, Serebrenik A, Devanbu P, Filkov V (2015) Gender and tenure diversity in github teams. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems, pp 3789–3798

  48. Von Hippel E (2007) Horizontal innovation networks—by and for users. Indust Corp Change 16(2):293–315

    Article  Google Scholar 

  49. Von Krogh G, Haefliger S, Spaeth S, Wallin MW (2012) Carrots and rainbows: Motivation and social practice in open source software development. MIS Quart:649–676

  50. Wang S, Chen T-H, Hassan AE (2018) Understanding the factors for fast answers in technical Q&A websites. Empir Softw Eng 23(3):1552–1593

    Article  Google Scholar 

  51. Weiss M (2011) Control and diversity in company-led open source projects. Open Sourc Bus Res, (April 2011)

  52. Ye Y, Kishida K (2003) Toward an understanding of the motivation open source software developers. In: Proceedings of the 25th international conference on software engineering (ICSE), pp 419–429

  53. Zhao M, Grossklags J, Chen K (2014) An exploratory study of white hat behaviors in a web vulnerability disclosure program. In: Proc. of the workshop on security information workers. ACM, pp 51–58

  54. Zhao M, Laszka A, Grossklags J (2017) Devising effective policies for bug-bounty platforms and security vulnerability discovery. J Inf Pol 7:372–418

    Google Scholar 

  55. Zhou J, Wang S, Bezemer C-P, Hassan AE (2020a) Bounties on technical Q&A sites: a case study of stack overflow bounties. Empir Softw Eng 25 (1):139–177

    Article  Google Scholar 

  56. Zhou J, Wang S, Bezemer C-P, Zou Y, Hassan AE (2020b) Studying the association between bountysource bounties and the issue-addressing likelihood of github issue reports. IEEE Trans Softw Eng

  57. Zhou M, Mockus A, Ma X, Zhang L, Mei H (2016) Inflow and retention in oss communities with commercial involvement: a case study of three hybrid projects. ACM Trans Softw Eng Methodol (TOSEM) 25(2): 1–29

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Shaowei Wang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is not related to Jiayuan Zhou’s and Haoxiang Zhang’s roles at Huawei.

Communicated by: Kelly Blincoe

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Wang, S., Zhang, H. et al. Studying backers and hunters in bounty issue addressing process of open source projects. Empir Software Eng 26, 81 (2021). https://doi.org/10.1007/s10664-021-09979-z

Download citation

Keywords

  • Bounty
  • Open source projects
  • GitHub
  • Issue addressing