Skip to main content
SpringerLink
Log in

Bounties on technical Q&A sites: a case study of Stack Overflow bounties

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Technical question and answer (Q&A) websites provide a platform for developers to communicate with each other by asking and answering questions. Stack Overflow is the most prominent of such websites. With the rapidly increasing number of questions on Stack Overflow, it is becoming difficult to get an answer to all questions and as a result, millions of questions on Stack Overflow remain unsolved. In an attempt to improve the visibility of unsolved questions, Stack Overflow introduced a bounty system to motivate users to solve such questions. In this bounty system, users can offer reputation points in an effort to encourage users to answer their question. In this paper, we study 129,202 bounty questions that were proposed by 61,824 bounty backers. We observe that bounty questions have a higher solving-likelihood than non-bounty questions. This is particularly true for long-standing unsolved questions. For example, questions that were unsolved for 100 days for which a bounty is proposed are more likely to be solved (55%) than those without bounties (1.7%). In addition, we studied the factors that are important for the solving-likelihood and solving-time of a bounty question. We found that: (1) Questions are likely to attract more traffic after receiving a bounty than non-bounty questions. (2) Bounties work particularly well in very large communities with a relatively low question solving-likelihood. (3) High-valued bounties are associated with a higher solving-likelihood, but we did not observe a likelihood for expedited solutions. Our study shows that while bounties are not a silver bullet for getting a question solved, they are associated with a higher solving-likelihood of a question in most cases. As questions that are still unsolved after two days hardly receive any traffic, we recommend that Stack Overflow users propose a bounty as soon as possible after those two days for the bounty to have the highest impact.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. https://stackoverflow.com/

  2. https://data.stackexchange.com/

  3. https://data.stackexchange.com/stackoverflow/query/968466

  4. https://fd.zaih.com/fenda

  5. https://techcrunch.com/2016/10/31/justin-kan-launches-video-qa-app-whale/

  6. https://stackoverflow.blog/2009/01/27/reputation-bounty-for-unanswered-questions/

  7. https://stackoverflow.com/help/bounty

  8. https://archive.org/details/stackexchange

  9. https://stackoverflow.blog/2011/09/23/bounty-reasons-and-post-notices/

  10. https://stackoverflow.com/users/userid?tab=reputation

  11. https://stackoverflow.com/questions/845/

  12. https://stackoverflow.com/posts/10038098/revisions

  13. https://github.com/SAILResearch/wip-18-jiayuan-SO-bounty-SupportMaterials/tree/master/data. Please use the given account (paperreviewer2019@gmail.com) and password (paper_reviewer_2019) to access the data. We will make the repository public once the paper is accepted.

  14. https://cran.r-project.org/web/packages/rms/index.html

  15. https://stackoverflow.com/questions/1809670/how-to-implement-serialization-in-c

  16. https://stackoverflow.com/questions/1809986/

  17. http://bit.ly/2HsnxbY

References

  • Ahasanuzzaman M, Asaduzzaman M, Roy CK, Schneider KA (2018) Classifying Stack Overflow posts on API issues. In: 2018 IEEE 25th international conference on software analysis, evolution and reengineering (SANER), pp 244–254

  • Anderson A, Huttenlocher D, Kleinberg J, Leskovec J (2012) Discovering value from community activity on focused question answering sites: a case study of Stack Overflow. In: Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 850–858

  • Anderson A, Huttenlocher D, Kleinberg J, Leskovec J (2013) Steering user behavior with badges. In: Proceedings of the 22nd international conference on World Wide Web, WWW ’13, pp 95–106

  • Bauer DF (1972) Constructing confidence sets using rank statistics. J Am Stat Assoc 67(339):687–690

    Article  Google Scholar 

  • Bonferroni C (1936) Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Scienze Economiche e Commericiali di Firenze 8:3–62

    MATH  Google Scholar 

  • Cavusoglu H, Li Z, Huang K-W (2015) Can gamification motivate voluntary contributions?: the case of Stack Overflow Q&A community. In: Proceedings of the 18th ACM conference companion on computer supported cooperative work & social computing, CSCW’15 companion, pp 171–174

  • Chen C, Xing Z, Liu Y (2017) By the community & for the community: a deep learning approach to assist collaborative editing in Q&A sites. Proc ACM Hum-Comput Interact 1(CSCW):32:1–32:21

    Article  Google Scholar 

  • Chen C, Chen X, Sun J, Xing Z, Li G (2018) Data-driven proactive policy assurance of post quality in community Q&A sites. Proc ACM Hum-Comput Interact 2 (CSCW):33:1–22:33

    Google Scholar 

  • Efron B (1986) How biased is the apparent error rate of a prediction rule? J Am Stat Assoc 81(394):461–470

    Article  MathSciNet  Google Scholar 

  • Farrar DE, Glauber RR (1967) Multicollinearity in regression analysis: the problem revisited. Rev Econ Stat, pp 92–107

  • Finifter M, Akhawe D, Wagner D (2013) An empirical study of vulnerability rewards programs. In: USENIX security symp, pp 273–288

  • Ford D, Lustig K, Banks J, Parnin C (2018) “We don’t do that here”: How collaborative editing with mentors improves engagement in social Q&A communities. In: Proceedings of the 2018 CHI conference on human factors in computing systems, CHI ’18, pp 608:1–608:12

  • Gwet K, et al. (2002) Inter-rater reliability: dependency on trait prevalence and marginal homogeneity. Statistical Methods for Inter-Rater Reliability Assessment Series 2:1–9

    Google Scholar 

  • Hanrahan BV, Convertino G, Nelson L (2012) Modeling problem difficulty and expertise in stackoverflow. In: Proceedings of the ACM 2012 conference on computer supported cooperative work companion. ACM, pp 91–94

  • Harrell Jr E (2006) Regression modeling strategies. Springer-Verlag New York, Inc., Secaucus

    Google Scholar 

  • Hata H, Guo M, Babar MA (2017) Understanding the heterogeneity of contributors in bug bounty programs. In: Proceedings of the 11th ACM/IEEE international symposium on empirical software engineering and measurement, ESEM ’17, pp 223–228

  • Hsieh G, Kraut RE, Hudson SE (2010) Why pay?: exploring how financial incentives are used for question & answer. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 305–314

  • Jan ST, Wang C, Zhang Q, Wang G (2017) Towards monetary incentives in social Q&A services. arXiv:1703.01333

  • Kanda T, Guo M, Hata H, Matsumoto K (2017) Towards understanding an open-source bounty: Analysis of bountysource. In: 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER), IEEE, pp 577–578

  • Krishnamurthy S, Tripathi AK (2006) Bounty programs in free/libre/open source software. In: The economics of open source software development. Elsevier, pp 165–183

  • Liu J, Zhou P, Yang Z, Liu X, Grundy J (2018) Fasttagrec: fast tag recommendation for software information sites. Autom Softw Eng 25(4):675–701

    Article  Google Scholar 

  • Long JD, Feng D, Cliff N (2003) Ordinal analysis of behavioral data. Handbook of psychology

  • Maillart T, Zhao M, Grossklags J, Chuang J (2017) Given enough eyeballs, all bugs are shallow? Revisiting Eric Raymond with bug bounty programs. Journal of Cybersecurity 3(2):81–90

    Article  Google Scholar 

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empir Softw Eng 21 (5):2146–2189

    Article  Google Scholar 

  • Munaiah N, Meneely A (2016) Vulnerability severity scoring and bounties: Why the disconnect?. In: Proceedings of the 2nd international workshop on software analytics, SWAN 2016, pp 8–14

  • Nakasai K, Hata H, Matsumoto K (2018) Are donation badges appealing? a case study of developer responses to eclipse bug reports. IEEE Software

  • Ponzanelli L, Mocci A, Bacchelli A, Lanza M, Fullerton D (2014) Improving low quality Stack Overflow post detection. In: 2014 IEEE international conference on software maintenance and evolution, pp 541–544

  • Rajbahadur GK, Wang S, Kamei Y, Hassan AE (2017) The impact of using regression models to build defect classifiers. In: Proceedings of the 14th international conference on mining software repositories (MSR), pp 135–145

  • Romano J, Kromrey JD, Coraggio J, Skowronek J (2006) Appropriate statistics for ordinal level data: Should we really be using t-test and cohen’sd for evaluating group differences on the nsse and other surveys. In: Annual meeting of the Florida association of institutional research, pp 1–33

  • Srba I, Bielikova M (2016) Why is Stack Overflow failing? Preserving sustainability in community question answering. IEEE Softw 33(4):80–89

    Article  Google Scholar 

  • Stack Exchange (2017) Stack exchange. https://archive.org/details/stackexchange. (last visited: Dec. 20, 2017)

  • Stack Overflow (2019) Stack Overflow: User privileges. https://stackoverflow.com/help/privileges, (last visited: Jan. 23, 2019)

  • Tantithamthavorn C, Hassan AE (2018) An experience report on defect modelling in practice: Pitfalls and challenges. In: Proceedings of the 40th international conference on software engineering: software engineering in practice. ACM, pp 286–295

  • Thongtanunam P, McIntosh S, Hassan AE, Iida H (2016) Revisiting code ownership and its relationship with software quality in the scope of modern code review. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, pp 1039–1050

  • Tian Y, Nagappan M, Lo D, Hassan AE (2015) What are the characteristics of high-rated apps? A case study on free android applications. In: 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 301–310

  • Treude C, Barzilay O, Storey M-A (2011) How do programmers ask and answer questions on the web?: Nier track. In: 2011 33rd international conference on software engineering (ICSE). IEEE, pp 804–807

  • Wang S, Lo D, Vasilescu B, Serebrenik A (2014) Entagrec: an enhanced tag recommendation system for software information sites. In: 2014 IEEE international conference on software maintenance and evolution. IEEE, pp 291–300

  • Wang S, Lo D, Vasilescu B, Serebrenik A (2018a) Entagrec ++: an enhanced tag recommendation system for software information sites. Empir Softw Eng 23(2):800–832

    Article  Google Scholar 

  • Wang S, Chen T-HP, Hassan AE (2018b) How do users revise answers on technical Q&A websites? A case study on Stack Overflow. IEEE Transactions on Software Engineering

  • Wang S, Chen T-H, Hassan AE (2018c) Understanding the factors for fast answers in technical Q&A websites. Empir Softw Eng 23(3):1552–1593

    Article  Google Scholar 

  • Wu Y, Wang S, Bezemer C-P, Inoue K (2018) How do developers utilize source code from Stack Overflow?. Empir Softw Eng, pp 637–673

  • Xia X, Lo D, Wang X, Zhou B (2013) Tag recommendation in software information sites. In: Proceedings of the 10th working conference on mining software repositories, MSR ?13, San Francisco, CA, USA, May 18-19, 2013

  • Zhang H, Wang S, Chen T-HP, Hassan AE (2019) An empirical study of obsolete answers on Stack Overflow. IEEE Transactions on Software Engineering

  • Zhao M, Grossklags J, Chen K (2014) An exploratory study of white hat behaviors in a web vulnerability disclosure program. In: Proc of the workshop on security information workers. ACM, pp 51–58

  • Zhao M, Laszka A, Grossklags J (2017) Devising effective policies for bug-bounty platforms and security vulnerability discovery. J Inf Policy 7:372–418

    Article  Google Scholar 

  • Zhou J (2019) Supplementary material for our paper. https://github.com/SAILResearch/wip-18-jiayuan-SO-bounty-SupportMaterials/blob/master/appendix.pdf

  • Zhou J, Wang S, Bezemer C-P, Zou Y, Hassan AE (2019) Bounties in open source development on github: A case study of bountysource bounties. arXiv:1904.02724

Download references

Author information

Authors and Affiliations

  1. Software Analysis and Intelligence Lab (SAIL), Queen’s University, Kingston, ON, Canada

    Jiayuan Zhou, Shaowei Wang & Ahmed E. Hassan

  2. Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB, Canada

    Cor-Paul Bezemer

Authors
  1. Jiayuan Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaowei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cor-Paul Bezemer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmed E. Hassan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shaowei Wang.

Additional information

Communicated by: Emerson Murphy-Hill

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, J., Wang, S., Bezemer, CP. et al. Bounties on technical Q&A sites: a case study of Stack Overflow bounties. Empir Software Eng 25, 139–177 (2020). https://doi.org/10.1007/s10664-019-09744-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09744-3

Keywords

Search

Navigation