Skip to main content
SpringerLink
Log in

Exploring how software developers work with mention bot in GitHub

  • Regular Paper
  • Published:
CCF Transactions on Pervasive Computing and Interaction Aims and scope Submit manuscript

Abstract

Recently, major software development platforms have started to provide automatic reviewer recommendation (ARR) services for pull requests to improve collaborative coding review process. However, the user experience of ARR is under-investigated. In this paper, we use a two-stage mixed-methods approach to study how software developers perceive and work with the Facebook mention bot, one of the most popular ARR bots in GitHub. Specifically, in Stage I, we conduct archival analysis on projects employing mention bot and a user survey to investigate bot performance. A year later, in Stage II, we revisit these projects and conduct additional surveys and interviews with three user groups: project owners, contributors and reviewers. Results show that developers appreciate mention bot saving their efforts, but are bothered by its unstable setting and unbalanced workload allocation. We conclude with design considerations for improving ARR services.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp. 712–721. IEEE Press, Piscataway (2013). http://dl.acm.org/citation.cfm?id=2486788.2486882

  • Badampudi, D., Britto, R., Unterkalmsteiner, M.: Modern code reviews—preliminary results of a systematic mapping study. In: Proceedings of the Evaluation and Assessment on Software Engineering, EASE ’19, pp. 340–345. ACM, New York (2019). https://doi.org/10.1145/3319008.3319354

  • Balachandran, V.: Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In: Proceedings of ICSE, pp. 931–940. IEEE Press (2013)

  • Chen, L., Wu, W., He, L.: How personality influences users’ needs for recommendation diversity? In: CHI ’13 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’13, pp. 829–834. ACM, New York (2013). https://doi.org/10.1145/2468356.2468505

  • erlend sh: Only activate mention bot on prs without comments (2016). https://github.com/facebook/mention-bot/issues/119. Accessed 16 Sept 2016

  • Facebook: mention-bot (2015). https://github.com/facebookarchive/mention-bot. Accessed 23 May 2019

  • Fejzer, M., Przymus, P., Stencel, K.: Profile based recommendation of code reviewers. J. Intell. Inf. Syst. 50(3), 597–619 (2018). https://doi.org/10.1007/s10844-017-0484-1

    Article  Google Scholar 

  • Ferwerda, B., Yang, E., Schedl, M., Tkalcic, M.: Personality traits predict music taxonomy preferences. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’15, pp. 2241–2246. ACM, New York (2015). https://doi.org/10.1145/2702613.2732754

  • GitHub: Understanding the github flow (2013). https://guides.github.com/introduction/flow/. Accessed 23 May 2019

  • GitHub: Github api v3 (2016). https://developer.github.com/v3/. Accessed 23 May 2019

  • GitHub: About code owners (2017a). https://help.github.com/articles/about-codeowners/. Accessed 23 May 2019

  • Github: request review in github (2017b). https://help.github.com/articles/requesting-a-pull-request-review/. Accessed 23 May 2019

  • Gousios, G., Pinzger, M., Deursen, A.V.: An exploratory study of the pull-based software development model. In: Proceedings of ICSE, pp. 345–355. ACM, New York (2014)

  • Gousios, G., Zaidman, A., Storey, M.A., Van Deursen, A.: Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of ICSE, pp. 358–368. IEEE Press (2015)

  • Gousios, G., Storey, M.A., Bacchelli, A.: Work practices and challenges in pull-based development: the contributor’s perspective. In: 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pp. 285–296 (2016a). https://doi.org/10.1145/2884781.2884826

  • Gousios, G., Storey, M.A., Bacchelli, A.: Work practices and challenges in pull-based development: the contributor’s perspective. In: Proceedings of ICSE, pp. 285–296. ACM, New York (2016b)

  • Hoaglin, D.C., Iglewicz, B., Tukey, J.W.: Performance of some resistant rules for outlier labeling. J. Am. Stat. Assoc. 81(396), 991–999 (1986)

    Article  MathSciNet  Google Scholar 

  • Jiang, J., He, J.H., Chen, X.Y.: Coredevrec: automatic core member recommendation for contribution evaluation. J. Comput. Sci. Technol. 30(5), 998–1016 (2015)

    Article  Google Scholar 

  • Lee, M.K., Kusbit, D., Metsky, E., Dabbish, L.: Working with machines: the impact of algorithmic and data-driven management on human workers. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, pp. 1603–1612. ACM, New York (2015). https://doi.org/10.1145/2702123.2702548

  • Pham, R., Singer, L., Liskin, O., Filho, F.F., Schneider, K.: Creating a shared understanding of testing culture on a social coding site. In: 2013 35th International Conference on Software Engineering (ICSE), pp. 112–121 (2013). https://doi.org/10.1109/ICSE.2013.6606557

  • Rahman, M.M., Roy, C.K., Collins, J.A.: Correct: Code reviewer recommendation in github based on cross-project and technology experience. In: 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C), pp. 222–231 (2016)

  • Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: CHI ’02 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’02, pp. 830–831. ACM, New York (2002). https://doi.org/10.1145/506443.506619

  • Stolze, M., Nart, F.: Well-integrated needs-oriented recommender components regarded as helpful. In: CHI ’04 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’04, pp. 1571–1571. ACM, New York (2004). https://doi.org/10.1145/985921.986147

  • Thongtanunam, P., Tantithamthavorn, C., Kula, R.G., Yoshida, N., Iida, H., Matsumoto, K.i.: Who should review my code? a file location-based code-reviewer recommendation approach for modern code review. In: Proceedings of SANER, pp. 141–150. IEEE (2015)

  • Tsay, J., Dabbish, L., Herbsleb, J.: Influence of social and technical factors for evaluating contribution in github. In: Proceedings of ICSE, pp. 356–366. ACM, New York (2014)

  • Webhooks: webhooks (2017). https://developer.github.com/webhooks/. Accessed 01 Sept 2017

  • Xia, X., Lo, D., Wang, X., Yang, X.: Who should review this change?: putting text and file location analyses together for more accurate recommendations. In: Proceedings of ICSME, pp. 261–270. IEEE (2015)

  • Yu, Y., Wang, H., Filkov, V., Devanbu, P., Vasilescu, B.: Wait for it: determinants of pull request evaluation latency on github. In: Proceedings of MSR, pp. 367–371. IEEE (2015)

  • Yu, Y., Wang, H., Yin, G., Wang, T.: Reviewer recommendation for pull-requests in github: what can we learn from code review and bug assignment? Inf. Softw. Technol. 74, 204–218 (2016)

    Article  Google Scholar 

  • Zanjani, M., Kagdi, H., Bird, C.: Automatically recommending peer reviewers in modern code review. Trans. Softw. Eng. 42, 530–542 (2016)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong

    Zhenhui Peng & Xiaojuan Ma

Authors
  1. Zhenhui Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaojuan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhenhui Peng.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, Z., Ma, X. Exploring how software developers work with mention bot in GitHub. CCF Trans. Pervasive Comp. Interact. 1, 190–203 (2019). https://doi.org/10.1007/s42486-019-00013-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42486-019-00013-2

Keywords

Search

Navigation