Skip to main content
SpringerLink
Log in

More than React: Investigating the Role of Emoji Reaction in GitHub Pull Requests

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Open source software development has become more social and collaborative, evident GitHub. Since 2016, GitHub started to support more informal methods such as emoji reactions, with the goal to reduce commenting noise when reviewing any code changes to a repository. From a code review context, the extent to which emoji reactions facilitate a more efficient review process is unknown. We conduct an empirical study to mine 1,850 active repositories across seven popular languages to analyze 365,811 Pull Requests (PRs) for their emoji reactions against the review time, first-time contributors, comment intentions, and the consistency of the sentiments. Answering these four research perspectives, we first find that the number of emoji reactions has a significant correlation with the review time. Second, our results show that a PR submitted by a first-time contributor is less likely to receive emoji reactions. Third, the results reveal that the comments with an intention of information giving, are more likely to receive an emoji reaction. Fourth, we observe that only a small proportion of sentiments are not consistent between comments and emoji reactions, i.e., with 11.8% of instances being identified. In these cases, the prevalent reason is when reviewers cheer up authors that admit to a mistake, i.e., acknowledge a mistake. Apart from reducing commenting noise, our work suggests that emoji reactions play a positive role in facilitating collaborative communication during the review process.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available in the GitHub repository, https://github.com/NAIST-SE/EmojiReaction_PR.

Notes

  1. https://github.com/search, 2021

  2. https://tinyurl.com/3rpdr6dp

  3. https://graphql.org/

  4. https://www.surveysystem.com/sscalc.htm

References

  • Agrawal R, Imieliński T, Swami A (1993) Mining association rules between sets of items in large databases. In: Proceedings of the 1993 ACM SIGMOD international conference on Management of data, pp 207–216

  • Agrawal V, Lin YH, Cheng J (2022) Understanding the characteristics of visual contents in open source issue discussions: A case study of jupyter notebook. In: The International Conference on Evaluation and Assessment in Software Engineering 2022, Association for Computing Machinery, p 249–254, https://doi.org/10.1145/3530019.3534082

  • Bai Q, Dan Q, Mu Z, Yang M (2019) A systematic review of emoji: Current research and future perspectives. Frontiers in psychology 10:2221

    Article  Google Scholar 

  • Baysal O, Kononenko O, Holmes R, Godfrey MW (2013) The influence of non-technical factors on code review. In: 2013 20th Working Conference on Reverse Engineering (WCRE), pp 122–131

  • Baysal O, Kononenko O, Holmes R, Godfrey MW (2016) Investigating Technical and Non-technical Factors Influencing Modern Code Review. Emp Softw Eng, 932–959

  • Borges H, Brito R, Valente MT (2019) Beyond textual issues: Understanding the usage and impact of github reactions. In: Proceedings of the XXXIII Brazilian Symposium on Software Engineering, pp 397–406

  • Charmaz K (2014) Constructing Grounded Theory. SAGE

  • Chen Z, Cao Y, Yao H, Lu X, Peng X, Mei H, Liu X (2021) Emoji-powered sentiment and emotion detection from software developers’ communication data. ACM Transactions on Software Engineering and Methodology (TOSEM) 30(2):1–48

    Google Scholar 

  • Chui M, Manyika J, Bughin J (2012) The social economy: Unlocking value and productivity through social technologies. Tech. rep, McKinsey Global Institute

    Google Scholar 

  • Claes M, Mäntylä M, Farooq U (2018) On the use of emoticons in open source software development. In: Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp 1–4

  • Dabbish L, Stuart C, Tsay J, Herbsleb J (2012) Social coding in github: transparency and collaboration in an open software repository. In: Proceedings of the ACM 2012 conference on computer supported cooperative work, pp 1277–1286

  • Dey T, Mousavi S, Ponce E, Fry T, Vasilescu B, Filippova A, Mockus A (2020) Detecting and characterizing bots that commit code. In: Proceedings of the 17th international conference on mining software repositories, pp 209–219

  • Ebert F, Castor F, Novielli N, Serebrenik A (2019) Confusion in code reviews: Reasons, impacts, and coping strategies. In: 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp 49–60

  • Eisenhardt KM (1989) Building theories from case study research. Academy of management review

  • El Asri I, Kerzazi N, Uddin G, Khomh F, Idrissi MJ (2019) An empirical study of sentiments in code reviews. Information and Software Technology 114:37–54

    Article  Google Scholar 

  • Fu L, Liang P, Zhang B (2022) Understanding code snippets in code reviews: A preliminary study of the openstack community. In: 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC), IEEE

  • Golzadeh M, Decan A, Legay D, Mens T (2021) A ground-truth dataset and classification model for detecting bots in github issue and pr comments. J Syst Softw 175:110911

    Article  Google Scholar 

  • Golzadeh M, Decan A, Chidambaram N (2022) On the accuracy of bot detection techniques. In: International Workshop on Bots in Software Engineering (BotSE). IEEE

  • Guntuku SC, Li M, Tay L, Ungar LH (2019) Studying cultural differences in emoji usage across the east and the west. Proceedings of the international AAAI conference on web and social media 13:226–235

    Article  Google Scholar 

  • Harrell Jr FE, Lee KL, Califf RM, Pryor DB, Rosati RA (1984) Regression modelling strategies for improved prognostic prediction. Stat Med, 143–152

  • Hastie T, Tibshirani R, Friedman J (2009) The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer

  • Hata H, Treude C, Kula RG, Ishio T (2019) 9.6 million links in source code comments: Purpose, evolution, and decay. In: Proceedings of the 41st International Conference on Software Engineering, IEEE Press, ICSE ’19, p 1211–1221

  • Herring SC, Dainas AR (2020) Gender and age influences on interpretation of emoji functions. ACM Trans Soc Comput 3(2):1–26

    Article  Google Scholar 

  • Hirao T, McIntosh S, Ihara A, Matsumoto K (2019) The review linkage graph for code review analytics: a recovery approach and empirical study. In: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 578–589

  • Hirao T, McIntosh S, Ihara A, Matsumoto K (2020) Code reviews with divergent review scores: An empirical study of the openstack and qt communities. IEEE Transactions on Software Engineering

  • Huang Q, Xia X, Lo D, Murphy GC (2018) Automating intention mining. IEEE Transactions on Software Engineering 46(10):1098–1119

    Article  Google Scholar 

  • Islam MR, Zibran M (2018) Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text. J Syst Softw 145:125–146

    Article  Google Scholar 

  • Jiang Y, Adams B, German DM (2013) Will my patch make it? and how fast? case study on the linux kernel. In: 2013 10th Working Conference on Mining Software Repositories (MSR), IEEE, pp 101–110

  • Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) Investigating code review quality: Do people and participation matter? In: 2015 IEEE international conference on software maintenance and evolution (ICSME), IEEE, pp 111–120

  • Kononenko O, Rose T, Baysal O, Godfrey M, Theisen D, de Water B (2018) Studying pull request merges: A case study of shopify’s active merchant. In: 2018 IEEE/ACM 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pp 124–133

  • Kuramoto H, Kondo M, Kashiwa Y, Ishimoto Y, Shindo K, Kamei Y, Ubayashi N (2022) Do visual issue reports help developers fix bugs?:-a preliminary study of using videos and images to report issues on github. In: 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC), IEEE, pp 511–515

  • Li W, Meng N, Li L, Cai H (2021) Understanding language selection in multi-language software projects on github. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), IEEE, pp 256–257

  • Maddila C, Bansal C, Nagappan N (2019) Predicting pull request completion time: a case study on large scale cloud services. In: Proceedings of the 2019 27th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 874–882

  • McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects. In: Proceedings of the 11th Working Conference on Mining Software Repositories, p 192–201

  • Mcintosh S, Kamei Y, Adams B, Hassan AE (2016) An empirical study of the impact of modern code review practices on software quality. Empirical Software Engineering p 2146–2189

  • Nayebi M (2020) Eye of the mind: Image processing for social coding. In: 2020 IEEE/ACM 42nd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER), IEEE, pp 49–52

  • Paternoster R, Brame R, Mazerolle P, Piquero A (1998) Using the correct statistical test for the equality of regression coefficients. Criminology 36(4):859–866

    Article  Google Scholar 

  • Pearson K (1900) X. on the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 50(302):157–175

  • Prana GAA, Treude C, Thung F, Atapattu T, Lo D (2019) Categorizing the content of github readme files. Empirical Software Engineering 24(3):1296–1327

    Article  Google Scholar 

  • Rigby PC, Storey MA (2011) Understanding Broadcast Based Peer Review on Open Source Software Projects. In: Proceedings of the 33rd International Conference on Software Engineering, pp 541–550

  • Rong S, Wang W, Mannan UA, de Almeida ES, Zhou S, Ahmed I (2022) An empirical study of emoji use in software development communication. Information and Software Technology 148:106912

    Article  Google Scholar 

  • Saadat S, Colmenares N, Sukthankar G (2021) Do bots modify the workflow of github teams? In: 2021 IEEE/ACM Third International Workshop on Bots in Software Engineering (BotSE), IEEE, pp 1–5

  • Sadowski C, Söderberg E, Church L, Sipko M, Bacchelli A (2018) Modern code review: a case study at google. In: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, pp 181–190

  • Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611

    Article  MathSciNet  MATH  Google Scholar 

  • Son T, Xiao T, Wang D, Kula RG, Ishio T, Matsumoto K (2021) More than react: Investigating the role of emojireaction in github pull requests. arXiv preprint arXiv:2108.08094

  • Tantisuwankul J, Nugroho YS, Kula RG, Hata H, Rungsawang A, Leelaprute P, Matsumoto K (2019) A topological analysis of communication channels for knowledge sharing in contemporary github projects. Journal of Systems and Software 158:110416

    Article  Google Scholar 

  • Thongtanunam P, Mcintosh S, Hassan AE, Iida H (2017) Review Participation in Modern Code Review. Empirical Software Engineering 22(2):768–817

    Article  Google Scholar 

  • Tigwell GW, Flatla DR (2016) Oh that’s what you meant! reducing emoji misunderstanding. In: Proceedings of the 18th international conference on human-computer interaction with mobile devices and services adjunct, pp 859–866

  • Venigalla ASM, Chimalakonda S (2021) Stackemo: towards enhancing user experience by augmenting stack overflow with emojis. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp 1550–1554

  • Wang D, Ueda Y, Kula RG, Ishio T, Matsumoto K (2021) Can we benchmark code review studies? a systematic mapping study of methodology, dataset, and metric. Journal of Systems and Software 180:111009

    Article  Google Scholar 

  • Wang D, Xiao T, Thongtanunam P, Kula RG, Matsumoto K (2021b) Understanding shared links and their intentions to meet information needs in modern code review. In: The Journal of Empirical Software Engineering (EMSE), vol 26, p to appear, https://doi.org/10.1007/s10664-021-09997-x

  • Xiao T, Wang D, Mcintosh S, Hata H, Kula RG, Ishio T, Matsumoto K (2021) Characterizing and mitigating self-admitted technical debt in build systems. IEEE Transactions on Software Engineering

  • Zhang X, Yu Y, Wang T, Rastogi A, Wang H (2022) Pull request latency explained: An empirical overview. Empirical Software Engineering 27(6):1–38

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Japanese Society for the Promotion of Science (JSPS) KAKENHI Grant Numbers 18H04094 and 20K19774 and 20H05706.

Author information

Authors and Affiliations

  1. Kyushu University, Fukuoka, Japan

    Dong Wang & Yasutaka Kamei

  2. Nara Institute of Science and Technology, Ikoma, Japan

    Tao Xiao, Teyon Son, Raula Gaikovina Kula, Takashi Ishio & Kenichi Matsumoto

Authors
  1. Dong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Teyon Son
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raula Gaikovina Kula
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Takashi Ishio
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yasutaka Kamei
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kenichi Matsumoto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Wang.

Ethics declarations

Conflicts of interest

The authors declare that Raula Gaikovina Kula and Yasutaka Kamei are members of the EMSE Editorial Board. All co-authors have seen and agree with the contents of the manuscript and there is no financial interest to report.

Additional information

Communicated by: Maria Teresa Baldassarre and Christoph Treude

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Registered Reports.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, D., Xiao, T., Son, T. et al. More than React: Investigating the Role of Emoji Reaction in GitHub Pull Requests. Empir Software Eng 28, 123 (2023). https://doi.org/10.1007/s10664-023-10336-5

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10336-5

Keywords

Search

Navigation