Empirical Software Engineering

, Volume 22, Issue 2, pp 768–817 | Cite as

Review participation in modern code review

An empirical study of the android, Qt, and OpenStack projects
  • Patanamon Thongtanunam
  • Shane McIntosh
  • Ahmed E. Hassan
  • Hajimu Iida
Article

Abstract

Software code review is a well-established software quality practice. Recently, Modern Code Review (MCR) has been widely adopted in both open source and proprietary projects. Our prior work shows that review participation plays an important role in MCR practices, since the amount of review participation shares a relationship with software quality. However, little is known about which factors influence review participation in the MCR process. Hence, in this study, we set out to investigate the characteristics of patches that: (1) do not attract reviewers, (2) are not discussed, and (3) receive slow initial feedback. Through a case study of 196,712 reviews spread across the Android, Qt, and OpenStack open source projects, we find that the amount of review participation in the past is a significant indicator of patches that will suffer from poor review participation. Moreover, we find that the description length of a patch shares a relationship with the likelihood of receiving poor reviewer participation or discussion, while the purpose of introducing new features can increase the likelihood of receiving slow initial feedback. Our findings suggest that the patches with these characteristics should be given more attention in order to increase review participation, which will likely lead to a more responsive review process.

Keywords

Code review Review participation Developer involvement 

Notes

Acknowledgments

This research was supported by the Grant-in-Aid for JSPS Fellows (Numbers 16J02861).

References

  1. Abelein U, Paech B (2013) Understanding the influence of user participation and involvement on system success a systematic mapping study. Empir Softw Eng (EMSE) 20(1):28–31CrossRefGoogle Scholar
  2. Bacchelli A, Bird C (2013) Expectations, Outcomes, and Challenges of Modern Code Review. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 712–721Google Scholar
  3. Bacchelli A, Lanza M, Robbes R (2010) Linking E-Mails and Source Code Artifacts. In: Proceedings of the 32nd International Conference on Software Engineering (ICSE), pp 375–384Google Scholar
  4. Balachandran V (2013) Reducing Human Effort and Improving Quality in Peer Code Reviews using Automatic Static Analysis and Reviewer Recommendation. In: Proceedings of the 35th International Conference on Software Engineering (ICSE), pp 931–940Google Scholar
  5. Bavota G, Russo B (2015) Four Eyes Are Better Than Two: On the Impact of Code Reviews on Software Quality. In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 81–90Google Scholar
  6. Baysal O, Kononenko O, Holmes R, Godfrey MW (2012) The Secret Life of Patches: A Firefox Case Study. In: Proceedings of the 19th Working Conference on Reverse Engineering (WCRE), pp 447–455Google Scholar
  7. Baysal O, Kononenko O, Holmes R, Godfrey MW (2015) Investigating technical and non-technical factors influencing modern code review. Empirical Software Engineering (EMSE) 1–28Google Scholar
  8. Beller M, Bacchelli A, Zaidman A, Juergens E (2014) Modern Code Reviews in Open-Source Projects: Which Problems Do They Fix?. In: Proceedings of the 11th Working Conference on Mining Software Repositorie (MSR), pp 202–211Google Scholar
  9. Bettenburg N, Hassan AE, Adams B, German DM (2013) Management of community contributions - A case study on the Android and Linux software ecosystems. Empirical Software Engineering (EMSE) 1–38Google Scholar
  10. Bird C, Gourley A, Devanbu P, Gertz M, Swaminathan A (2006) Mining Email Social Networks. In: Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR), pp 137–143Google Scholar
  11. Bird C, Gourley A, Devanbu P (2007) Detecting Patch Submission and Acceptance in OSS Projects. In: Proceedings of the 4th International Workshop on Mining Software Repositories (MSR), pp 26–29Google Scholar
  12. Bird C, Nagappan N, Murphy B, Gall H, Devanbu P (2011) Don’t Touch My Code! Examining the Effects of Ownership on Software Quality. In: Proceedings of the 8th joint meeting of the European Software Engineering Conference and the Internaltional Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 4–14Google Scholar
  13. Bosu A, Carver JC (2013) Impact of Peer Code Review on Peer Impression Formation: A Survey. In: Proceedings of the 7th International Symposium on Empirical Software Engineering and Measurement (ESEM), pp 133–142Google Scholar
  14. Cramėr H (1999) Mathematical methods of statistics. University Press, Princeton, p 9MATHGoogle Scholar
  15. Efron B (1986) How biased is the apparent error rate of a prediction Rule? J Am Stat Assoc 81(394):461–470MathSciNetCrossRefMATHGoogle Scholar
  16. Fagan ME (1999) Design and code inspections to reduce errors in program development. IBM Syst J 38(2-3):258–287CrossRefGoogle Scholar
  17. Fowler M, Foemmel M (2006) Continuous integration. http://www.thoughtworks.com/ContinuousIntegration.pdf
  18. Gousios G, Pinzger M, van Deursen A (2014) An Exploratory Study of the Pull-based Software Development Model. In: Proceedings of the 36th International Conference on Software Engineering (ICSE), pp 345–355Google Scholar
  19. Guzzi A, Bacchelli A, Lanza M, Pinzger M, Van Deursen A (2013) Communication in Open Source Software Development Mailing Lists. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 277–286Google Scholar
  20. Hamasaki K, Kula RG, Yoshida N, Erika CCA, Fujiwara K, Iida H (2013) Who does what during a Code Review? An extraction of an OSS Peer Review Repository. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 49–52Google Scholar
  21. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiological Society of North America 143(1):29–36Google Scholar
  22. Harrell FE Jr (2002) Regression Modeling Strategies: With Application to Linear Models, Logistic Regression, and Survival Analysis, 1st. Springer, BerlinGoogle Scholar
  23. Harrell FE Jr (2015) rms: Regression Modeling Strategies. http://biostat.mc.vanderbilt.edu/rms
  24. Hassan AE (2008) Automated Classification of Change Messages in Open Source Projects. In: Proceedings of the 23rd Symposium on Applied Computing (SAC), pp 837–841Google Scholar
  25. Hassan AE (2009) Predicting Faults Using the Complexity of Code Changes. In: Proceedings of the 31st International Conference on Software Engineering (ICSE), pp 78–88Google Scholar
  26. Hinkle DE, Wiersma W, Jurs SG (1998) Applied statistics for the behavioral sciences, 4th edn. Houghton Mifflin, BostonGoogle Scholar
  27. Jiang Y, Adams B, German DM (2013) Will My Patch Make It? And How Fast? Case Study on the Linux Kernel. In: Proceeding of the 10th International Working Conference on Mining Software Repositories (MSR), pp 101–110Google Scholar
  28. Kamei Y, Shihab E, Adams B, Hassan AE, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. Trans Softw Eng (TSE) 39(6):757–773CrossRefGoogle Scholar
  29. Kim S, Whitehead EJ Jr, Zhang Y (2008) Classifying software changes: Clean or buggy? Trans Softw Eng (TSE) 34(2):181–196CrossRefGoogle Scholar
  30. Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey MW (2015) Investigating Code Review Quality : Do People and Participation Matter ?. In: Proceedings of the 31st International Conference on Software Maintenance and Evolution (ICSME), pp 111–120Google Scholar
  31. McIntosh S, Kamei Y, Adams B, Hassan AE (2014) The Impact of Code Review Coverage and Code Review Participation on Software Quality. In: Proceedings of the 11th International Working Conference on Mining Software Repositorie (MSR), pp 192–201Google Scholar
  32. McIntosh S, Kamei Y, Adams B, Hassan AE (2015) An Empirical Study of the Impact of Modern Code Review Practices on Software Quality. Empirical Software Engineering (EMSE)Google Scholar
  33. Mishra R, Sureka A (2014) Mining Peer Code Review System for Computing Effort and Contribution Metrics for Patch Reviewers. In: Proceedings of the 4th Workshop on Mining Unstructured Data (MUD), pp 11–15Google Scholar
  34. Mockus A, Votta LG (2000) Identifying Reasons for Software Changes using Historic Databases. In: Proceedings of the 16th International Conference on Software Maintainance (ICSM), pp 120–130Google Scholar
  35. Morales R, Mcintosh S, Khomh F (2015) Do Code Review Practices Impact Design Quality? A Case Study of the Qt, VTK, and ITK Projects. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER)Google Scholar
  36. Mukadam M, Bird C, Rigby PC (2013) Gerrit software code review data from android. In: Proceedings of the 10th International Working Conference on Mining Software Repositories (MSR), pp 45–48Google Scholar
  37. Nagappan M, Zimmermann T, Bird C (2013) Diversity in Software Engineering Research. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 466–476Google Scholar
  38. Nagappan N, Zeller A, Zimmermann T, Herzig K, Murphy B (2010) Change bursts as defect predictors. In: Proceedings of the 21st international Symposium on Software Reliability Engineering (ISSRE), pp 309–318Google Scholar
  39. Nurolahzade M, Nasehi SM, Khandkar SH, Rawal S (2009) The Role of Patch Review in Software Evolution: An Analysis of the Mozilla Firefox. In: Proceedings of the join International and Anual ERCIM Workshops on Principles of Software Evolution and Software Evolution Workshop (IWPSE-Evol), pp 9–17Google Scholar
  40. Porter A, Siy H, Mockus A, Votta L (1998) Understanding the sources of variation in software inspections. Trans Softw Eng Methodol (TOSEM) 7(1):41–79CrossRefGoogle Scholar
  41. Ratcliffe JG (2009) Moving Software Quality Upstream: The Positive Impact of Lightweight Peer Code Review. In: Pacific NW Software Quality Conference, pp 1–10Google Scholar
  42. Raymond ES (1999) The cathedral and the bazaar. Knowl, Technol Policy 12 (3):23–49CrossRefGoogle Scholar
  43. Rea LM, Parker RA (2014) Designing and conducting survey research: A comprehensive guide. John Wiley & SonsGoogle Scholar
  44. Rigby PC, Bird C (2013) Convergent Contemporary Software Peer Review Practices. In: Proceedings of the 9th joint meeting of the European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE), pp 202–212Google Scholar
  45. Rigby PC, Storey MA (2011) Understanding Broadcast Based Peer Review on Open Source Software Projects. In: Proceeding of the 33rd International Conference on Software Engineering (ICSE), pp 541–550Google Scholar
  46. Rigby PC, German DM, Storey MA (2008) Open Source Software Peer Review Practices: A Case Study of the Apache Server. In: Proceedings of the 30th International Conference on Software Engineering (ICSE), pp 541–550Google Scholar
  47. Rigby PC, Cleary B, Painchaud F, Storey MA, German DM (2012) Contemporary peer review in action: Lessons from open source development. IEEE Softw 29(6):56–61CrossRefGoogle Scholar
  48. Rigby PC, German DM, Cowen L, Storey MA (2014) Peer review on open-source software projects: Parameters, Statistical Models, and Theory. Transactions on Software Engineering and Methodology (TOSEM) 23(4):Article No. 35Google Scholar
  49. Sarle WS (1990) The VARCLUS procedure, 4th edn. SAS Institute, IncGoogle Scholar
  50. Sauer C, Jeffery DR, Land L, Yetton P (2000) The effectiveness of software development technical reviews: a behaviorally motivated program of research. Trans Softw Eng (TSE) 26(1):1–14CrossRefGoogle Scholar
  51. Shihab E, Jiang ZM, Hassan AE (2009) Studying the Use of Developer IRC Meetings in Open Source Projects. In: Prceedings of the 25th International Conference on Software Maintenance (ICSM), pp 147–156Google Scholar
  52. Shull F, Basili V, Boehm B, Brown AW, Costa P, Lindvall M, Port D, Rus I, Tesoriero R, Zelkowitz M (2002) What We Have Learned About Fighting Defects. In: Proceedings of the 8th International Software Metrics Symposium (METRICS), pp 249–258Google Scholar
  53. Tantithamthavorn C, McIntosh S, Hassan AE, Ihara A, Matsumoto K (2015) The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models. In: Proceedings of the 37th International Conference on Software Engineering (ICSE), pp 812–823Google Scholar
  54. Tao Y, Dang Y, Xie T, Zhang D, Kim S (2012) How Do Software Engineers Understand Code Changes?: An Exploratory Study in Industry. In: Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering (FSE), pp 51:1–51:11Google Scholar
  55. Thongtanunam P, Yang X, Yoshida N, Kula RG, Ana Erika CC, Fujiwara K, Iida H (2014) ReDA: A Web-based Visualization Tool for Analyzing Modern Code Review Dataset. In: The proceeding of the 30th International Conference on Software Maintenance and Evolution (ICSME), pp 606–609Google Scholar
  56. Thongtanunam P, McIntosh S, Hassan AE, Iida H (2015a) Investigating Code Review Practices in Defective Files: An Empirical Study of the Qt System. In: Proceedings of the 12th International Working Conference on Mining Software Repositories (MSR), pp 168–179Google Scholar
  57. Thongtanunam P, Tantithamthavorn C, Kula RG, Yoshida N, Iida H, Matsumoto K (2015b) Who Should Review My Code? A File Location-Based Code-Reviewer Recommendation Approach for Modern Code Review. In: Proceedings of the 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER), pp 141–150Google Scholar
  58. Thongtanunam P, McIntosh S, Hassan AE, Iida H (2016) Revisiting Code Ownership and its Relationship with Software Quality in the Scope of Modern Code Review. In: Proceedings of the 38th International Conference on Software Engineering (ICSE), pp 1039–1050Google Scholar
  59. Tsay J, Dabbish L, Herbsleb J (2014) Let’s Talk About It: Evaluating Contributions through Discussion in GitHub. In: Proceedings of the 22nd International Symposium on the Foundations of Software Engineering (FSE), pp 144–154Google Scholar
  60. Ukkonen E (1985) Algorithms for approximate string matching. Inf Control 64 (1-3):100–118MathSciNetCrossRefMATHGoogle Scholar
  61. Weißgerber P, Neu D, Diehl S (2008) Small Patches Get In !. In: Proceedings of the 2008 international working conference on Mining software repositories (MSR’08), pp 67–75Google Scholar
  62. Zanjani M, Kagdi H, Bird C (2015) Automatically recommending peer reviewers in modern code review. IEEE Trans Softw Eng:1–13Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Software Design and Analysis Lab (SDLAB)Nara Institute of Science and TechnologyNaraJapan
  2. 2.Department of Electrical and Computer EngineeringMcGill UniversityMontréalCanada
  3. 3.Software Analysis and Intelligence Lab (SAIL)Queen’s UniversityKingstonCanada

Personalised recommendations