Empirical Software Engineering

, Volume 23, Issue 6, pp 3346–3393 | Cite as

Early prediction of merged code changes to prioritize reviewing tasks

  • Yuanrui Fan
  • Xin Xia
  • David Lo
  • Shanping Li


Modern Code Review (MCR) has been widely used by open source and proprietary software projects. Inspecting code changes consumes reviewers much time and effort since they need to comprehend patches, and many reviewers are often assigned to review many code changes. Note that a code change might be eventually abandoned, which causes waste of time and effort. Thus, a tool that predicts early on whether a code change will be merged can help developers prioritize changes to inspect, accomplish more things given tight schedule, and not waste reviewing effort on low quality changes. In this paper, motivated by the above needs, we build a merged code change prediction tool. Our approach first extracts 34 features from code changes, which are grouped into 5 dimensions: code, file history, owner experience, collaboration network, and text. And then we leverage machine learning techniques such as random forest to build a prediction model. To evaluate the performance of our approach, we conduct experiments on three open source projects (i.e., Eclipse, LibreOffice, and OpenStack), containing a total of 166,215 code changes. Across three datasets, our approach statistically significantly improves random guess classifiers and two prediction models proposed by Jeong et al. (2009) and Gousios et al. (2014) in terms of several evaluation metrics. Besides, we also study the important features which distinguish merged code changes from abandoned ones.


Code review Predictive model Features 



This work was partially supported by NSFC Program (No. 61602403 and 61572426).


  1. Abdi H (2007) Bonferroni and Šidák corrections for multiple comparisons. Encycl Meas Stat 3:103–107Google Scholar
  2. Ackerman A F, Fowler P J, Ebenau RG (1984) Software inspections and the industrial production of software. In: Proceedings of a symposium on software validation: inspection-testing-verification-alternatives. Elsevier North-Holland Inc., pp 13–40Google Scholar
  3. Arisholm E, Briand L C, Fuglerud M (2007) Data mining techniques for building fault-proneness models in telecom java software. In: The 18th IEEE international symposium on software reliability, 2007. ISSRE’07. IEEE, pp 215–224Google Scholar
  4. Aurum A, Petersson H, Wohlin C (2002) State-of-the-art: software inspections after 25 years. Software Testing. Verif and Reliab 12(3):133–154CrossRefGoogle Scholar
  5. Bacchelli A, Bird C (2013) Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 712–721Google Scholar
  6. Bao L, Xing Z, Xia X, Lo D, Li S (2017) Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR). IEEE, pp 170–181Google Scholar
  7. Baysal O, Kononenko O, Holmes R, Godfrey M W (2013) The influence of non-technical factors on code review. In: 2013 20th working conference on reverse engineering (WCRE). IEEE, pp 122–131Google Scholar
  8. Bhattacharya P, Neamtiu I (2010) Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. In: 2010 IEEE international conference on software maintenance (ICSM). IEEE, pp 1–10Google Scholar
  9. Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefGoogle Scholar
  10. Cliff N (2014) Ordinal methods for behavioral data analysis. Psychology Press, New YorkCrossRefGoogle Scholar
  11. Costa C, Figueiredo J, Sarma A, Murta L (2016) TIPMerge: recommending developers for merging branches. In: Proceedings of the 2016 24th ACM SIGSOFT international symposium on foundations of software engineering. ACM, pp 998–1002Google Scholar
  12. DeGroot MH, Schervish MJ (2012) Probability and statistics. Pearson Education, BostonGoogle Scholar
  13. Elkan C (2001) The foundations of cost-sensitive learning. In: International joint conference on artificial intelligence. Lawrence Erlbaum Associates Ltd, vol 17, pp 973–978Google Scholar
  14. Fagan M E (2001) Design and code inspections to reduce errors in program development. In: Pioneers and their contributions to software engineering. Springer, Berlin, pp 301–334CrossRefGoogle Scholar
  15. Fenton N, Neil M, Marsh W, Hearty P, Marquez D, Krause P, Mishra R (2007) Predicting software defects in varying development lifecycles using Bayesian nets. Inf Softw Technol 49(1):32–43CrossRefGoogle Scholar
  16. Gonzalez-Barahona J M, Izquierdo-Cortazar D, Robles G, del Castillo A (2014) Analyzing gerrit code review parameters with bicho. Electron Commun EASSTGoogle Scholar
  17. Gousios G, Pinzger M, Deursen AV (2014) An exploratory study of the pull-based software development model. In: Proceedings of the 36th international conference on software engineering. ACM, pp 345– 355Google Scholar
  18. Gousios G, Zaidman A, Storey M A, Van Deursen A (2015) Work practices and challenges in pull-based development: the integrator’s perspective. In: Proceedings of the 37th international conference on software engineering-volume 1. IEEE Press, pp 358–368Google Scholar
  19. Graves T L, Karr A F, Marron J S, Siy H (2000) Predicting fault incidence using software change history. IEEE Trans Softw Eng 26(7):653–661CrossRefGoogle Scholar
  20. Grbac T G, Mausa G, Basic B D (2013) Stability of software defect prediction in relation to levels of data imbalance. In: SQAMIA, pp 1–10Google Scholar
  21. Hall M A, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten I H (2009) The WEKA data mining software: an update. Sigkdd Explor 11(1):10–18CrossRefGoogle Scholar
  22. Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier, AmsterdamzbMATHGoogle Scholar
  23. He H, Garcia E A (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284CrossRefGoogle Scholar
  24. Herzig K, Just S, Zeller A (2013) It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In: Proceedings of the 2013 international conference on software engineering. IEEE Press, pp 392–401Google Scholar
  25. Huang J, Ling C X (2005) Using AUC and accuracy in evaluating learning algorithms. IEEE Trans Knowl Data Eng 17(3):299–310CrossRefGoogle Scholar
  26. Huang Q, Xia X, Lo D (2017) Supervised vs unsupervised models: a holistic look at effort-aware just-in-time defect prediction. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 159–170Google Scholar
  27. Jeong G, Kim S, Zimmermann T, Yi K (2009) Improving code review by predicting reviewers and acceptance of patches. In: Research on software analysis for error-free computing center Tech-Memo (ROSAEC MEMO 2009-006), pp 1–18Google Scholar
  28. Jiang T, Tan L, Kim S (2013a) Personalized defect prediction. In: 2013 IEEE/ACM 28th international conference on automated software engineering (ASE). IEEE, pp 279–289Google Scholar
  29. Jiang Y, Adams B, German D M (2013b) Will my patch make it? and how fast? Case study on the linux kernel. In: 2013 10th IEEE working conference on mining software repositories (MSR). IEEE, pp 101– 110Google Scholar
  30. Kamei Y, Shihab E, Adams B, Hassan A E, Mockus A, Sinha A, Ubayashi N (2013) A large-scale empirical study of just-in-time quality assurance. IEEE Trans Softw Eng 39(6):757–773CrossRefGoogle Scholar
  31. Khoshgoftaar T M, Geleyn E, Nguyen L, Bullard L (2002) Cost-sensitive boosting in software quality modeling. In: 7th IEEE international symposium on high assurance systems engineering, 2002. Proceedings. IEEE, pp 51–60Google Scholar
  32. Kim S, Whitehead E J, Zhang Y (2008) Classifying software changes: clean or buggy? IEEE Trans Softw Eng 34(2):181–196CrossRefGoogle Scholar
  33. Kononenko O, Baysal O, Guerrouj L, Cao Y, Godfrey M W (2015) Investigating code review quality: Do people and participation matter?. In: 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 111–120Google Scholar
  34. Lamkanfi A, Demeyer S, Giger E, Goethals B (2010) Predicting the severity of a reported bug. In: 2010 7th IEEE working conference on mining software repositories (MSR). IEEE, pp 1–10Google Scholar
  35. Leßenich O, Siegmund J, Apel S, Kästner C, Hunsen C (2016) Indicators for merge conflicts in the wild: survey and empirical study. Autom Softw Eng 1–35. CrossRefGoogle Scholar
  36. Lessmann S, Baesens B, Mues C, Pietsch S (2008) Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Trans Softw Eng 34(4):485–496CrossRefGoogle Scholar
  37. Liu X Y, Zhou Z H (2006) The influence of class imbalance on cost-sensitive learning: an empirical study. In: Sixth international conference on data mining, 2006. ICDM’06. IEEE, pp 970–974Google Scholar
  38. Liu M, Miao L, Zhang D (2014) Two-stage cost-sensitive learning for software defect prediction. IEEE Trans Reliab 63(2):676–686CrossRefGoogle Scholar
  39. Mann H B, Whitney D R (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 50–60MathSciNetCrossRefGoogle Scholar
  40. Matsumoto S, Kamei Y, Monden A, Ki Matsumoto, Nakamura M (2010) An analysis of developer metrics for fault prediction. In: Proceedings of the 6th international conference on predictive models in software engineering. ACM, p 18Google Scholar
  41. McIntosh S, Kamei Y, Adams B, Hassan A E (2014) The impact of code review coverage and code review participation on software quality: a case study of the qt, vtk, and itk projects. In: Proceedings of the 11th working conference on mining software repositories. ACM, pp 192–201Google Scholar
  42. Mende T, Koschke R (2009) Revisiting the evaluation of defect prediction models. In: Proceedings of the 5th international conference on predictor models in software engineering. ACM, p 7Google Scholar
  43. Mockus A, Weiss D M (2000) Predicting risk of software changes. Bell Labs Tech J 5(2):169–180CrossRefGoogle Scholar
  44. Mukadam M, Bird C, Rigby P C (2013) Gerrit software code review data from android. In: 2013 10th IEEE working conference on mining software repositories (MSR). IEEE, pp 45–48Google Scholar
  45. Rahman F, Posnett D, Devanbu P (2012) Recalling the imprecision of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering. ACM, p 61Google Scholar
  46. Rajbahadur G K, Wang S, Kamei Y, Hassan A E (2017) The impact of using regression models to build defect classifiers. In: Proceedings of the 14th international conference on mining software repositories. IEEE Press, pp 135–145Google Scholar
  47. Ratzinger J, Pinzger M, Gall H (2007) EQ-Mine:predicting short-term defects for software evolution. In: International conference on fundamental approaches to software engineering. Springer, Berlin, pp 12–26Google Scholar
  48. Rigby P C, German D M (2006) A preliminary examination of code review processes in open source projects. Tech. rep., Technical Report DCS-305-IR, University of VictoriaGoogle Scholar
  49. Rigby P C, German D M, Storey M A (2008) Open source software peer review practices: a case study of the apache server. In: Proceedings of the 30th international conference on Software engineering. ACM, pp 541–550Google Scholar
  50. Romano D, Pinzger M (2011) Using source code metrics to predict change-prone java interfaces. In: 2011 27th IEEE international conference on software maintenance (ICSM). IEEE, pp 303–312Google Scholar
  51. Scott A J, Knott M (1974) A cluster analysis method for grouping means in the analysis of variance. Biometrics 30(3):507–512CrossRefGoogle Scholar
  52. Shimagaki J, Kamei Y, McIntosh S, Hassan A E, Ubayashi N (2016) A study of the quality-impacting practices of modern code review at sony mobile. In: Proceedings of the 38th international conference on software engineering companion. ACM, pp 212–221Google Scholar
  53. Shull F, Seaman C (2008) Inspecting the history of inspections: an example of evidence-based technology diffusion. IEEE Softw 25(1):88–90. CrossRefGoogle Scholar
  54. Tamrawi A, Nguyen T T, Al-Kofahi J M, Nguyen T N (2011) Fuzzy set and cache-based approach for bug triaging. In: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, pp 365–375Google Scholar
  55. Thongtanunam P, McIntosh S, Hassan A E, Iida H (2016) Review participation in modern code review. Empir Softw Eng 1–50Google Scholar
  56. Tian Y, Nagappan M, Lo D, Hassan A E (2015) What are the characteristics of high-rated apps? A case study on free android applications. In: 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 301–310Google Scholar
  57. Tsay J, Dabbish L, Herbsleb J (2014) Influence of social and technical factors for evaluating contribution in GitHub. In: Proceedings of the 36th international conference on Software engineering. ACM, pp 356–366Google Scholar
  58. Upton G J (1992) Fisher’s exact test. J R Stat Soc A Stat Soc 155(3):395–402CrossRefGoogle Scholar
  59. Votta L G (1993) Does every inspection need a meeting? ACM SIGSOFT Softw Eng Notes 18(5):107–114CrossRefGoogle Scholar
  60. Weiss G M, McCarthy K, Zabar B (2007) Cost-sensitive learning vs. sampling: which is best for handling unbalanced classes with unequal error costs? DMIN 7:35–41Google Scholar
  61. Weißgerber P, Neu D, Diehl S (2008) Small patches get in! In: Proceedings of the 2008 international working conference on mining software repositories. ACM, pp 67–76Google Scholar
  62. Wilcoxon F (1945) Individual comparisons by ranking methods. Biol Bull 1 (6):80–83Google Scholar
  63. Wolpert D H, Macready W G (1999) An efficient method to estimate bagging’s generalization error. Mach Learn 35(1):41–55CrossRefGoogle Scholar
  64. Xia X, Lo D, Shihab E, Wang X, Yang X (2015a) Elblocker: predicting blocking bugs with ensemble imbalance learning. Inf Softw Technol 61:93–106CrossRefGoogle Scholar
  65. Xia X, Lo D, Wang X, Yang X (2015b) Who should review this change?: Putting text and file location analyses together for more accurate recommendations. In: 2015 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 261–270Google Scholar
  66. Xia X, Lo D, Pan S J, Nagappan N, Wang X (2016a) Hydra: massively compositional model for cross-project defect prediction. IEEE Trans Softw Eng 42 (10):977–998CrossRefGoogle Scholar
  67. Xia X, Lo D, Wang X, Yang X (2016b) Collective personalized change classification with multiobjective search. IEEE Trans Reliab 65(4):1810–1829CrossRefGoogle Scholar
  68. Xia X, Lo D, Ding Y, Al-Kofahi J M, Nguyen TN, Wang X (2017) Improving automated bug triaging with specialized topic model. IEEE Trans Softw Eng 43(3):272–297CrossRefGoogle Scholar
  69. Yang X, Kula R G, Yoshida N, Iida H (2016) Mining the modern code review repositories: a dataset of people, process and product. In: Proceedings of the 13th international conference on mining software repositories. ACM, pp 460–463Google Scholar
  70. Zanetti M S, Scholtes I, Tessone C J, Schweitzer F (2013) Categorizing bugs with social networks: a case study on four open source software communities. In: 2013 35th international conference on software engineering (ICSE). IEEE, pp 1032–1041Google Scholar
  71. Zhang Y, Lo D, Xia X, Xu B, Sun J, Li S (2015) Combining software metrics and text features for vulnerable file prediction. In: 2015 20th international conference on engineering of complex computer systems (ICECCS). IEEE, pp 40–49Google Scholar
  72. Zheng J (2010) Cost-sensitive boosting neural networks for software defect prediction. Expert Syst Appl 37(6):4537–4543CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyZhejiang UniversityHangzhouChina
  2. 2.Faculty of Information TechnologyMonash UniversityMelbourneAustralia
  3. 3.School of Information SystemsSingapore Management UniversitySingaporeSingapore

Personalised recommendations