The Choice of Code Review Process: A Survey on the State of the Practice

  • Tobias BaumEmail author
  • Hendrik Leßmann
  • Kurt Schneider
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10611)


Code review has been known to be an effective quality assurance technique for decades. In the last years, industrial code review practices were observed to converge towards “change-based/modern code review”, but with a lot of variation in the details of the processes. Recent research also proposed hypotheses on factors that influence the choice of process. However, all current research in this area is based on small and largely non-random samples of cases. Therefore, we set out to assess the current state of the practice and to test some of these hypotheses with a survey among commercial software development teams. We received responses from 240 teams. They support many of the stated hypotheses, e.g., that change-based code review is the dominating style of code review in the industry, and that teams doing change-based code review have a lower risk that review use fades away. However, other hypotheses could not be confirmed, mainly that the balance of effects a team tries to reach with code reviews acts as a mediator in determining the details of the review process. Apart from these findings, we contribute the survey data set as a foundation for future research.


Code reviews Code inspections and walkthroughs Change-based code review Modern code review Empirical software engineering 



The authors would like to thank all pre-testers and all participants of the survey for the time and effort they donated. We would further like to thank Philipp Diebold and Paul Clarke for providing questions from their surveys for reuse.


  1. 1.
    Asundi, J., Jayant, R.: Patch review processes in open source software development communities: a comparative case study. In: 40th Annual Hawaii International Conference on System Sciences, HICSS 2007, p. 166c. IEEE (2007)Google Scholar
  2. 2.
    Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 712–721. IEEE (2013)Google Scholar
  3. 3.
    Baker Jr., R.A.: Code reviews enhance software quality. In: Proceedings of the 19th International Conference on Software Engineering, pp. 570–571. ACM (1997)Google Scholar
  4. 4.
    Basili, V., Caldiera, G., Lanubile, F., Shull, F.: Studies on reading techniques. In: Proceedings of the Twenty-First Annual Software Engineering Workshop, vol. 96, p. 002 (1996)Google Scholar
  5. 5.
    Basili, V.R., Green, S., Laitenberger, O., Lanubile, F., Shull, F., Sørumgård, S., Zelkowitz, M.V.: The empirical investigation of perspective-based reading. Empirical Softw. Eng. 1(2), 133–164 (1996)CrossRefGoogle Scholar
  6. 6.
    Baum, T.: Detailed table with review effects (team level) and their connections to contextual factors and process variants for “factors influencing code review processes in industry” (2016).
  7. 7.
    Baum, T., Leßmann, H., Schneider, K.: Online material for survey on code review use.
  8. 8.
    Baum, T., Liskin, O., Niklas, K., Schneider, K.: A faceted classification scheme for change-based industrial code review processes. In: 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, Vienna (2016)Google Scholar
  9. 9.
    Baum, T., Liskin, O., Niklas, K., Schneider, K.: Factors influencing code review processes in industry. In: Proceedings of the ACM SIGSOFT 24th International Symposium on the Foundations of Software Engineering. ACM, Seattle (2016)Google Scholar
  10. 10.
    Baysal, O., Kononenko, O., Holmes, R., Godfrey, M.W.: Investigating technical and non-technical factors influencing modern code review. Empirical Softw. Eng. 21, 932–959 (2016)CrossRefGoogle Scholar
  11. 11.
    Bernhart, M., Strobl, S., Mauczka, A., Grechenig, T.: Applying continuous code reviews in airport operations software. In: 2012 12th International Conference on Quality Software (QSIC), pp. 214–219. IEEE (2012)Google Scholar
  12. 12.
    Bosu, A., Carver, J.C.: Impact of peer code review on peer impression formation: a survey. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 133–142. IEEE (2013)Google Scholar
  13. 13.
    Ciolkowski, M., Laitenberger, O., Biffl, S.: Software reviews: the state of the practice. IEEE Softw. 20(6), 46–51 (2003)CrossRefGoogle Scholar
  14. 14.
    Fagan, M.E.: Design and code inspections to reduce errors in program development. IBM Syst. J. 15(3), 182–211 (1976)CrossRefGoogle Scholar
  15. 15.
    Forrester Research Inc.: The value and importance of code reviews, March 2010. Accessed 13 June 2017
  16. 16.
    Gousios, G., Pinzger, M., Deursen, A.V.: An exploratory study of the pull-based software development model. In: Proceedings of the 36th International Conference on Software Engineering, pp. 345–355. ACM, Hyderabad (2014)Google Scholar
  17. 17.
    Harjumaa, L., Tervonen, I., Huttunen, A.: Peer reviews in real life-motivators and demotivators. In: Fifth International Conference on Quality Software (QSIC 2005). IEEE (2005)Google Scholar
  18. 18.
    Jacob, R., Heinz, A., Décieux, J.P.: Umfrage: Einführung in die Methoden der Umfrageforschung. Walter de Gruyter (2013)Google Scholar
  19. 19.
    Kollanus, S., Koskinen, J.: Software inspections in practice: six case studies. In: Münch, J., Vierimaa, M. (eds.) PROFES 2006. LNCS, vol. 4034, pp. 377–382. Springer, Heidelberg (2006). doi: 10.1007/11767718_31CrossRefGoogle Scholar
  20. 20.
    Kuhrmann, M., Diebold, P., Münch, J., Tell, P., Garousi, V., Felderer, M., Trektere, K., Linssen, O., Hanser, E., Prause, C.R.: Hybrid software and system development in practice: waterfall, scrum, and beyond. In: ICSSP 2017 (2017)Google Scholar
  21. 21.
    Laitenberger, O., Vegas, S., Ciolkowski, M.: The state of the practice of review and review technologies in Germany. Tech. report 011.02, Virtual Softw. Eng. Competence Center (VISEK) (2002)Google Scholar
  22. 22.
    Laitenberger, O., DeBaud, J.M.: An encompassing life cycle centric survey of software inspection. J. Syst. Softw. 50(1), 5–31 (2000)CrossRefGoogle Scholar
  23. 23.
    March, J.G., Simon, H.A.: Organizations. Wiley, London (1958)Google Scholar
  24. 24.
    Rigby, P.C., Bird, C.: Convergent contemporary software peer review practices. In: Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering, pp. 202–212. ACM, Saint Petersburg (2013)Google Scholar
  25. 25.
    Rigby, P.C., Storey, M.A.: Understanding broadcast based peer review on open source software projects. In: Proceedings of the 33rd International Conference on Software Engineering, pp. 541–550. ACM (2011)Google Scholar
  26. 26.
    SmartBear: The state of code quality 2016. Accessed 13 June 2017
  27. 27.
    Sudman, S., Bradburn, N.M.: Asking Questions: A Practical Guide to Questionnaire Design. Jossey-Bass Publishers, San Francisco (1982)Google Scholar
  28. 28.
    Thongtanunam, P., McIntosh, S., Hassan, A.E., Iida, H.: Investigating code review practices in defective files: an empirical study of the QT system. In: Proceedings of the 12th Working Conference on Mining Software Repositories, MSR 2015 (2015)Google Scholar
  29. 29.
    Wang, J., Shih, P.C., Wu, Y., Carroll, J.M.: Comparative case studies of open source software peer review practices. Inf. Softw. Technol. 67, 1–12 (2015)CrossRefGoogle Scholar
  30. 30.
    Winter, M., Vosseberg, K., Spillner, A.: Umfrage 2016 “Softwaretest in Praxis und Forschung”. dpunkt.verlag (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.FG Software EngineeringLeibniz Universität HannoverHannoverGermany

Personalised recommendations