Using knowledge units of programming languages to recommend reviewers for pull requests: an empirical study

Abstract

Determining the right code reviewer for a given code change requires understanding the characteristics of the changed code, identifying the skills of each potential reviewer (expertise profile), and finding a good match between the two. To facilitate this task, we design a code reviewer recommender that operates on the knowledge units (KUs) of a programming language. We define a KU as a cohesive set of key capabilities that are offered by one or more building blocks of a given programming language. We operationalize our KUs using certification exams for the Java programming language. We detect KUs from 10 actively maintained Java projects from GitHub, spanning 290K commits and 65K pull requests (PRs). We generate developer expertise profiles based on the detected KUs. We use these KU-based expertise profiles to build a code reviewer recommender (KUREC). We compare KUREC’s performance to that of seven baseline recommenders. KUREC ranked first along with the top-performing baseline recommender (RF) in a Scott-Knott ESD analysis of recommendation accuracy (the top-5 accuracy of KUREC is 0.84 (median) and the MAP@5 is 0.51 (median)). From a practical standpoint, we highlight that KUREC’s performance is more stable (lower interquartile range) than that of RF, thus making it more consistent and potentially more trustworthy. We also design three new recommenders by combining KUREC with our baseline recommenders. These new combined recommenders outperform both KUREC and the individual baselines. Finally, we evaluate how reasonable the recommendations from KUREC and the combined recommenders are when those deviate from the ground truth. We observe that KUREC is the recommender with the highest percentage of reasonable recommendations (63.4%). Overall we conclude that KUREC and one of the combined recommenders (e.g., AD_HYBRID) are overall superior to the baseline recommenders that we studied. Future work in the area should thus (i) consider KU-based recommenders as baselines and (ii) experiment with combined recommenders.

Appendices

Appendix A Java Certification Exams and Knowledge Units

1.1 A.1 Java SE 8 Programmer I Exam

Table 6 lists the topics and subtopics covered in the Java SE 8 Programmer I Exam as per Oracle’s official webpage.

Table 6 Topics and subtopics from the Java SE 8 Programmer I Exam

1.2 A.2 Java SE 8 Programmer II Exam

Table 7 lists the topics and subtopics covered in the Java SE 8 Programmer II Exam as per Oracle’s official webpage.

Table 7 Topics and subtopics from the Java SE 8 Programmer II Exam

1.3 A.3 Java EE Developer Exam

Table 8 lists the topics and subtopics covered in the Java EE Developer Exam as per Oracle’s official webpage.

Table 8 Topics and subtopics from the Java EE Developer Exam

1.4 A.4 Mapping Process From Certification Exams to Knowledge Units (KUs)

In our mapping process, the first two authors work collaboratively to minimize subjectivity bias. The authors conducted the entire mapping process together and engaged in extensive discussions regarding the handling of corner cases. The collaboration entails a shared effort between the first two authors to conduct the mapping process together. This involves jointly reviewing the topics and subtopics of exam certifications, mapping the topics to KUs, and discussing how to handle corner cases. It also includes ongoing communication to ensure a consistent and coherent approach to data analysis. It is important to note that the authors did not calculate interrater agreement for this mapping process. Our decision to forgo interrater agreement analysis was based on our collaboration approach, which aimed to achieve a shared and consistent understanding of the data (e.g., topics and subtopics of Java certification exams).

We detail our process for mapping from certification exams to KUs. To map from certification exams to KUs, we follow three steps. First, we exclude subtopics (from either exam) that are unsuitable for our study. Next, we rearrange subtopics (within exams and across exams) such that subtopics lie in the most specific topic possible. Finally, we convert each topic into a KU (one-to-one mapping) and interpret its subtopics as the key capabilities of the KU. Tables 6 and 7 show the topics and subtopics of the Java SE 8 certification exam I (E1) and the Java SE 8 certification exam II (E2), respectively. In the following, we explain the mapping process in detail (we use the notation [Ei, Tj, Sk] to refer to a subtopic Sk from topic Tj of exam Ei).

Exclusion of Subtopics We only considered the subtopics that can be automatically detected from the source code using static analysis. For instance, the subtopic “Use autoclose resources with a try-with-resources statement” can be detected by searching for “try-with-resources” code blocks using an appropriate Java parser. Our rationale is to ensure that KUs are computable for any Java system, irrespective of how those are built or used by end-users. In this vein, subtopics that are inherently conceptual or that pose great challenges to be detected in practice (e.g., involve problems that are still under investigation by the SE research community) are excluded from our operationalization. We discuss all individual cases below.

The subtopic [E1, T1, S5] under “[E1, T1] Java Basics” is about comparing and contrasting the features and components of Java, such as: platform independence, object orientation, encapsulation, etc. This topic is thus more of a concept, which can not be operationalized and extracted automatically. After discussion, the first two authors concluded that none of the subtopics of “[E1, T1] Java Basics” can be extracted automatically (e.g., [E1, T1, S5] comparing and contrasting the features and components of Java such as: platform independence, object orientation, encapsulation, etc). As a consequence, we did not include “[E1, T1] Java Basics” in our study.

Some of the subtopics of topics are about the understanding of a concept which can not be operationalized and extracted automatically. For example, the subtopic “[E1, T7, S1] Describe inheritance and its benefits” is about understanding the concept of inheritance. The subtopic “[E1, T8, S3] Describe the advantages of exception handling” is about understanding the concept of exception handling. The authors did not include such cases of subtopics as capabilities of KUs. We highlight all these cases in Table 9.

The authors decided not to include the topic “[E2, T19] Object-Oriented Design Principles” in our study. The topic “[E2, T19] Object-Oriented Design Principles” is about the implementation and detection of different design patterns (e.g., singleton design pattern and factory design pattern). Detection of such design patterns is a non-trivial task and is itself an active research area in the software engineering community (Tsantalis et al. 2006; Xiong and Li 2019). Hence, the authors did not include the topic “[E2, T19] Object-Oriented Design Principles” in this study.

The authors did not include the topic “[E3, T30] Create Java Web Application using JSPs”. To create Java web application using JSPs, developers need to create a JSP server pages (e.g., files with “.jsp” extension) containing the JSP tags which are different from Java files. In this study, we only focus on studying the Java files (e.g., files with “.java” extension). Therefore, the authors decided not to include the topic “[E3, T30]” in this study.

Rearrangement of Subtopics Certain subtopics from exam E1 are conceptually related to topics of exam E2 and vice-versa. In those cases, we prioritize the topic that is more specific and move the subtopic in question to that topic. For example, “[E2, T10] Java Class Design" contains the subtopic “[E2, T10, S2] Implement inheritance including visibility modifiers and composition". However, such a subtopic would also fit under “[E1, T7] Working with Inheritance." Since the latter topic is more specific than the former, we move [E2, T10, S2] into [E1, T7]. We also observed rare cases in which it made sense to perform subtopic moving within an exam (as opposed to cross exams). In the following, we describe all subtopic rearrangements in detail.

The subtopics of “[E1, T9] Working with selected classes from the Java API" were moved into several topics of E2:

• (1) The subtopics “[E1, T9, S1] Manipulate data using the StringBuilder class and its methods” and “[E1, T9, S2] Create and manipulate strings” were moved into “[E2, T10] String Processing”.

• (2) The subtopic “[E1, T9, S3] Create and manipulate calendar data using classes from java.time.LocalDateTime, java.time.LocalDate,java.time.LocalTime, java.time.format.DateTimeFormatter, java.time.Period.” was moved into “[E2, T16] Use Java SE 8 Date/Time API”.

• (3) The subtopic “[E1, T9, S4] Declare and use an ArrayList of a given type” was moved into “[E2, T12] Generics and Collection”.

• (4) The subtopic “[E1, T9, S5] Write a simple Lambda expression that consumes a Lambda Predicate expression” was moved into “[E2, T13] Lambda Built-in Functional Interfaces”.

The subtopics “[E1, T8, S2] Create a try-catch block” and “[E1, T8, S4] Create and invoke a method that throws an exception” were moved from “[E1.T8] Handling Exception” into “[E2.T15] Exceptions and Assertions”.

All subtopics from the topic “[E2, T10] Java Class Design” were moved into other topics of E1 and E2 as follows:

• (1) The subtopics “[E2, T10, S1] Implement encapsulation”, “[E2, T10, S4] Override hasCode, equals, and toString methods from Object class”, and “[E2, T10, S6] Develop code that uses the static keyword on initialize blocks, variables, and methods” were moved into “[E1, T6] Working with Methods and Encapsulation”.

• (2) The subtopics “[E2, T10, S2] Implement inheritance including visibility modifiers and composition” and “[E2, T10, S3] Implement polymorphism” were moved into “[E1, T7] Working with Inheritance”.

• (3) The subtopic “[E2, T10, S5] Create and use singleton classes and immutable classes” was moved into “[E2, T11] Advanced Class Design”. Three subtopics of the topic “[E2, T12] Generics and Collection KU” were moved into “[E2, T14] Java Stream API”. The subtopics (1) “[E2, T12, S5] Iterate Collection using forEach methods of Streams”, (2) “[E2, T12, S7] Filter a collection by using Stream filter API with lambda expressions”, and (3) “[E2, T12, S8] Use method references with streams” were moved into “[E2, T14] Java Stream API”.

One-to-one Mappings After performing the exclusions and subtopic rearrangements, we created a one-to-one mapping between topics and KUs. For example, the topic “[E1, T5] Using Loop Constructs” is mapped to the “[K4] Loop KU”. All of the subtopics in [E1, T5] are interpreted as key capabilities associated with [K4]. For instance, the subtopic “[E1, T5, S1] Create and use while loops” is considered a key capability associated with the Loop KU. As a result of our mapping process, we identified 28 KUs, which are shown in Table 9.

Table 9 The mapping from certification exams to Knowledge Unit (KUs)

Table 10 Knowledge Units derived from the Java SE 8 Programmer I Exam, Java SE 8 Programmer II Exam, and Java EE Developer Exam

B Sensitivity Analysis of Choosing a Threshold for Identifying Reasonable Recommendations

The developer who has more prior knowledge about the changed files of the PR is more capable of reviewing that PR.

Prior study also uses a threshold to identify a developer as a reasonable reviewer (Al-Zubaidi et al. 2020). Al-Zubaidi et al. (2020) identify a developer as a reasonable reviewer if the developer previously reviewed at least one of the changed files of the PR in question. We believe that such a threshold could be very low resulting that the developer may not have recent knowledge of the majority of the remaining changed files of the PR. We need to set a threshold that is not very low or very high. A low threshold value means that developers have minimum prior knowledge about the changed files. On the other hand, a high threshold value may not satisfy our objective of distributing workload to other capable reviewers. To identify an appropriate threshold we perform a sensitivity analysis. We calculate the percentage of reasonable recommendations across recommenders using three different threshold values such as 30, 50 and 80 (see Table 11). We observe that our initial assumptions hold for all of these threshold values: KUREC and the combined recommender (e.g., AD_HYBRID) achieve the highest percentage of reasonable recommendation. Finally, we select the threshold=50 as an appropriate threshold value for our study since the value is not very low or not very high.

Table 11 The percentage of reasonable reviewers across recommenders using different thresholds