From Aristotle to Ringelmann: a large-scale analysis of team productivity and coordination in Open Source Software projects

Abstract

Complex software development projects rely on the contribution of teams of developers, who are required to collaborate and coordinate their efforts. The productivity of such development teams, i.e., how their size is related to the produced output, is an important consideration for project and schedule management as well as for cost estimation. The majority of studies in empirical software engineering suggest that - due to coordination overhead - teams of collaborating developers become less productive as they grow in size. This phenomenon is commonly paraphrased as Brooks’ law of software project management, which states that “adding manpower to a software project makes it later”. Outside software engineering, the non-additive scaling of productivity in teams is often referred to as the Ringelmann effect, which is studied extensively in social psychology and organizational theory. Conversely, a recent study suggested that in Open Source Software (OSS) projects, the productivity of developers increases as the team grows in size. Attributing it to collective synergetic effects, this surprising finding was linked to the Aristotelian quote that “the whole is more than the sum of its parts”. Using a data set of 58 OSS projects with more than 580,000 commits contributed by more than 30,000 developers, in this article we provide a large-scale analysis of the relation between size and productivity of software development teams. Our findings confirm the negative relation between team size and productivity previously suggested by empirical software engineering research, thus providing quantitative evidence for the presence of a strong Ringelmann effect. Using fine-grained data on the association between developers and source code files, we investigate possible explanations for the observed relations between team size and productivity. In particular, we take a network perspective on developer-code associations in software development teams and show that the magnitude of the decrease in productivity is likely to be related to the growth dynamics of co-editing networks which can be interpreted as a first-order approximation of coordination requirements.

Appendix A:

1.1 A1 Data Set

Table 3 summarises the 58 projects in our data set. For each project we show the project name and programming language, the time span of the data retrieved, as indicated by the times of the first and last retrieved commits, the total number of commits and the total number of unique developers during the analysed time span.

Table 3 Summary of the 58 OSS projects in our data set

1.2 A2 GitHub Data Set

For each project in our data set we queried the GitHub API with the query https://api.github.com/repos/%3Cowner%3E/%3Crepo%3E/commits%3Fpage=%3Cn%3E, where <owner> is a GitHub user account and <repo > is the name of a project repository belonging to this user. The query returns a paginated json list of the 30 most recent commits in the master branch of the project. By varying the parameter <n >, we control the pagination and can trace back the commit history until the very first commit.

Each element in the json list represents a commit with all Git-relevant information (see Section 3.1). More specifically, it contains the names and email addresses of both the author and the committer.¹¹ The author is the person who authored the code in the commit and the committer is the one with write permissions in the repository who merged the commit in the project code base. These two identities may not be the same when pull requests are considered, as the developers requesting the pull typically do not have write access. Since we quantify contributions in terms of amount of code written, we take the author email from the commit data as a unique identifier for individual developers. In cases where the author email is empty, we conservatively skip the commit.

The commit SHA contained in the json list can be used to execute a commitspecific query in the GitHub API of the form:

https://api.github.com/repos/%3Cowner%3E/%3Crepo%3E/commits/%3CSHA%3E

The result is again a json list which provides detailed information about the list and diffs of all files changed in the commit. We retrieve this additional information and use it to (i) quantify the precise contribution to the source code at the level of individual characters and (ii) construct the time-varying coordination networks of developers who have co-edited files (see Section 5.1).

1.2.1 A2.1 Merged Pull Requests

Upon merging a pull request, typically through the GitHub interface, the commit tree of the project is modified by including a special merge commit. The basics of this process is illustrated in Fig. 13.

Fig. 13

Simplified illustration of merging a pull request. A potential contributor forks the main branch of a project (light blue) into his/her own local repository (green). After some activity in both repositories a pull request is created and merged as indicated by the dashed arrow. This results in two commits - C5’ and C6 - in the main branch. The merge commit C6 (dark blue) has two parent links and should be excluded

In this example, a potential contributor forks the main branch after the second commit. Subsequent local changes are then made to the master branch and to the remote repository, represented by commits C4 and C5 respectively. After C5, the potential contributor creates a pull request asking for the changes in C5 to be incorporated in the main code base. Assuming the pull request is approved and no conflicts exist, C5 is merged by creating two commits - C5’ and C6. C5’ is almost identical to C5 in that it has the same author and committer fields as well as diffs.¹² C6 is a special merge commit that contains the same diffs as C5 and C5, but differs on the author and committer information. The author and committer in C6 are those of the maintainer who approved and merged the pull request, and not those of the developer who originally wrote the code in C5 and C5’. Thereby including commit C6 in the analysis would wrongly attribute the contained diff to the maintainer and inflate his/her contribution in terms of code written.

We deal with this problem by noticing that merge commits always have at least two parent pointers - one to the replicated commit from the forked repository, and one to the last commit in the main branch. In some cases when changes are merged from more than one remote branches, the merge commit will have a parent pointer to each of these remotes. Since the parent pointers are also available in our data set, we exclude all commits that have two or more parent pointers.¹³

An additional complication is that Git also allows integrating changes by socalled rebasing. Different from pull requests, which generate a merge commit, in rebasing all changes are applied on top of the last commit of the branch being rebased into. The result is a single commit with only one parent link that is added at the end of the rebased branch and that incorporates these changes. Since we cannot distinguish the developer who rebased from those who authored the changes, we exclude such commits from our analysis. Even though the parent pointer rule cannot be applied here, most well-structured projects contain indicative commit messages that can be used to this end. We exclude all commits with commit messages that contain any of the keywords merge pull request, merge remote-tracking, and merge branch, regardless of punctuation.

We note that all summary statistics regarding the number of commits in this paper (e.g. Table 3) are calculated after applying the above two exclusion methods.

1.3 A3 Model Fits for Project-Wise Scaling of Productivity

For each project in our data set, we estimated the model in (2) relating the team size s to the mean team-member contribution 〈c ^′〉. For a small number of those projects, the team size s varies in a rather narrow range, thus questioning logarithmic transformations of both the parameter s and 〈c ^′〉 in the linear model of (2).¹⁴ We thus additionally use a model variation with a logarithmic transformation of 〈c ^′〉, while keeping s linear, i.e.:

$$ \log\langle c^{\prime}\rangle=\hat{\beta}_{3}+\hat{\alpha}_{3}\cdot s $$

(6)

We denote this model as Log-Lin, while referring to the original model in which we perform a logarithmic transformation of both parameters as Log-Log.

For each project, we fit both models and select the one which yields the largest coefficient of determination r ² as the appropriate model for this project. The resulting project-dependent scaling coefficients are summarized in Table ??.

Table 4 Estimation of two linear models for Fig. 6 with single-commit developers removed

The result confirms that our finding of decreasing returns to scale at the aggregate level (Section 4.1) also holds for individual projects. Virtually all projects exhibit negative scaling of the mean team-member contribution with the team size, except for two projects for which no significant scaling coefficient could be determined. At any rate, the absence of significant positive coefficients for any of the projects allows us to conclude that there is no evidence for super-linear scaling in our data set.

1.4 A4 Effect of One-Time Contributors

In order to quantify the extent to which our results of team productivity may be influenced by contributors who committed to a project only once, we identified single-commit developers in all of the studied projects. Figure 14 shows the fraction of one-time contributors in all of the studied projects, validating the intuition that they comprise a sizable part of the development team.

Fig. 14

Fraction of commits submitted by one-time contributors, i.e., developers who never contributed a second commit

In order to ensure that our results about the scaling of productivity are not qualitatively affected by the large fraction of single-commit developers, we have additionally filtered the commit logs of all projects, filtering out the commits of all developers who committed only once. By this study, we focus on the contributions of a core team of that particularly rules out single-commit developers. Using this filtered commit log, we then recomputed all model fits in the paper. In Tables 5 and 6, we report the scaling exponents. We observe no qualitative changes regarding our observation of decreasing returns to scale. We additionally reanalyzed all individual projects, again filtering out all contributions by single-commit developers. We report on the project-wise scaling exponents in the bracketed values in Table ??, again not observing any qualitative changes of our results for individual projects.

Table 5 Estimation of two linear models for Fig. 6 with single-commit developers removed

Table 6 Estimation of two linear models for Fig. 7 with single-commit developers removed

1.5 A5 Inference Versus Prediction from Linear Models

In Section 4.1 we introduced two linear models¹⁵ as a means of quantifying the negative trends observed in Figs. 6 and 7. In particular we introduced

$$\begin{array}{@{}rcl@{}} \log\langle n\rangle = \beta_{0} + \alpha_{0} \cdot \log s + \epsilon_{0}\\ \log\langle c\rangle = \beta_{1} + \alpha_{1} \cdot \log s + \epsilon_{1}\\ \log\langle n^{\prime}\rangle = \beta_{2} + \alpha_{2} \cdot \log s + \epsilon_{2}\\ \log\langle c^{\prime}\rangle = \beta_{3} + \alpha_{3} \cdot \log s + \epsilon_{3} \end{array} $$

(7)

where 〈n〉 is the mean number of commits per active devel 〈n ^′〉 is the mean number of commits per team member, 〈c〉 is the mean commit contribution per active developer, 〈c ^′〉 is the mean contribution per team member and 𝜖 _0,1,2,3 denote the errors of the models.

We note that for these models to provide reliable predictions¹⁶ the following conditions must be met: (a) Var(𝜖 _0,1,2,3,4| logs) = σ ², for all s (homoskedasticity), (b) $\epsilon _{0,1,2,3} \sim \mathcal {N}$(0, σ ₂) (normality of the error distribution) and (c) E(𝜖 _0,1,2,3| logs)=0 (linear model is correct).

We test for homoskedasticity by running the Koenker studentised version of the Breusch-Pagan test (Koenker 1981). This test regresses the squared residuals on the predictor in (7) and uses the more widely applied Lagrange Multiplier (LM) statistics instead of the F-statistics. Although more sophisticated procedures, e.g. Whites test, would account for a non-linear relation between the residuals and the predictor, we find that the Breusch-Pagan test is sufficient to detect heteroskedasticity in our data. The consequence of violating the homoskedasticity assumption is that the estimated variance of the slopes α _0,1,2,3 will be biased, hence the statistics used to test hypotheses will be invalid. Thus, to account for the presence of heteroskedasticity, we use robust methods to calculate heteroskedasticity-consistent standard errors. More specifically, we use an MMtype robust regression estimator, as described in and provided by the R package robustbase.

As for normality of the errors 𝜖 _0,1,2,3, a violation of this assumption would render exact t and F statistics incorrect. However, our use of a robust MM estimator addresses possible non-normality of residuals, as it is resistant to the influence of outliers.

The last assumption pertains to the overall feasibility of the linear model. A common way to assess it is to plot the residuals from estimating (7) versus the fitted values, commonly known as a Tukey-Anscombe plot. A strong trend in the plot is evidence that the relationship between the dependent and independent variable is not captured well by a linear model. As a result, predicting the dependent variable from the calculated slope is likely to be unreliable, especially if the relationship between the variables is highly non-linear.

In Fig. 15 we show the Tukey-Anscombe plots for the four regression models in (7). While we cannot readily observe a prominent trend, we, nevertheless, see two qualitatively different regimes. Specifically the residuals in the lower ranges are close to zero, while they are relatively symmetrically distributed beyond this range. Looking at the line fits in Figs. 6 and 7 we see that the reason for this discrepancy are the outliers in the region of large team sizes, which fall close to the fitted regression lines. Therefore the residuals corresponding to these outliers will be close to zero. Investigating these specific data points reveals that they belong exclusively to the Specs project.

Fig. 15

Residuals versus fitted values for (7). The titles above each plot correspond to the respective scatterplots in Figs. 6 and 7

To actually quantify a possible trend in Fig. 15 we calculate the normalized mutual information (NMI) between the residuals and the fitted values.¹⁷ As expected the NMI is rather low - 0.04 (top-left), 0.02 (top-right), 0.04 (bottom-left) and 0.03 (bottom-right) - an indication that there is no pronounced systematic error in the linear model. However, even though the NMI values are low, we find that there are all statistically different from zero at p = 0.05.¹⁸

Therefore, despite the evidence against a systematic error in these linear models, assumption (c) cannot be technically satisfied. We, thus, conservatively avoid using the linear models for predictive means. Since the NMI values, however, are rather low, the regression models are sufficient for our purposes of simply quantifying the observed negative trends. We argue that the practical significance of such small effect sizes is negligible with respect to introducing strong systematic errors that could obscure a salient non-linear relationship. Effectively, we can only retroactively infer a significant negative relationship between team size and productivity, but cannot forecast team production given team size. We caution that such inference is also a subject to high variability, as indicated by the low r ² values (see Section 5.2), and is thus valid only on average.

Finally, an argument against the significance of the slopes in (7) is the relatively large sample size of N=13998. Known as the “p-value problem” (Lin et al. 2013), the issue pertains to applying small-sample statistical inference to large samples. Statistical inference is based on the notion that under a null hypothesis a parameter of interest equals a specific value, typically zero, which represents “no effect”. In our example, we are interested in estimating the slopes α _0,1,2,3 with an associated null hypothesis that sets them to zero. It is precisely this representation of the “no effect” by a particular number that becomes problematic with large samples. In large samples the standard error of the estimated parameter becomes so small that even tiny differences between the estimate and the null hypothesis become statistically significant. Hence, unless the estimated parameter is equal to the null hypothesis with an infinite precision, there is always a danger that the statistical significance we find is due to random fluctuations in the data. One way to alleviate the issue is to consider the size of the effect (as we did above) and assess whether the practical significance of the effect is important for the context at hand, even if it is significant in the strict statistical sense.

Another way is to demonstrate that the size and significance of the effect cannot arise by a random fluctuation. To this end we again resort to a bootstrap approach. For each scatterplot in Figs. 6 and 7, we generate 10, 000 bootstrap samples by shuffling the data points. We then estimate the regression models on each bootstrap sample and record the corresponding slope estimate $\hat {\alpha }_{0,1,2,3}$, regardless of its statistical significance.¹⁹ We find that the slopes of the 10, 000 bootstrapped regression models are restricted in the ranges [ −0.02, 0.02], [ −0.04, 0.04], [ −0.04, 0.03] and [ −0.04, 0.06] for $\hat {\alpha }_{0}$, $\hat {\alpha }_{1}$, $\hat {\alpha }_{2}$ and $\hat {\alpha }_{3}$, respectively. Comparing those ranges to the empirical slopes in Tables 1 and 2, we see that by eliminating the relationship between team size and productivity we cannot reproduce the strength of the negative trend found in the dataset. It is the information lost from the shuffling procedure that accounts for the statistical significance of α _0,1,2,3. Hence, it is safe to conclude that our analysis does not suffer from spuriously significant results introduced by large samples.

1.6 A6 Calculating Overlapping Source Code Regions

Our method of identifying overlapping source code changes between co-edits of the same file is based on the information in the chunk header of a diff between two versions of a committed file. Such a file diff shows only those portions of the file that were actually modified by a commit. In git parlance these portions are known as chunks. Each of these chunks is prepended by one line of header information enclosed between @@…@@. The header indicates the lines which were modified by a given commit to this file. Therefore, from all chunk headers within a file diff we can obtain the line ranges affected by the commit and eventually calculate the overlapping source code regions between two different commits to the same file.

As a concrete example, assume a productivity time window of 7 days in which the file foo.txt was modified first by developer A and then by developer B in commits C _A and C _B , respectively. The diff of foo.txt in commit C _A may contain the following chunk header:

$$\textsf{@@ -10,15 \qquad+10,12 @@} $$

The content of the header is split in two parts identified by “ −”and “+”: −10,15 and +10,12. The two pairs of numbers indicate the line ranges, outside which the two versions (before and after C _A ) of foo.txt are identical. More specifically, -10,15 means that starting from line 10, C _A made changes to the following 15 lines, i.e., it affected the line range [10 - 25]. What the result of these changes was is given in the second part of the header. +10,12 indicates that starting from line 10 in the new state of the file, the following 12 lines are different compared to the [10 - 25] line range. Beyond these 12 lines, the old and the new state of foo.txt are identical, provided there are no more chunks in the file diff. Therefore, the line range [10 - 25] in the old state of foo.txt and the line range [10 - 22] in the new state after C _A , are the only differences introduced by the commit. This could be caused for example by the removal of three lines from the line range [10 - 25], together with other modifications in the same range.

Since C _A comes prior to C _B in our example, we associate the second part of the chunk header, i.e., line range [10 - 25], to C _A as it represents the state of foo.txt after the changes from C _A were applied and before those from C _B . Now assume that the diff of foo.txt in C _B has only one chunk with the following header:

$$\textsf{@@ \(-\)10,30\qquad +10,40 @@} $$

In other words, lines [10 - 40] from the old state of foo.txt were modified by C _B , and the changes are reflected in lines [10 - 50] in the new state of foo.txt after C _B .²⁰ Note that, lines [10 - 40] represent the state of foo.txt after C _A , but before C _B . Therefore to compute the overlapping source code ranges between C _B and C _A , we need to compare the line ranges [10 - 40] and [10 - 25] and calculate the overlap. In this case, the overlap is 15 lines, which is the weight we attribute to the coordination link from developer B to A in this particularly simple example. The procedure described above is applied to all pairs of commits by different developers which have edited a common file within a given productivity time window of 7 days. Processing the chunk information in the above way thus allows us to extract linebased, weighted and directed co-editing networks which capture the association between developers and source code regions.