An ensemble-based model for predicting agile software development effort

Abstract

To support agile software development projects, an array of tools and systems is available to plan, design, track, and manage the development process. In this paper, we explore a critical aspect of agile development i.e., effort prediction, that cuts across these tools and agile project teams. Accurate effort prediction can improve the planning of a sprint by enabling optimal assignments of both stories and developers. We develop a model for story-effort prediction using variables that are readily available when a story is created. We use seven predictive algorithms to predict a story’s effort. Interestingly, none of the predictive algorithms consistently outperforms others in predicting story effort across our test data of 423 stories. We develop an ensemble-based method based on our model for predicting story effort. We conduct computational experiments to show that our ensemble-based approach performs better in comparison to other ensemble-based benchmarking approaches. We then demonstrate the practical application of our predictive model and our ensemble-based approach by optimizing sprint planning for two projects from our dataset using an optimization model.

Appendices

Appendix A

The K-Nearest Neighbor (KNN) algorithm uses k closest data points in the training data (determined by some distance metric —in our case Euclidean), where k is provided by the user, to determine its prediction. Each neighbor of a target point can equally influence the prediction (uniform weight) or inverse of the distance from the target point (distance-based weight). For our experiments, the parameter optimized was the number of neighbors (k). Exhaustive searches from 2 neighbors to the maximum possible in the training dataset were used; the best number of neighbors to minimize prediction errors is 14. Further, we used a distance-based measure such that closer points had higher influence than those that were farther out.

Decision trees (DT) represent a set of hierarchical decisions on the feature variables of the training data. A decision at a particular node, called a split criterion, is conditional on one or more feature variables. Different criteria can be used to measure the quality of the split. In our case, mean absolute error was used as the measurement criterion. As the depth of a tree increases, the tree is over fitted to the training data i.e., all instances can be classified by a dedicated leaf node of their own. However, such trees suffer when exposed to test datasets. To balance overfitting, optimization for the maximum depth of the tree was found to be 2. We considered depth value from 1 to 199.

Ridge regression (RR), in addition to minimizing the residual sum of squares (RSS), aims to minimize the shrinkage penalty using a tuning parameter. As the tuning parameter approaches zero, the ridge regression produces a least squares model. As the tuning parameter increases, the flexibility of the ridge regression fit decreases, leading to decreased variances but increased bias. The optimal value of the tuning parameter was found to be 3.51. The tuning parameter value varied from 0.0001 to 50 with increments of 0.001.

A Bayesian network (BN) learns the joint probability distribution of the target variable using a causal model that provides the prior and posterior probabilities of variables. New information can be incorporated into the model using belief updating procedures. Bayesian networks are popular in machine learning and software effort prediction literature due to their accuracy (Hearty et al. 2009). We implemented the procedure in Pendharkar et al. (2005) that provided point predictions for a target story.

Support Vector Machine (SVM) is a generalization of maximal margin classifier that identifies a separating hyperplane (p-1 dimensional flat space in a p dimensional space) that classifies the one-dimensional space. However, this approach is limited by cases where the perfect separating hyperplane is not available. Support Vector Classifiers address this problem by identifying soft hyperplanes that almost separate the classes. Data points that lie on the hyperplane are known as support vectors. These points determine the support vector classifier performance. Tolerance for points on the wrong side of the hyperplane can be tuned by a penalty parameter C. A greater value of C will increase the tolerance for data points on the wrong side. The hyperplane can be linear or nonlinear. Support Vector Regression (SVR), a nonparametric method, uses kernel functions to identify the decision boundary. Kernel functions can be linear, polynomial, radial, or sigmoid, among others. In our experiments, the penalty parameter C was identified to be 1.9 using a Radial Basis Function (rbf) kernel whose width was identified to 0.3. We used a multi-level grid search to tune parameters. The possible values for penalty parameters were varied from 0.01 to 10 whereas the values for kernel widths varied from 0.1 to 10.

In Artificial Neural Network (ANN), derived features are created from non-linear combinations of input variables (input nodes). The target variable to be predicted, in turn, is modeled as a linear function of the derived features. The units computing the derived features are known as hidden nodes since they are not visible to the user. Hidden nodes transform the nodes’ inputs using a weighted linear summation. A regularization parameter is used to limit overfitting of the neural network to the training data. In our experiments, the regularization parameter was determined to be 0.000081. The value of regularization parameter varied from 1.00E-06 to 1 with increments of 1.00E-05.

In Random Forest (RF), a number of decision trees are applied to subsamples of the training dataset. Every time these trees are built, a different set of predictors is chosen as split candidates. This produces decorrelated trees which reduce the overall variance. For our experiments, the random forest was optimized to 5 trees in the forest with 4 features selected to determine the best split. Possible values for the number of trees in the forest and features were varied from 1 to 20 each, with all possible combinations considered to identify optimal parameter values.

Extra Trees (ET), similar to RF, fit multiple randomized decision trees on subsamples of the dataset. However, variables are chosen from the entire dataset rather than bootstrapping and at random. The number of trees was optimized to 5 trees and 4 features selected to determine the best split. Possible values for the number of trees and features varied from 1 to 20 each, with all possible combinations considered to identify optimal parameter values.

Adaptive Boost (AB) works by incrementally focusing on cases that are difficult to predict. A base estimator (in our case, decision tree) is identified by fitting to the original dataset. Incrementally, adjusted estimators are fitted to the dataset such that weight is increased for wrongfully classified data points. In our experiments, the boosting was terminated after two estimators were fit to the data. Possible values considered were from one to 30 estimators.

Gradient Boosting (GB) optimizes least squares loss function by adding weak learners (decision trees) to an additive model. As new trees are added at each boosting stage, existing trees are not changed. The optimal number of boosting stages was identified as 28. The possible range of stages considered was 1-60.

Stacking (ST) is an ensemble approach which aggregates individual 1^st-level algorithms with a 2^nd- level algorithm (often known as the meta-regressor). Individual first-order algorithms provide predictions which form the input to the meta-regressor. Five algorithms selected in our experiments formed the first order algorithms with respective hyperparameters identified in the experiments. We used Support Vector Machine with “rbf” kernel as the meta-regressor.

For an extensive discussion on each of these algorithms, readers are referred to (Hastie et al. 2008), (James et al. 2015), (Aggarwal 2015), and (Shmueli et al. 2016).

Appendix B

Tables in this appendix (Tables 24, 25 and 26) provide effect sizes with Cliff’s delta, 95% confidence interval, and the Vargha and Delaney (2000) ($\hat {A_{12}}$) statistic (VDA), which compares the probability of yielding lower absolute error for individual predictive algorithms. For each effect size computation, we consider the error values across all test dataset (N = 423). Reversing the order of algorithms will provide the same Cliff’s delta multiplied by -1. Similarly, confidence interval values will reverse with different signs. The VDA statistic for the reverse order of the algorithms is VDA_Reverse = |1 − VDA_{originalorder}|. For example, the effect size of RR and SVM is 0.10752, confidence interval is [0.02946, 0.18394], and VDA statistic is 0.553759. Statistically significant pairs are highlighted.

Table 24 Effect Sizes for Individual Predictive Algorithms (MAE)

Table 25 Effect Sizes for Individual Predictive Algorithms (MBE)

Table 26 Effect Sizes for Individual Predictive Algorithms (RMSE)

Appendix C

Tables in this appendix (Tables 27, 28 and 29) provide effect sizes with Cliff’s delta, 95% confidence interval, and the Vargha and Delaney (2000) ($\hat {A_{12}}$) statistic (VDA) for ensemble algorithms. For each effect size computation, we consider the error values across all test dataset (N = 423). Statistically significant pairs are highlighted.

Table 27 Effect Sizes for Ensemble Algorithms (MAE)

Table 28 Effect Sizes for Ensemble Algorithms (MBE)

Table 29 Effect Sizes for Ensemble Algorithms (RMSE)