Abstract
Regression conformal prediction produces prediction intervals that are valid, i.e., the probability of excluding the correct target value is bounded by a predefined confidence level. The most important criterion when comparing conformal regressors is efficiency; the prediction intervals should be as tight (informative) as possible. In this study, the use of random forests as the underlying model for regression conformal prediction is investigated and compared to existing stateoftheart techniques, which are based on neural networks and knearest neighbors. In addition to their robust predictive performance, random forests allow for determining the size of the prediction intervals by using outofbag estimates instead of requiring a separate calibration set. An extensive empirical investigation, using 33 publicly available data sets, was undertaken to compare the use of random forests to existing stateoftheart conformal predictors. The results show that the suggested approach, on almost all confidence levels and using both standard and normalized nonconformity functions, produced significantly more efficient conformal predictors than the existing alternatives.
Introduction
One of the main motivations for using the conformal prediction (CP) framework (Vovk et al. 2006) is that it provides guarantees for the prediction error; the probability of making incorrect predictions is bounded by a userprovided confidence threshold. In contrast to other learning frameworks that provide similar types of guarantees, e.g., PAC learning (Valiant 1984), CP makes it possible to assess the uncertainty of each single prediction. Hence, rather than just providing a bound on the prediction error for the entire distribution, CP allows for providing different bounds for different instances, something which may be very valuable in many practical applications. For example, knowing that the error of a certain model for predicting the stock price is bounded by 100 dollars with \(95~\%\) probability, is not as informative as knowing that for the specific stock we are interested in, the prediction error is, in fact, bounded by ten dollars, i.e., this particular stock is actually easier to predict than the average one. Similarly, in the medical domain, it is of course important to be able to assess the confidence in predictions related to individual patients instead of groups of patients.
CP employs some underlying predictive model, which may have been generated by any standard learning algorithm, for obtaining prediction regions rather than single point predictions. A prediction region corresponds to a set of class labels in a classification context, and to an interval in a regression context. A prediction error, in this framework, occurs when the correct label of a (test) instance is not included in the prediction region. The guarantee given by the conformal prediction framework, under the standard i.i.d. assumption, is that the probability of making an error is bounded by a predetermined confidence level. This means that the number of errors can be controlled, typically reducing the error level by increasing the sizes of the prediction regions or vice versa. There is an obvious resemblance of the conformal prediction framework to standard statistical hypothesis testing, where the type I and II errors are controlled by the choice of significance level.
Since all conformal predictors are valid, i.e., the probability of excluding the correct label is bounded by the confidence level, the main criterion when comparing different conformal predictors is their efficiency, i.e., the sizes of output prediction regions. Efficiency is, for classification, often measured as the (average) number of labels present in the prediction sets, and for regression as the (average) size of the intervals.
CP relies on realvalued functions, called nonconformity functions, that provide estimates for how different a new example is from a set of old examples. In a predictive modeling scenario, nonconformity functions use the underlying model to determine how strange the relationship between the feature vector (the input) and an output value for a certain new instance is, compared to a set of previously observed instances.
It is possible to design many different nonconformity functions for a specific predictive model, and each of them will define a different conformal predictor. All of these conformal predictors will be valid, but there may be significant differences in terms of efficiency. In the extreme case, even a function that returns the same nonconformity score for all examples will be valid, but the prediction regions will be very wide.
CP was originally introduced as a transductive approach for support vector machines (Gammerman et al. 1998). Transductive CP requires learning a new model for each new test instance to be predicted, which of course may be computationally prohibitive. For this reason, inductive conformal prediction (ICP) was suggested (Vovk et al. 2006). In ICP, which is the focus of this study, only one model is induced from the training data and that model is then used for predicting all test instances. In ICP, however, the calculation of the nonconformity scores requires a separate data set (called the calibration set) that was not used by the algorithm when learning the model. Consequently, it becomes very important how the training data is divided into the proper training set and the calibration set; using too few calibration instances will result in imprecise confidence values, while too few proper training instances may lead to weaker underlying models.
Looking specifically at ICP regression, there are very few published papers providing a systematic evaluation of different underlying models and nonconformity functions. As a matter of fact, until now, most studies focus on one specific underlying model, and use a very limited number of data sets, making them serve mainly as proofsofconcept; see e.g., Papadopoulos et al. (2002); Papadopoulos and Haralambous (2011). With this in mind, there is an apparent need for larger studies, explicitly evaluating techniques for producing efficient conformal predictors. Such studies should preferably explore various learning algorithms and use a sufficiently large number of data sets to allow for statistical inference, thus making it possible to establish best practices. In this paper, we compare using random forests (Breiman 2001) as the underlying model for conformal prediction regression to existing stateoftheart conformal regressors, which are based on artificial neural networks (ANN) and knearest neighbors (kNN). We investigate a number of nonconformity functions, and we specifically examine the option to use outofbag estimates for the necessary calibration.
In summary, the main contributions of this paper are:

a novel method for regression conformal prediction, which utilizes random forests together with a nonconformity function that exploits outofbag examples as a calibration set;

the first largescale empirical investigation of methods for regression conformal prediction, which include stateoftheart learning algorithms and multiple nonconformity functions that are evaluated on a large number of datasets;

significant findings concerning the relative efficiency of different conformal predictors, which provide new evidence for what may be considered best practices for regression conformal prediction.
In the next section, we formalize the conformal prediction framework and discuss related work. In Sect. 3, we describe the proposed approach for regression conformal prediction using random forests as well as competing stateoftheart approaches. The setup for, and the results from, the empirical investigation are presented in Sect. 4. Finally, we summarize the main conclusions from the study and outline directions for future work in Sect. 5.
Background
In this section, we first provide a formalization of inductive conformal prediction, which is the theoretical foundation for this paper. We then briefly discuss its relation to alternative frameworks and summarize the main related previous studies upon which our study builds.
Inductive conformal prediction
An inductive conformal classifier or regressor only needs to be trained once, using the following scheme:

1.
Divide the training set \(Z = \{(x_1, y_1),\ldots , (x_l, y_l)\}\) into two disjoint subsets \(Z^t\) (a proper training set) and \(Z^c\) (a calibration set):

\(Z^t = \{(x_1, y_1),\ldots , (x_m, y_m)\}\)

\(Z^c = \{(x_{m+1}, y_{m+1}),\ldots , (x_l, y_l)\}\)


2.
Train the underlying model \(h_Z\) using \(Z^t\).

3.
For each calibration instance \((x_i, y_i) \in Z^c\):

let \(h_Z\) predict the output value for \(x_i\) so that \(\hat{y}_i = h_Z(x_i)\) and

calculate the nonconformity score \(\alpha _i\) using the nonconformity function.

For a novel (test) instance, the input pattern \(x_j\) is supplied to the underlying model, resulting in a prediction \(\hat{y}_j\). Then a nonconformity score \(\alpha _j^{\tilde{y}}\) is produced for every tentative target value \(\tilde{y}\). The \(p\)value of each tentative target \(\tilde{y}\) is then calculated by comparing \(\alpha _j^{\tilde{y}}\) to the nonconformity scores of the calibration set \(S = \{\alpha _{1},\ldots , \alpha _{q}\}\):
If \(p(\tilde{y}) < \delta \), the probability for \(\tilde{y}\) being the true target for \(x_j\) is smaller than \(\delta \), i.e., \(\tilde{y}\) can be excluded from the prediction region, at that confidence level. In classification, \(\tilde{y}\) represents a possible class label, and all possible labels are tested one at a time. In regression, we cannot consider every possible output value in that manner, so a conformal regressor will instead directly establish the prediction interval, for each test instance, given the confidence level. In regression, the nonconformity function is most often simply the absolute error, see e.g., Papadopoulos and Haralambous (2011); Papadopoulos et al. (2002, 2011):
Then, given a significance level \(\delta \) and a set of calibration scores \(S = \{\alpha _{1},\ldots , \alpha _{q}\}\), we locate the smallest \(\alpha _{s(\delta )} \in S\) that satisfies the equation
Since it is not possible to consider each \(\tilde{y}\) in regression, it is also not possible to calculate the nonconformity scores \(\alpha _j^{\tilde{y}}\) for the test instance \(x_j\). Instead, \(\alpha _{s(\delta )}\) forms a probabilistic bound for the nonconformity scores at significance level \(\delta \); that is, with probability \(1\delta \), the nonconformity of \(x_j\) will be at most \(\alpha _{s(\delta )}\). Thus, at significance \(\delta \), we can reject any label for which \(\alpha _j^{\tilde{y}} > \alpha _{s(\delta )}\), and must conversely include all labels for which \(\alpha _j^{\tilde{y}} \le \alpha _{s(\delta )}\). Using (2), \(\alpha _j^{\tilde{y}} = \alpha _{s(\delta )}\) exactly when \(y  \tilde{y} = \alpha _{s(\delta )}\), hence, by formulating the prediction region as
where \(s(\delta )\) is found from (3) above, \(\hat{Y}_j^{\delta }\) will cover the true output \(y_j\) with probability \(1\delta \). It must be noted that when using (2) and (4), the conformal regressor will, for any specific significance level \(\delta \), always produce prediction intervals of the same size for every \(x_j\); i.e., it does not consider the difficulty of a certain instance \(x_j\) in order to provide as informative predictions as possible, which often is a key motivation for using conformal prediction in the first place. It is, however, possible to employ normalized nonconformity functions, where the absolute error is scaled using the expected accuracy of the underlying model; see e.g., Papadopoulos and Haralambous (2011); Papadopoulos et al. (2011). The motivation for this, from a conformal prediction standpoint, is that if two instances have identical conformity scores using (2), but the first is expected to be more accurate than the second, then the second is actually stranger (more nonconforming) than the first. Using a normalized nonconformity function, the resulting prediction intervals will be smaller for instances that are deemed “easy” and larger for “harder” instances. When using a normalized nonconformity function, nonconformity scores are calculated using:
where \(\sigma _i\) is an estimate of the accuracy of the underlying model for \(\hat{y}_i\). Naturally, there are several ways to estimate the accuracy; one suggestion is to train another model for predicting the errors; see e.g., Papadopoulos and Haralambous (2011). Other approaches use properties of the underlying model; see e.g., Papadopoulos et al. (2011). With normalized nonconformity functions, the prediction interval for \(\hat{Y}_j^\delta \) is:
where \(\sigma _j\) is an estimate of the accuracy of the underlying model, for that instance.
Related work
As mentioned in the introduction, there are other machine learning frameworks that provide some sort of guarantee of the prediction error. Specifically, PAClearning (Valiant 1984) will provide upper bounds on the probability of its error with respect to some confidence level. PAC theory only assumes that the instances are generated independently by some completely unknown distribution, but for the resulting bounds to be interesting in practice, the data set must be quite clean. Unfortunately, this is rarely the case for realworld data, which will lead to very loose bounds, see e.g., Nouretdinov et al. (2001), where the crudeness of PAC theory is demonstrated. In addition, the PAC bounds are for the overall error and not for individual predictions. The Bayesian framework can, on the other hand, be used to complement individual predictions with probabilistic measures of their quality. These measures are, however, based on some a priori assumption about the underlying distribution. When the assumed prior is violated, there is no guarantee that the resulting intervals produced by the Bayesian methods actually contain the true target as often as indicated by the confidence level, i.e., the resulting predictions are not valid. In Papadopoulos et al. (2011), CP is compared to the popular Bayesian method called Gaussian Processes (GP) (Rasmussen and Christopher 2005). The results show that when the (artificial) data set satisfied the GP prior, the intervals produced by GPregression were valid, and slightly tighter than the corresponding intervals produced by CP. On a number of realworld data sets, however, the predictive regions produced by GPregression were no longer valid, i.e., they may become misleading when the correct prior is not known.
The CP framework has been applied to classification using several popular learning algorithms, such as ANNs (Papadopoulos 2008), kNN (Nguyen and Luo 2012), SVMs (Devetyarov and Nouretdinov 2010; Makili et al. 2011), decision trees (Johansson et al. 2013a), random forests (Bhattacharyya 2011; Devetyarov and Nouretdinov 2010) and evolutionary algorithms (Johansson et al. 2013b; Lambrou et al. 2011). Although we in this study consider regression tasks, there is some overlap with previous studies on classification when it comes to design choices. Specifically, in Johansson et al. (2013a), the underlying learning algorithm is also decision trees. However, in the previous study, the focus was on how properties of the algorithm, e.g. split evaluation metric, pruning and the smoothening function, affect the efficiency of classification trees. In this paper, we instead study forests of regression trees, and investigate both standard and normalized nonconformity functions in this context. Moreover, the use of outofbag estimates for the calibration was suggested for random forests in Devetyarov and Nouretdinov (2010), and was also used for bagged ANNs in Löfström et al. (2013). None of these studies, however, evaluate efficiency in a systematic way while considering different underlying models and nonconformity functions. In particular, no normalized nonconformity functions were evaluated in these studies.
There are also a number of studies on conformal prediction for regression, using, for instance, ridge regression (Papadopoulos et al. 2002) and ANNs (Papadopoulos and Haralambous 2010). Two interesting and fairly recent studies do in fact evaluate normalized nonconformity functions for ANNs (Papadopoulos and Haralambous 2011) and kNearest Neighbors (Papadopoulos et al. 2011). Unfortunately, both studies use very few data sets, thus precluding statistical analysis. Despite these shortcomings, the suggested approaches must be regarded as stateoftheart for ICP regression, making them natural benchmarks to compare our proposed methods against.
Conformal prediction has also been successfully used in a number of applications where confidence in the predictions is of concern, including prediction of space weather parameters (Papadopoulos and Haralambous 2011), estimation of software project effort (Papadopoulos et al. 2009b), early diagnostics of ovarian and breast cancers (Devetyarov et al. 2012) and diagnosis of acute abdominal pain (Papadopoulos et al. 2009a).
Methods
In this section, we first describe the proposed method for utilizing random forests for regression conformal prediction, including some variants, and then describe the competing stateoftheart approaches.
Regression conformal prediction using random forests
A random forest (Breiman 2001) is a set of decision trees (Breiman et al. 1984; Quinlan 1986), where each tree is generated in a specific way to introduce diversity among the trees, and where predictions of the forest are formed by voting. A decision tree is a treestructured (directed, acyclic and connected) graph, where each internal (nonleaf) node is labeled with a test on some attribute, with one arc leading to a unique (child) node for each possible outcome of the test, and where the leaf nodes of the tree are labeled with values to be predicted. If the predicted values are (categorical) class labels, the decision tree is called a classification tree, while if the predicted values are numeric, the tree is called a regression tree. When using a decision tree to predict a value for an example, starting at the root node, the test at the current internal node is performed and the arc corresponding to the outcome of the test is followed, until a leaf node is reached, for which the corresponding predicted value is returned. For a forest of classification trees, the predicted value is typically formed by selecting the majority among the predictions of the individual trees, while for a forest of regression trees, the resulting prediction is typically formed by taking the average of the individual predictions.
The standard procedure to generate a decision tree is to employ a recursive partitioning, or divideandconquer, strategy, starting with all training examples at the root node of the tree, and then evaluating all available tests to partition the examples, choosing the one that maximizes some evaluation metric, e.g., information gain for classification trees or variance for regression trees, labeling the current node with this test, partitioning the examples according to the outcome of the test and continuing building the tree recursively with each resulting subset, until some termination criterion is met, e.g., all examples in the subset have the same value on the target attribute, and forming a value to predict from the examples in the subset.
In order to introduce the necessary diversity among the trees in a random forest, each tree is trained on a bootstrap replicate of the original training set (Breiman 1996), i.e., a new training (multi)set, or bag, of the same size as the original set is formed by randomly selecting examples with replacement from the original set. This means that some of the original examples may be duplicated in the bootstrap replicate, while other examples are excluded. The latter ones, for a specific tree, are said to be outofbag for that tree. To further increase diversity, each tree in the forest is created using the random subspace method (Ho 1998), i.e., only a randomly selected subset of all available attributes are evaluated when choosing the split at each internal node during the construction of the decision tree.
In this study, the implementation of the random forest algorithm from the MatLab ^{Footnote 1} statistics toolbox, called TreeBagger, was used. The parameters were set to the default values for regression trees, i.e., the number of attributes to evaluate at each internal node was set to one third of the total number of attributes and mean square error was used as the split criterion.
Given that random forests are frequently observed to result in stateoftheart predictive performance, see e.g., Caruana and NiculescuMizil (2006), random forest models can be expected to be more accurate than the underlying models that are currently used in regression conformal predictors, i.e., kNN and ANN models. It is, however, not obvious whether or not the use of random forests will result in smaller prediction intervals, when the models are used as the basis for CP. Another important question is whether or not anything could be gained from using outofbag estimates for the calibration, something which is an option for random forests, but not for the previous model types, which have to resort to using a separate calibration set.
In this study, we investigate nonconformity functions that are based on absolute errors (2). The first two nonconformity functions that will be considered for random forests use no normalization, i.e., the intervals are produced using (4). The first approach, called RFi, employs standard ICP, i.e., a separate calibration set is used. In the second approach, called RFo, outofbag instances are instead used for the calibration. This, of course, makes it possible to use all training instances for both the training and the calibration. More specifically, when producing the nonconformity score for a calibration instance \(z_i\), the ensemble used for producing the prediction \(\hat{y}_i\) consists of all trees that were not trained using \(z_i\), i.e., \(z_i\) was outofbag for those trees.
It should be noted that when using outofbag instances instead of a separate calibration set, the actual underlying model, i.e., the random forest, is no longer used when calculating the nonconformity scores and \(p\)values. In fact, various subsets of the forest are used for the outofbaginstances, but the entire forest is used for the test instances. In other words, the nonconformity functions applied to the calibration and test instances are defined differently as
where \(\theta \) is a random factor determining the subset of trees for which \(x\) is outofbag. In general, the use of different nonconformity functions could clearly cause the resulting conformal predictor to become invalid, i.e., the probability of excluding the true target value would no longer be bounded by the provided confidence level. However, we argue that the conformal predictor in our particular setting, i.e., when using outofbag estimates for the calibration, must be valid. In principle, the same random component may also be used when predicting the target value for the test instance (by only considering a random subset of the forest when predicting the target of the test instance), and in that case the same nonconformity function (7) would obviously be used for all instances, hence not violating the assumptions underlying the ICP framework. When instead using the whole forest for the test instance, as proposed here, one would expect the predicted values to be closer to the true target, than when using a random subset of the trees. In fact, it is wellknown that outofbag error estimates tend to overestimate the actual error made by a random forest, simply because a larger forest is normally a stronger model. Not until the random forest is so large that the randomized subensembles will be as accurate as the entire forest, is this bias eliminated. Consequently, the expected nonconformity of a test instance is less than (or for a very large forest equal to) the expected nonconformity of a calibration instance, i.e., the probability of including nonconforming targets in the prediction region is unchanged or increased when using the whole forest. Hence, rather than increasing the risk for generating an invalid conformal predictor, one would expect the conformal predictor using outofbag instances to be conservative. Therefore, the proposed setup should be, if anything, less efficient than if the whole forest was used together with additional calibration instances. Naturally, the validity will be investigated in the experimentation in order to support this reasoning empirically.
We also investigate three normalized nonconformity functions, i.e., the prediction regions may vary for different test instances. RFia and RFoa both use an additional (linear) ANN to predict the logarithm of the error of the underlying model, for each instance. RFia is identical to the procedure used in Papadopoulos and Haralambous (2011) and Papadopoulos and Haralambous (2010), but of course uses a random forest as the underlying model instead of an ANN. The resulting nonconformity function is:
where \(\mu _i\) is the prediction of the value \(ln( y_i  \hat{y}_i )\) produced by the linear ANN, and \(\beta \) is a parameter, used to control the sensitivity of the nonconformity measure. Naturally, this ANN was trained on all pairs \((x_j, ln( y_j  \hat{y}_j )\), from the proper training set. Using this nonconformity function, the prediction intervals become:
The only difference between RFia and the novel setup RFoa is that RFia uses a separate calibration set, while RFoa uses the outofbag instances for the calibration, i.e., the additional ANN is trained using the logarithm of the outofbag errors as targets. RFok, finally, is another novel setup, which instead of employing an additional ANN for predicting the logarithm of the error, considers the average outofbag error (normalized with the Euclidean distance) for the \(k\) closest instances. That is, RFok is based on the same Eq. (9), but here, \(\mu _i\) is defined as the logarithm of the average outofbag error of the \(k\) nearest neighbors. The motivation for this novel, and quite straightforward nonconformity function, is that if neighboring instances have small outofbag errors, the prediction for the new instance should be accurate, i.e., that instance should be considered as relatively easy. The exact number of neighbors to use is optimized (between \(1\) and \(45\)) for each fold based on the average interval size of the resulting conformal regressor. This fitting is similar to when the ANN is trained to learn the outofbag errors. Naturally, since both RFoa and RFok utilize the outofbag estimates for the calibration, they make it possible to use all available data as a proper training set for the random forest.
Since the normalization functions used in RFoa and RFok depend on the outofbag errors of the calibration instances, one may again raise concerns on the validity of the corresponding conformal regressors. Starting with RFok, we claim that this normalization is unbiased. More specifically, the difficulty of any instance (calibration or test) is estimated in the same manner, using its \(k\) closest calibration set neighbors, not counting a calibration instance as its own closest neighbor. Hence, for both test and calibration instances, the difficulty estimate is based on a set of examples that does not include the instance to which the estimate applies. Consequently, there is no reason to suspect that the estimated difficulty of a test instance is any less accurate than that of a calibration instance. The error rate of RFok is thus not expected to be affected by the normalization function used, and RFok is therefore expected to keep the (slightly conservative) validity of the RFo nonconformity function it is based on. In RFoa, on the other hand, the difficultyestimating ANN model has been trained on the outofbag error of all calibration instances, i.e., when predicting the difficulty of some calibration instance, that particular instance will have been used in the training of the ANN, whereas the same does not apply to any test instance. So, in this case, there is indeed a bias towards the calibration set.
Consequently, for RFoa there are two forces working in opposite directions; the inherent conservatism in using outofbag estimates and the bias towards the calibration set when estimating the difficulty of an instance. When the latter bias is small, e.g., if the ANN is relatively weak, the resulting error rate will most likely be smaller than the confidence level, but for a larger bias, the error rate may actually be higher than the confidence level. With this in mind, it is important to recognize that RFoa is the only setup evaluated for which there is a known risk that validity is not guaranteed. Again, the empirical investigation will study how the error rate is affected by these nonconformity functions in practice.
Competing approaches
In the empirical evaluation, we compare the different variants of our suggested method to the stateoftheart techniques from Papadopoulos and Haralambous (2011) and Papadopoulos et al. (2011). In both these papers, ICP methods were used, i.e., separate calibration sets were required. The first competing method, suggested and described in detail in Papadopoulos and Haralambous (2011), uses an ANN as the underlying model. In the most basic format (here referred to as ANN), it uses the standard nonconformity function (2) and produces intervals using (4). When using a normalized nonconformity function, the method, which is here referred to as ANNa, uses a linear ANN to predict the logarithm of the errors, and produces intervals using (9) and (10).
The second competing method is based on distanceweighted \(k\)nearest neighbor regressors, and is suggested and described in Papadopoulos et al. (2011). In the basic format, this method (referred to as kNN) also uses the standard way of calculating nonconformity scores (2) and prediction intervals (4). In Papadopoulos et al. (2011), the authors evaluate a number of novel normalized nonconformity functions. The most efficient (here called kNNc) combines two different aspects of kNN, in order to produce as good estimates of the accuracy as possible. More specifically, the prediction from a kNN regressor is deemed to be more accurate, for a specific instance, if (i) the \(k\) nearest neighbors are close to the current test instance and (ii) the \(k\) nearest neighbors agree in their predictions. The resulting nonconformity function is
where \(\lambda \) and \(\xi \) are the measures of accuracy (difficulty) while \(\gamma \) and \(\rho \) are parameters controlling the sensitivity of each measure. Consequently, the prediction intervals are calculated using
It must be noted that an internal normalization of the measures is used to make sure that the two measures are of the same magnitude and robust over all data sets. For a detailed discussion on these and other difficulty estimators, see Papadopoulos et al. (2011).
Empirical evaluation
In this section, we first describe the experimental setup, i.e., what algorithms, data sets and performance metrics have been chosen, and then report the results from the experiment.
Experimental setup
In the first (main) experiment, the competing methods were reimplemented, and a largescale study, using \(33\) publicly available data sets was performed. The considered data sets are small to medium sized; ranging from approximately \(500\) to \(10,000\) instances. All but one data set are from the UCI Bache and Lichman (2013), Delve Rasmussen et al. (1996) or KEEL AlcaláFdez et al. (2011) repositories. The data sets are described in Table 1, where #inst. is the number of instances, #attrib. is the number of input attributes and #calInst, is the number of instances used for calibration in the standard ICP settings.
In the evaluation, we look at standard and normalized nonconformity functions separately. Naturally, the normalized nonconformity functions are the most important, since they provide prediction intervals of different sizes. In the second experiment, we employed the exact same settings as in the previous studies, including using only a handful of data sets, and compare our results directly to the published results.
In Experiment 1, a 10\(\times \)10fold crossvalidation scheme was used. The number of calibration instances was set to
where \(Z\) is the full training set, i.e., starting at \(99\) calibration instances for data sets with 400–799 examples, and adding \(100\) calibration instances for every additional \(400\) examples in the full training set. Before the experimentation, all target values were normalized to \([0,1]\), in order to obtain more readable efficiency comparisons across data sets. With this scaling, the size of a prediction interval, of course, expresses the fraction of the target range covered by the interval.
Regarding parameter values, we elected to use identical settings over all data sets and, when applicable, methods. Specifically, all random forests consisted of 500 random trees. For kNN regressors, \(k\) was set to \(25\), since some preliminary experiments showed that this actually produced higher efficiency than selecting different \(k\)values based on internal crossvalidation results. Similarly, all ANNs had exactly \(20\) hidden units. The sensitivity parameters had to be adjusted based on the much smaller normalized target ranges. Again, some preliminary experiments showed that the exact values were not vital, so the following values were used in Experiment 1: \(\beta = 0.01\) and \(\gamma = \rho = 1.0\).
Two things were measured for each method and data set in the experiments; the error rate, i.e., the fraction of target values in the test set that fall outside the predicted regions, and the efficiency, i.e., the size of the predicted intervals. For valid conformal predictors, the error rates should not (in the long run) exceed the chosen confidence threshold. Hence, by investigating the error rates, we may confirm (or reject) that a certain conformal predictor actually is valid. Note that this is here considered to be a binary property, i.e., we do not consider one method to be more valid than another. Given that we have a set of valid regression conformal predictors, the perhaps most interesting aspect to compare is the size of the predicted regions, as this directly corresponds to how informative these regions are. Such a comparison could be done in different ways, e.g., comparing extreme values, but we have opted for comparing the average sizes over all prediction regions. In fact, we report the median value from the ten runs of tenfold crossvalidation.
In order to be able to do a direct comparison with published results, we used the same settings for our methods in Experiment 2 as originally employed for the specific data sets, with regard to number of folds and number of calibration instances. In addition, in this experiment, the targets were not normalized. It may be noted that parameters like \(k\) in kNN and the number of hidden neurons in the ANNs, in the original studies, were optimized based on accuracy results using internal crossvalidation. All sensitivity parameters (\(\beta \), \(\gamma \) and \(\rho \)) were, however, despite the fact that the importance of the parameters will be heavily affected by the actual range of the target values, somewhat ad hoc set to \(0.5\). Consequently, we too set \(\beta = 0.5\) for all our methods in Experiment 2.
All experimentation was performed in MatLab, in particular using the Neural network and the Statistics toolboxes.
Experimental results
Table 2 demonstrates validity for the methods utilizing standard nonconformity functions. Looking at the error rates, i.e., the fraction of test instances for which the true target value falls outside the predicted region, it is reassuring to see that the empirical results for each and every data set is very close to the predetermined confidence levels. In addition, it can be noted that RFo tends to be slightly conservative, which supports the reasoning about validity in Sect. 3.1.
Looking at the interval sizes tabulated in Table 3, while remembering that the output was normalized so that an interval size of \(1.0\) would cover the entire range of the target values, it can be seen from the averaged values that the methods at the \(90~\%\) confidence level returned valid prediction intervals covering, approximately, \(25~\%\) of the range. The corresponding average values for the 95 and \(99~\%\) confidence levels are (approximately) 30 and \(50~\%\), respectively. Clearly, these valid prediction intervals must be considered informative.
In order to compare the efficiency of the five different techniques, and to find out if there are any statistically significant differences, we used the recommended procedure in Garcıa and Herrera (2008) and performed a Friedman test Friedman (1937), followed by Bergmann–Hommel’s dynamic procedure Bergmann and Hommel (1988) to establish all pairwise differences. Table 4 shows the adjusted p values. The most important result is that RFo is either significantly or substantially more efficient than all three competing methods, at the different confidence levels. Specifically, it should be noted that RFo outperformed RFi, clearly showing that the use of outofbag instances for the calibration is beneficial.
Table 5 demonstrates the validity for the methods utilizing normalized nonconformity functions. Again we see that all methods, including RFoa, produced valid and wellcalibrated conformal predictors. Actually, when using 10\(\times \)10fold crossvalidation, the empirical error rates for most individual data sets are very close to the confidence level. Clearly, these results support our argumentation above that the suggested setup RFok will produce valid (possibly slightly conservative) conformal regressors. Although RFoa was argued above to be associated with a risk of producing nonvalid predictions due to a bias in the difficulty estimation function, using the specific settings and parameter values employed here, this bias turned out to be compensated for by the conservative outofbag error estimates, thus resulting in empirical error rates below the confidence threshold even for this nonconformity function.
Table 6 shows the interval widths for the normalized nonconformity functions. Comparing these to the results in Table 3, it is obvious that the prediction intervals here are much smaller, i.e., applying normalized nonconformity functions does not only provide tuned prediction intervals for each specific test instance, but the resulting intervals are also substantially tighter on average. Looking at individual methods, the mean ranks identify three groups, which are the same for all confidence levels; RFok is the most efficient method, followed by RFoa and then the other three methods. Clearly, this is a very strong result in favor of using random forests with outofbag examples for calibration.
Studying the adjusted p values in Table 7, we see that RFok is indeed significantly more efficient (for \(\alpha =0.05\)) than all other methods, with the exception of RFoa, on all three confidence levels. In addition, RFoa is either significantly or substantially more efficient than the existing methods utilizing separate calibration sets. Again, it is important to note that the two setups utilizing the outofbag instances clearly outperformed using a separate calibration set.
Summarizing the main experiment, we see that all variants of the novel method produced empirically valid conformal predictors. Most importantly, the suggested approach, i.e., using random forests as the underlying model and utilizing outofbag instances for the calibration, clearly outperformed the existing alternatives with regard to efficiency. Finally, when comparing the specific nonconformity functions, the novel, theoretically sound and quite straightforward method to estimate the accuracy of the underlying model based on outofbag errors for neighboring instances, actually turned out to be the most efficient.
In order to analyze and explain the results further, the left part of Table 8 shows the accuracy (measured using Root Mean Square Error) for the different underlying models. Looking at mean values and ranks, the most obvious result is that the kNN models are the weakest. We also see that the random forests are generally the most accurate, and that there is a small but systematic advantage in using all the data for the training. The fact that the ANN has a better mean rank, but a worse average error, than RFi is explained by the fact that the two random forest setups have very similar accuracy.
The right part of Table 8 shows the quality of the difficulty estimators. More specifically, the numbers tabulated are the correlations between the estimated difficulty and the actual error made by the underlying model on the test instances. It should be noted that these estimations are calculated in quite different ways by the different setups. ANNa, RFia and RFoa all train a separate model (a linear ANN) to predict the actual error for each instance, while RFok uses the average outofbag error from the \(k\) nearest neighbors. kNNc, finally, does not explicitly use or model the errors of the underlying model; instead an instance is deemed to be easier if the \(k\) nearest neighbors are (relatively) close and agree in their predictions.
Comparing the different estimators, we see that the estimates produced by RFok have the highest correlation with the model errors. The second best is actually kNNc, followed by the approaches using a separate ANN model as estimator.
From this analysis, and the comparison between normalized and standard nonconformity functions above, it is obvious that although the accuracy of the underlying model is very important, the quality of the nonconformity function is vital for the efficiency. Specifically, using normalized nonconformity functions will increase the efficiency and the quality of the difficulty estimations has an apparent impact on the efficiency.
In order to determine the importance of parameter values, a limited posthoc analysis was performed, see Table 9. As described above, the parameters \(\beta , \rho \) and \(\gamma \) balance the difficulty estimations against the error in the nonconformity functions. In previous studies, all parameter values were set to \(0.5\), which is not a very robust choice since the importance of the parameter value is heavily affected by the range of the target variable. In this study, as described in Sect. 4.1, the target variables were normalized to \([0,1]\), so both errors and error estimates take on much smaller values. With this in mind, it was obvious that \(\beta \) too must be smaller. In the experimentation, we set \(\beta = 0.01\), based on some initial experimentation. When looking at the efficiencies obtained using different values for \(\beta \), we see that \(0.01\) is actually the best choice, even if the difference when compared to \(\beta = 0\) is marginal, on most data sets. Larger values for \(\beta \), on the other hand, clearly reduce the efficiency. So, the conclusion is that although the parameter value \(\beta \) is very important and dependent on the target range, all reasonable values produce conformal predictors with similar efficiency. Looking at the kNNc setup, it should be noted that there are actually two different parameters, making it possible to balance the two different parts of the difficulty estimation. In this study, however, this was not evaluated, instead \(\rho \) and \(\gamma \) were both set to \(1.0\). From the posthoc analysis, we can see that while larger values (i.e., \(1.25\) or \(1.5\)) would have increased the efficiency slightly for the lower confidence levels, the resulting differences are almost never large enough to change the ordering of the evaluated setups for specific data sets in the main experiment. In addition, for the confidence level of \(99~\%, \rho = \gamma = 1.0\) was actually the best parameter setting.
In order to directly compare the efficiency of our methods to the published results in Papadopoulos and Haralambous (2011) and Papadopoulos et al. (2011), Table 10 compares the interval sizes obtained in Experiment 2 to the interval sizes published. Starting with the standard nonconformity functions, we immediately see that RFi and RFo almost always produced smaller prediction intervals than kNN. For several data sets and confidence levels, the differences are quite large. From a direct comparison, it is quite obvious that both RFo and RFi were more efficient than kNN, winning five of six data sets on all confidence levels. When compared to ANN, however, the results vary over the different confidence levels.
For normalized nonconformity functions, we see that RFoa and, in particular, RFok are the most efficient over all data sets and confidence levels. Counting wins and losses, RFoa and RFok outperformed both kNNc and ANNa on a majority of data sets. To summarize this comparison with published results, we see that the results from the main experiment are confirmed, i.e., conformal regressors based on random forests, especially when utilizing outofbag instances for the calibration, outperform the existing techniques, both when using standard and normalized nonconformity functions.
Concluding remarks
In this paper, the use of random forests has been proposed as a strong candidate for regression conformal prediction, since it allows for the necessary calibration to be performed on the outofbag examples, thus making it possible to utilize all available data as a proper training set. In one of the largest empirical evaluations to date on regression conformal prediction, the random forest approach was compared to existing stateoftheart approaches, based on ANN and kNN, for both the standard and normalized settings, i.e., when generating prediction intervals of uniform and varying sizes, respectively. The results show that the suggested approach, on almost all confidence levels and using both standard and normalized nonconformity functions, produced significantly more efficient conformal predictors than the existing alternatives. In particular, the most efficient setup overall was found to be one suggested in this paper, i.e., a random forest conformal predictor calibrated using a normalized nonconformity function based on outofbag errors of neighboring instances. The empirical evidence hence strongly suggests that random forests in conjunction with outofbag calibration is a highly competitive conformal regressor.
There are several possible directions for future research. One direction concerns the type of model to use for estimating the difficulty of each instance in the normalized setting. Currently, fairly simple models have been evaluated, i.e., kNN and linear ANNs, and gains are to be expected from considering more elaborate techniques, as well as from performing further parameter tuning. Another direction concerns investigating the application of other stateoftheart machine learning algorithms, e.g., SVMs, to the regression conformal prediction framework and comparing the resulting conformal predictors to random forests with outofbag calibration.
Notes
References
AlcaláFdez, J., Fernández, A., Luengo, J., Derrac, J., & García, S. (2011). Keel datamining software tool: Data set repository, integration of algorithms and experimental analysis framework. MultipleValued Logic and Soft Computing, 17(2–3), 255–287.
Bache, K., & Lichman, M. (2013). UCI machine learning repository, URL http://archive.ics.uci.edu/ml.
Bergmann, B., & Hommel, G. (1988). Improvements of general multiple test procedures for redundant systems of hypotheses. In Multiple hypotheses testing (pp. 100–115). New York: Springer.
Bhattacharyya, S. (2011). Confidence in predictions from random tree ensembles. In: IEEE ICDM (pp. 71–80).
Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984). Classification and regression trees. Belmont: Wadsworth.
Breiman, L. (1996). Bagging predictors. Machine Learning, 24(2), 123–140.
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.
Caruana, R., & NiculescuMizil, A. (2006). An empirical comparison of supervised learning algorithms. In Proceedings of the 23rd International Conference on Machine Learning (pp. 161–168). ACM.
Devetyarov, D., & Nouretdinov, I. (2010). Prediction with confidence based on a random forest classifier. Artificial Intelligence Applications and Innovations, 7, 37–44.
Devetyarov, D., Nouretdinov, I., Burford, B., Camuzeaux, S., GentryMaharaj, A., Tiss, A., et al. (2012). Conformal predictors in early diagnostics of ovarian and breast cancers. Progress in Artificial Intelligence, 1(3), 245–257.
Flake, G. W., & Lawrence, S. (2002). Efficient svm regression training with smo. Machine Learning, 46(1–3), 271–290.
Friedman, M. (1937). The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of American Statistical Association, 32, 675–701.
Gammerman, A., Vovk, V., & Vapnik, V. (1998). Learning by transduction. In Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (pp. 148–155). Burlington: Morgan Kaufmann.
Garcıa, S., & Herrera, F. (2008). An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. Journal of Machine Learning Research, 9(2677–2694), 66.
Ho, T. K. (1998). The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(8), 832–844.
Johansson, U., Boström, H., & Löfström, T. (2013a). Conformal prediction using decision trees. In IEEE International Conference on Data Mining (pp. 330–339).
Johansson, U., König, R., Löfström, T., & Boström, H. (2013b). Evolved decision trees as conformal predictors. In IEEE Congress on Evolutionary Computation (pp. 1794–1801).
Lambrou, A., Papadopoulos, H., & Gammerman, A. (2011). Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine, 15(1), 93–99.
Löfström, T., Johansson, U., & Boström, H. (2013). Effective utilization of data in inductive conformal prediction. In The IEEE 2013 International Joint Conference on Neural Networks (IJCNN).
Makili, L., Vega, J., DormidoCanto, S., Pastor, I., & Murari, A. (2011). Computationally efficient svm multiclass image recognition with confidence measures. Fusion Engineering and Design, 86(6), 1213–1216.
Nguyen, K., & Z, Luo. (2012). Conformal prediction for indoor localisation with fingerprinting method. Artificial Intelligence Applications and Innovations, 3, 214–223.
Nouretdinov, I., Vovk, V., Vyugin, M., & Gammerman, A. (2001). Pattern recognition and density estimation under the general i.i.d. assumption. Computational Learning Theory (pp. 337–353)., volume 2111 of Lecture Notes in Computer Science Berlin Heidelberg: Springer.
Papadopoulos, H. (2008). Inductive conformal prediction: Theory and application to neural networks. Tools in Artificial Intelligence, 18(315–330), 2.
Papadopoulos, H., & Haralambous, H. (2010). Neural networks regression inductive conformal predictor and its application to total electron content prediction. Artificial Neural Networks ICANN 2010 (pp. 32–41)., volume 6352 of Lecture Notes in Computer Science Berlin Heidelberg: Springer.
Papadopoulos, H., & Haralambous, H. (2011). Reliable prediction intervals with regression neural networks. Neural Networks, 24(8), 842–851.
Papadopoulos, H., Proedrou, K., Vovk, V., & Gammerman, A. (2002). Inductive confidence machines for regression. In Machine Learning: ECML 2002 (pp. 345–356). New York: Springer.
Papadopoulos, H., Gammerman, A., & Vovk, V. (2009a). Reliable diagnosis of acute abdominal pain with conformal prediction. Engineering Intelligent Systems, 17(2), 127.
Papadopoulos, H., Papatheocharous, E., & Andreou, A. S. (2009b). Reliable confidence intervals for software effort estimation. In AIAI Workshops (pp. 211–220).
Papadopoulos, H., Vovk, V., & Gammerman, A. (2011). Regression conformal prediction with nearest neighbours. Journal of Artificial Intelligence Research, 40(1), 815–840.
Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1, 81–106.
Rasmussen, C. E., & Christopher, K. I. (2005). Gaussian processes for machine learning. Cambridge: MIT Press.
Rasmussen, C. E., Neal, R. M., Hinton, G. E., van Camp, D., Revow, M., Ghahramani, Z., Kustra, R., & Tibshirani, R. (1996). Delve data for evaluating learning in valid experiments. www.cs.toronto.edu/delve.
Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11), 1134–1142.
Vovk, V., Gammerman, A., & Shafer, G. (2006). Algorithmic learning in a random world. New York: Springer.
Acknowledgments
We thank the anonymous reviewers for many helpful comments and suggestions. This work was supported by the Swedish Foundation for Strategic Research through the project HighPerformance Data Mining for Drug Effect Detection (IIS110053) and the Knowledge Foundation through the project Big Data Analytics by Online Ensemble Learning (20120192).
Author information
Affiliations
Corresponding author
Additional information
Editors: Toon Calders, Rosa Meo, Floriana Esposito, and Eyke Hullermeier.
Rights and permissions
About this article
Cite this article
Johansson, U., Boström, H., Löfström, T. et al. Regression conformal prediction with random forests. Mach Learn 97, 155–176 (2014). https://doi.org/10.1007/s1099401454530
Received:
Accepted:
Published:
Issue Date:
Keywords
 Conformal prediction
 Random forests
 Regression