Bayesian Additive Regression Trees using Bayesian model averaging

Abstract

Bayesian Additive Regression Trees (BART) is a statistical sum of trees model. It can be considered a Bayesian version of machine learning tree ensemble methods where the individual trees are the base learners. However, for datasets where the number of variables p is large the algorithm can become inefficient and computationally expensive. Another method which is popular for high-dimensional data is random forests, a machine learning algorithm which grows trees using a greedy search for the best split points. However, its default implementation does not produce probabilistic estimates or predictions. We propose an alternative fitting algorithm for BART called BART-BMA, which uses Bayesian model averaging and a greedy search algorithm to obtain a posterior distribution more efficiently than BART for datasets with large p. BART-BMA incorporates elements of both BART and random forests to offer a model-based algorithm which can deal with high-dimensional data. We have found that BART-BMA can be run in a reasonable time on a standard laptop for the “small n large p” scenario which is common in many areas of bioinformatics. We showcase this method using simulated data and data from two real proteomic experiments, one to distinguish between patients with cardiovascular disease and controls and another to classify aggressive from non-aggressive prostate cancer. We compare our results to their main competitors. Open source code written in R and Rcpp to run BART-BMA can be found at: https://github.com/BelindaHernandez/BART-BMA.git.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3

References

  1. Albert, J.H., Chib, S.: Bayesian analysis of binary and polychotomous response data. J. Am. Stat. Assoc. 88(422), 669–679 (1993)

    MathSciNet  Article  MATH  Google Scholar 

  2. Archer, K., Kimes, R.: Empirical characterization of random forest variable importance measures. Comput. Stat. Data Anal. 52(4), 2249–2260 (2008). doi:10.1016/j.csda.2007.08.015

    MathSciNet  Article  MATH  Google Scholar 

  3. Beaumont, M.A., Rannala, B.: The Bayesian revolution in genetics. Nat. Rev. Genet. 5(4), 251–261 (2004)

    Article  Google Scholar 

  4. Bleich, J., Kapelner, A., George, E.I., Jensen, S.T.: Variable selection for BART: an application to gene regulation. Ann. Appl. Stat. 8(3), 1750–1781 (2014)

    MathSciNet  Article  MATH  Google Scholar 

  5. Breiman, L.: Bagging predictors. Mach. Learn. 26, 123–140 (1996a)

    MATH  Google Scholar 

  6. Breiman, L.: Stacked regressions. Mach. Learn. 24, 41–64 (1996b)

    MathSciNet  MATH  Google Scholar 

  7. Breiman, L.: Random forests. Mach. Learn. 45, 5–32 (2001). doi:10.1186/1478-7954-9-29

    Article  MATH  Google Scholar 

  8. Breiman, L., Friedman, J., Olshen, R., Stone, C.: Classification and Regression Trees. Wadsworth, Belmont (1984)

    Google Scholar 

  9. Bühlmann, P., Van De Geer, S.: Statistics for High-Dimensional Data: Methods, Theory and Applications. Springer, Berlin (2011)

    Google Scholar 

  10. Chipman, H., George, E.I., McCulloch, R.E.M.: Bayesian CART model search. J. Am. Stat. Assoc. 93(443), 935–948 (1998)

    Article  Google Scholar 

  11. Chipman, H., George, E.I., Mcculloch, R.E.M.: BART: Bayesian additive regression trees. Ann. Appl. Stat. 4(1), 266–298 (2010)

    MathSciNet  Article  MATH  Google Scholar 

  12. Chipman, H., McCulloch, R., Dorie, V.: Package dbarts (2014). https://cran.r-project.org/web/packages/dbarts/dbarts.pdf

  13. Cortes, I.: Package conformal (2014). https://cran.r-project.org/web/packages/conformal/conformal.pdf

  14. Daz-Uriarte, R., Alvarez de Andrés, S.: Gene selection and classification of microarray data using random forest. BMC Bioinform. 7, 3 (2006). doi:10.1186/1471-2105-7-3

    Article  Google Scholar 

  15. Friedman, J.H.: Multivariate adaptive regression splines (with discussion and a rejoinder by the author). Ann. Stat. 19, 1–67 (1991)

    Article  Google Scholar 

  16. Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001). doi:10.1214/aos/1013203451

  17. Fujikoshi, Y., Ulyanov, V.V., Shimizu, R.: Multivariate Statistics: High-Dimensional and Large-Sample Approximations, vol. 760. Wiley, Hoboken (2011)

    Google Scholar 

  18. Geurts, P., Ernst, D., Wehenkel, L.: Extremely randomized trees. Mach. Learn. 63(1), 3–42 (2006)

    Article  MATH  Google Scholar 

  19. Ham, J., Chen, Y., Crawford, M.M., Ghosh, J.: Investigation of the random forest framework for classification of hyperspectral data. IEEE Trans. Geosci. Remote Sens. 43(3), 492–501 (2005). doi:10.1109/TGRS.2004.842481

    Article  Google Scholar 

  20. Harris, K., Girolami, M., Mischak, H.: Pattern Recognition in Bioinformatics, Lecture Notes in Computer Science, chap. Definition of Valid Proteomic Biomarkers: A Bayesian Solution, pp. 137–149. Springer, Berlin (2009)

  21. Hawkins, D.M.: Fitting multiple change-point models to data. Comput. Stat. Data Anal. 37(3), 323–341 (2001)

    MathSciNet  Article  MATH  Google Scholar 

  22. Hernández, B., Parnell, A.C., Pennington, S.R.: Why have so few proteomic biomarkers “survived” validation? (sample size and independent validation considerations). Proteomics 14(13–14), 1587–1592 (2014)

    Article  Google Scholar 

  23. Hernández, B., Pennington, S.R., Parnell, A.C.: Bayesian methods for proteomic biomarker development. EuPA Open Proteomics 9, 54–64 (2015)

    Article  Google Scholar 

  24. Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: methods & evaluation. Artif. Intell. 206, 79–111 (2014)

    MathSciNet  Article  MATH  Google Scholar 

  25. Johansson, U., Boström, H., Löfström, T., Linusson, H.: Regression conformal prediction with random forests. Mach. Learn. 97(1–2), 155–176 (2014)

    MathSciNet  Article  MATH  Google Scholar 

  26. Kapelner, A., Bleich, J.: bartmachine: machine learning with Bayesian additive regression trees. ArXiv e-prints (2014a)

  27. Kapelner, A., Bleich, J.: Package bartMachine (2014b). http://cran.r-project.org/web/packages/bartMachine/bartMachine.pdf

  28. Killick, R., Eckley, I., Haynes, K., Fearnhead, P.: Package changepoint (2014). http://cran.r-project.org/web/packages/changepoint/changepoint.pdf

  29. Killick, R., Fearnhead, P., Eckley, I.: Optimal detection of changepoints with a linear computational cost. J. Am. Stat. Assoc. 107(500), 1590–1598 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  30. Lakshminarayanan, B., Roy, D.M., Teh, Y.W.: Particle Gibbs for Bayesian additive regression trees. arXiv preprint arXiv:1502.04622 (2015)

  31. Lakshminarayanan, B., Roy, D.M., Teh, Y.W.: Mondrian forests for large-scale regression when uncertainty matters. In: Artificial Intelligence and Statistics, pp. 1478–1487. (arXiv:1506.03805, 2015) (2016)

  32. Liaw, A., Matthew, W.: Package randomForest (2015). http://cran.r-project.org/web/packages/randomForest/randomForest.pdf

  33. Logothetis, C.J., Gallick, G.E., Maity, S.N., Kim, J., Aparicio, A., Efstathiou, E., Lin, S.H.: Molecular classification of prostate cancer progression: foundation for marker-driven treatment of prostate cancer. Cancer Discov. 3(8), 849–861 (2013)

    Article  Google Scholar 

  34. Lynch, C.: Big data: how do your data grow? Nature 455(7209), 28–29 (2008)

    Article  Google Scholar 

  35. Madigan, D., Raftery, A.E.: Model selection and accounting for model uncertainty in graphical models using Occam’s window. J. Am. Stat. Assoc. 89(428), 1535–1546 (1994)

    Article  MATH  Google Scholar 

  36. Meinshausen, N.: Quantile regression forests. J. Mach. Learn. Res. 7, 983–999 (2006)

    MathSciNet  MATH  Google Scholar 

  37. Morgan, J.N.: History and potential of binary segmentation for exploratory data analysis. J. Data Sci. 3, 123–136 (2005)

    Google Scholar 

  38. Morgan, J.N., Sonquist, J.A.: Problems in the analysis of survey data and a proposal. J. Am. Stat. Assoc. 58(302), 415–434 (1963)

    Article  MATH  Google Scholar 

  39. Nicodemus, K.K., Malley, J.D., Strobl, C., Ziegler, A.: The behaviour of random forest permutation-based variable importance measures under predictor correlation. BMC Bioinform. 11, 110 (2010). doi:10.1186/1471-2105-11-110

    Article  Google Scholar 

  40. Norinder, U., Carlsson, L., Boyer, S., Eklund, M.: Introducing conformal prediction in predictive modeling. A transparent and flexible alternative to applicability domain determination. J. Chem. Inf. Model. 54(6), 1596–1603 (2014)

    Article  Google Scholar 

  41. Pratola, M.: Efficient Metropolis–Hastings proposal mechanisms for Bayesian regression tree models. Bayesian Anal. 11(3), 885–911 (2016)

    MathSciNet  Article  MATH  Google Scholar 

  42. Quinlan, J.: Induction of decision trees. Mach. Learn. 1(1), 81–106 (1986). doi:10.1023/A:1022643204877

    Google Scholar 

  43. Quinlan, J.R.: Discovering rules by induction from large collections of examples. In: Michie, D. (ed.) Expert Systems in the Micro Electronic Age. Edinburgh University Press, Edinburgh (1979)

    Google Scholar 

  44. Raghavan, V., Bollmann, P., Jung, G.S.: A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst. (TOIS) 7(3), 205–229 (1989)

    Article  Google Scholar 

  45. Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 461–464 (1978)

    MathSciNet  Article  MATH  Google Scholar 

  46. Svetnik, V., Liaw, A., Tong, C., Culberson, J.C., Sheridan, R.P., Feuston, B.P.: Random forest: a classification and regression tool for compound classification and QSAR modeling. J. Chem. Inf. Comput. Sci. 43(6), 1947–1958 (2003). doi:10.1021/ci034160g

    Article  Google Scholar 

  47. Wager, S., Hastie, T., Efron, B.: Confidence intervals for random forests: the jackknife and the infinitesimal jackknife. J. Mach. Learn. Res. 15(1), 1625–1651 (2014)

    MathSciNet  MATH  Google Scholar 

  48. Wilkinson, D.J.: Bayesian methods in bioinformatics and computational systems biology. Brief. Bioinform. 8(2), 109–16 (2007). doi:10.1093/bib/bbm007

    Article  Google Scholar 

  49. Wu, Y., Tjelmeland, H., West, M.: Bayesian CART: prior specification and posterior simulation. J. Comput. Graph. Stat. 16(1), 44–66 (2007)

    MathSciNet  Article  Google Scholar 

  50. Yao, Y.: Estimation of a noisy discrete-time step function: Bayes and empirical Bayes approaches. Ann. Stat. 4(12), 1434–1447 (1984)

    MathSciNet  Article  MATH  Google Scholar 

  51. Zhao, T., Liu, H., Roeder, K., Lafferty, J., Wasserman, L.: The huge package for high-dimensional undirected graph estimation in R. J. Mach. Learn. Res. 13(1), 1059–1062 (2012)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

We would like to thank Drs Chris Watson, John Baugh, Mark Ledwidge and Professor Kenneth McDonald for kindly allowing us to use the cardiovascular dataset described. Hernández’s research was supported by the Irish Research Council. Raftery’s research was supported by NIH Grants Nos. R01-HD054511, R01-HD070936, and U54-HL127624, and by a Science Foundation Ireland E.T.S. Walton visitor award, Grant Reference 11/W.1/I2079. Protein biomarker discovery work in the Pennington Biomedical Proteomics Group is supported by grants from Science Foundation Ireland (for mass spectrometry instrumentation), the Irish Cancer Society (PCI11WAT), St Lukes Institute for Cancer Research, the Health Research Board (HRA_POR / 2011 / 125), Movember GAP1 and the EU FP7 (MIAMI). The UCD Conway Institute is supported by the Program for Research in Third Level Institutions as administered by the Higher Education Authority of Ireland.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Belinda Hernández.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 5 KB)

Appendices

Appendix A: BART full conditional distribution \(p(R_j|X,T_j,\sigma ^{-2})\)

Using the forms of \(p(\mu _{ij})\) and \(p(\sigma ^{-2})\) described in Sect. 2.2.2 gives rise to the following full conditional distribution of the partial residuals for the BART model:

$$\begin{aligned} p(R_j|X,T_j,\sigma ^{-2})\propto & {} \prod _{i=1}^b\left( n_i\sigma ^{-2} +\left( \frac{0.5}{e\sqrt{m}}\right) ^{-2}\right) ^{-\frac{1}{2}}{\sigma ^{-2}}^{\frac{n+\nu }{2}-1}\nonumber \\&\times \exp \left( -\frac{\sigma ^{-2}}{2}\left( \sum _{\iota =1}^{n_i} R_{\iota ij}^2 +\nu \lambda \right) \right) \nonumber \\&\exp \left( \frac{n_i^2\bar{R}_{i j}^2\sigma ^{-4}}{2\left( n_i\sigma ^{-2}+\left( \frac{0.5}{e\sqrt{m}}\right) ^{-2}\right) } \right) , \end{aligned}$$
(11)

where \(n_i\) is the number of observations in terminal node i of tree j and \(\bar{R}_{ij}\) is the mean of the partial residuals \(R_j\) for terminal node i in tree j.

Appendix B: BART for classification

For binary classification, Chipman et al. (2010) follows the latent variable probit approach of Albert and Chib (1993). Latent variables \(Z_k\) are introduced so that

$$\begin{aligned} Y_k={\left\{ \begin{array}{ll} 1 &{}\text {if }Z_k>0 \\ 0 &{}\text {otherwise} . \end{array}\right. } \end{aligned}$$

The sum of trees prior is then placed on the \(Z_k\) so that \(Z_k \sim N\left( \sum _{j=1}^m g(x_k;T_j,M_j),1 \right) \). It follows that \(Y_k\) is Bernoulli with

$$\begin{aligned} P(Y_k=1|x_k) = \Phi \left( \sum _{j=1}^m g(x_k;T_j,M_j)\right) , \end{aligned}$$
(12)

where \(\Phi \) is the standard normal cumulative distribution function (CDF), used here as the link function. Note that there is no residual variance parameter \(\sigma ^2\) in the classification version of the model.

Using the same prior distribution structure as in Sect. 2.2.2, the full posterior distribution of this version of the model is:

$$\begin{aligned} \begin{aligned}&p(T,M,Z|X,Y) \propto p(Y|Z) p(Z|X,T,M)\\&\quad \times \left[ \prod _{j}\prod _{i}p(\mu _{ij}|T_j)p(T_j)\right] , \end{aligned} \end{aligned}$$
(13)

where the top level of the likelihood (i.e., the first term on the right-hand side) is a deterministic function of the latent variables. The conditional prior distributions of the terminal node parameters \(\mu _{ij}|T_j\) are set exactly as described in Sect. 2.2.2 except that \(\sigma _0=\frac{3}{e\sqrt{m}}\) instead of \(\sigma _0=\frac{0.5}{e\sqrt{m}}\). This is in order to assign high prior probability to the interval \((\Phi [-3],\Phi [3])\) which corresponds to the 0.1 and \(99.9\%\) quantiles of the normal CDF.

The fitting algorithm proposed by Chipman et al. (2010) for the classification model is nearly identical to that of their standard algorithm. The only difference is that the latent variables \(Z_k\) introduce an extra step in the Gibbs algorithm. The full conditional distributions of \(Z_k|\ldots \) are:

$$\begin{aligned} Z_k|\ldots \sim {\left\{ \begin{array}{ll} \min {\left[ N \left( \sum _j g(x_k;T_j,M_j),1 \right) ,0\right] } &{}\text {if }Y_k=1, \\ \max {\left[ N \left( \sum _j g(x_k;T_j,M_j),1 \right) , 0\right] } &{}\text {if }Y_k=0. \end{array}\right. } \end{aligned}$$
(14)

The partial residuals in the tree updates are of course now based on the latent variables \(Z_k\) for updating of individual trees.

Appendix C: BART-BMA algorithm overview

figureb

Appendix D: BART-BMA post hoc Gibbs sampler

In order to provide credible intervals for the point predictions, \(\hat{Y}\), provided by BART-BMA, we run a post hoc Gibbs sampler. For each sum of trees model \(\mathcal {T}_\ell \) in Occam’s window, a separate chain in the MCMC algorithm is run. For each model \(\mathcal {T}_\ell \), each terminal node parameter \(\mu _{ij}\) in each tree \(T_j\) is then updated followed by an update of \(\sigma ^{2}\). The details of the updates for the full conditional of \(p(\mu _{ij}|T_j,R_j,\sigma ^2)\) and of \(p(\sigma ^2)\) are explained in further detail in the following sections. The Gibbs sampler yields credible and prediction intervals for each set of sum of trees models accepted by BART-BMA along with the updates for \(\sigma ^{-2}\) for each set of trees accepted in the final BART-BMA model. The final simulated sample from the overall posterior distribution is obtained by selecting a number of iterations from the Gibbs sampler for each sum of trees model proportional to its posterior probability, and combining them. The post hoc Gibbs sampler used by BART-BMA is far less computationally expensive than that of BART as it requires only an update for \(\mu _{ij}\) and \(\sigma \) from the full conditional of each sum of trees model, which is merely a draw from a normal distribution and an inverse-Gamma distribution, respectively (see Sects. D.1, D.2, respectively).

Appendix D.1: Update of \(p(M_j|T_j,R_{\iota ji},\sigma ^2)\)

Let \(M_j = (\mu _{1j} \ldots \mu _{ij})\) index the \(b_j\) terminal node parameters of tree \(T_j\), and \(R_{kij}\) be the partial residuals for observations k belonging to terminal node i used as the response variable to grow tree \(T_j\). BART-BMA assumes that the prior on terminal node parameters is \(\mu _{ij}|T_j,\sigma \mathtt {\sim } N(0,\frac{\sigma ^2}{a})\), as in Chipman et al. (1998). The prior distribution of the partial residual is \(R_j|\ldots \mathtt {\sim }N(\mu _{ij},\sigma ^2)\).

The full conditional distribution of \(M_j\) is then

$$\begin{aligned} p(M_j| T_j,R_{kij},\sigma )&\propto p(R_{kij}|T_j,M_j,\sigma )p(M_j| T_j) \nonumber \\&\propto \prod _{k=1}^{n_i} p(R_{kij}|T_j,M_j,\sigma ) p(M_j |T_j) , \end{aligned}$$
(15)

where k indexes the observations within terminal node i of tree \(T_j\) and \(n_i\) refers to the number of observations which fall in terminal node i.

The draw from the full conditional of \(p(M_j|\ldots )\) is then a draw from the normal distribution

$$\begin{aligned} M_j|T_j,R_{kij},\sigma \mathtt {\sim } N\left( \frac{\sum _{k=1}^{n_i}{R_{\iota ij}}}{n_i+a},\frac{\sigma ^2}{n_i+a}\right) . \end{aligned}$$
(16)

The full conditional of \(M_j|\ldots \) depends only on \(\sigma \) in the variance parameter, making it slightly more efficient than the update of \(M_j\) using the BART prior which depends on \(\sigma \) in both the mean and variance parameter.

Appendix D.2: Update of \(p(\sigma ^2)\)

BART-BMA performs the update for \(p(\sigma )\) in the same way as (Chipman et al. 2010). The full conditional distribution of \(\sigma ^2\) is:

$$\begin{aligned} p(\sigma ^2|R_j,T_j,M_j) \propto \prod _{k=1}^n p\left( R_j|T_j,M_j,\sigma ^2 \right) p\left( \sigma ^2\right) ,\nonumber \\ \end{aligned}$$
(17)

where \(R_j \mathtt {\sim } N\left( \sum _{j=1}^m g(x_k,T_j,M_j ),\sigma ^2\right) \) and \(\frac{1}{\sigma ^2} \mathtt {\sim } \text{ Gamma }(\zeta ,\eta )\), where \(\zeta \) and \(\eta \) are equal to \(\frac{\nu }{2}\) and \(\frac{\nu \lambda }{2}\), respectively.

BART-BMA makes the draw for \(\sigma ^2\) in terms of the precision \(\sigma ^{-2}=\frac{1}{\sigma ^2}\) where \(p(\sigma ^{-2}|R_j,T_j,M_j )\) is calculated as:

$$\begin{aligned} \sigma ^{-2}|R_j,T_j,M_j&\mathtt {\sim } \text{ Gamma } \left( \zeta +\frac{1}{2} , \frac{P}{2} + \frac{1}{\eta }\right) , \end{aligned}$$
(18)

where \(P=\sum _{k}\left[ Y_k-\sum _{j}g(x_k,T_j,M_j)\right] ^2\) . The next value of \(\sigma ^{-2}\) is then drawn from (18), and the value of \(\sigma \) is calculated by getting the reciprocal square root of that value.

Appendix E: Greedy tree growing extra details

Appendix E.1: The PELT algorithm

Univariate changepoint detection algorithms in general search for distributional changes in an ordered series of data. For example, if normality is assumed then such an algorithm may look for changes in the mean or variance of the data. Searching for predictive split points for a single variable in a tree has an equivalent goal, i.e., it is desirable to find split points which maximise the separation of the response variable between the left- and right-hand daughter nodes. For this reason, we use a changepoint detection algorithm called PELT (Pruned Exact Linear Time) in BART-BMA to find predictive split points and greedily grow trees.

PELT was originally proposed to detect changepoints in an ordered series of data \(y_{1:n}=(y_1, \ldots , y_n)\) by minimising the function

$$\begin{aligned} \min _\delta \left[ \sum _{\theta =1}^{\Theta +1} \left[ C(y_{(\delta _{\theta -1}+1):\delta _{\theta }})+ D \right] \right] . \end{aligned}$$
(19)

Here, there are \(\Theta \) changepoints in the series at positions \(\delta _{1:\Theta }=(\delta _1, \ldots , \delta _\Theta )\) which results in \(\Theta +1\) segments. Each changepoint position \(\delta _{\theta }\) can take the value \(1 \ldots n-1\). For example, if a changepoint occurs at position \(\delta _1=5\) and another occurs at position \(\delta _2=12\), the second segment where \(\theta =2\) will contain the values for \(y_{(6:12)}\). The function \(C(\cdot )\) is a cost function of each segment \(\theta \) containing observations \(y_{(\delta _{\theta -1}+1):\delta _{\theta }}\). In the results which follow, the cost function used is twice the negative log likelihood assuming that y has a univariate normal distribution. Finally, D is a penalty for adding additional changepoints, default values for which are discussed below.

Table 8 Coverage for out-of-sample 50% prediction intervals and average interval width for BART-BMA, RF using conformal intervals bartMachine and dbarts for the Friedman example
Table 9 Coverage for out-of-sample 75% prediction intervals and average interval width for BART-BMA, RF using conformal prediction bartMachine and dbarts for the Friedman example

PELT extends the optimal partitioning method of Yao (1984) by eliminating any changepoints which cannot be optimal. This is achieved by observing that if there exists a candidate changepoint s where \(\delta<s<S\) which reduces the overall cost of the sequence, then the changepoint at \(\delta \) can never be optimal and so is removed from consideration (Killick et al. 2012). An algorithm describing how we use PELT to greedily grow trees is described in Algorithm 3:

figurec

One disadvantage of using PELT for large datasets is that the number of changepoint detected by the PELT algorithm is linearly related to the number of observations, which can reduce the speed of the BART-BMA algorithm for large n. Our experience is that \(D=10\log (n)\) performs well as a general default for the PELT penalty when \(n<200\). For larger values of n, we recommend using a higher value for D or the grid search option instead (see Sect. 3.4.2) in order to limit the number of split points detected per variable. We implement a version of PELT which is equivalent to the PELT.meanvar.norm function from the changepoint package in R (Killick et al. 2014). This function searches for changes in the mean and variance of variables which are assumed to be normally distributed. Additional changepoint are accepted if there is support for their inclusion according to the log likelihood ratio statistic.

Appendix E.2: Updating splitting rules

By default, we choose the best numcp% of the total splitting rules before the tree is grown and only trees using the most predictive splitting rules are considered for inclusion. However, the best splitting rules can also be updated for each internal node i in each tree \(T_j\), similarly to how RF creates trees. We have found that updating splitting rules at each internal node generally results in fewer trees \(T_j\) being included in each sum of trees model; however, each tree \(T_j\) within the sum of trees models averaged over in the final model tends to be deeper and to choose splits that are similar to the primary splits of trees in the RF. We have found that updating the splitting rules at each internal node can in some cases increase the predictive accuracy, but generally at the expense of computational speed.

Appendix F: Out of sample prediction intervals

This appendix shows the results for the calibration of the Friedman example using 50 and 75% prediction intervals (Tables 8, 9).

Fig. 4
figure4

Example of experiments to guide default value chosen to determine the size of Occam’s window \(o=1000\)

Fig. 5
figure5

Example of experiments conducted to guide default value chosen to determine the size of the PELT parameter pen where \(D=pen\log (n)\) given Occam’s window \(o=1000\)

Fig. 6
figure6

Example of experiments conducted to guide default value chosen to determine the numcp

Appendix G: Choice of default values for BART-BMA

This section will show some of the preliminary investigations which guided the choice of default settings for the BART-BMA algorithm such as the choice of the size of Occam’s window, the penalty on the PELT parameter and the size of sum of tree models to be averaged over. In the results that follow, the following datasets are shown: Ozone, Compactiv, Ankara and Baseball. These were also used as the benchmark datasets for the bartMachine package Kapelner and Bleich (2014b).

For each of the four datasets shown, varying amounts of random noise variables were appended to test the sensitivity of the parameters to the dimensionality of the dataset. In all, 17 different values for the number of random noise variables appended were tested ranging from 100 to 15, 000 so each parameter value of interest was run/tested a total of 68 times.

For the value of Occam’s window, 20 values were evaluated ranging from 100 to 100, 000. A contour plot showing the relative RMSE for the Baseball, Ankara, Compactiv and Ozone datasets can be seen in Fig. 4. Here, the RMSE value for each dataset has been divided by its minimum value which allows for fair comparison across datasets.

In general, we recommend a default value of \(OW=1000\) as it seems to work well on the majority of datasets tested as can be seen here (and in other datasets not shown). It was decided that the additional computational complexity involved in setting \(OW=10{,}000\) was not worth the marginal gain in accuracy for the datasets tested.

Figure 5 shows the same experiments conducted by ranging the multiple pen used in the PELT penalty \(D=pen\log (n)\) from 1 to 20. Here, we can see that a value of \(D=10\log (n)\) works well in the majority of the datasets shown.

Figure 6 shows the same datasets where Occam’s window is fixed at its default value of OW \(=\) 1000 and the PELT penalty parameter is fixed at \(D=10 \hbox {Log}(n)\). Here, 7 increments for numcp were chosen ranging from 5 to 100%.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hernández, B., Raftery, A.E., Pennington, S.R. et al. Bayesian Additive Regression Trees using Bayesian model averaging. Stat Comput 28, 869–890 (2018). https://doi.org/10.1007/s11222-017-9767-1

Download citation

Keywords

  • Bayesian Additive Regression Trees
  • Bayesian model averaging
  • Random forest
  • Biomarker selection
  • Small n large p