Bayesian Additive Regression Trees using Bayesian model averaging
Bayesian Additive Regression Trees (BART) is a statistical sum of trees model. It can be considered a Bayesian version of machine learning tree ensemble methods where the individual trees are the base learners. However, for datasets where the number of variables p is large the algorithm can become inefficient and computationally expensive. Another method which is popular for high-dimensional data is random forests, a machine learning algorithm which grows trees using a greedy search for the best split points. However, its default implementation does not produce probabilistic estimates or predictions. We propose an alternative fitting algorithm for BART called BART-BMA, which uses Bayesian model averaging and a greedy search algorithm to obtain a posterior distribution more efficiently than BART for datasets with large p. BART-BMA incorporates elements of both BART and random forests to offer a model-based algorithm which can deal with high-dimensional data. We have found that BART-BMA can be run in a reasonable time on a standard laptop for the “small n large p” scenario which is common in many areas of bioinformatics. We showcase this method using simulated data and data from two real proteomic experiments, one to distinguish between patients with cardiovascular disease and controls and another to classify aggressive from non-aggressive prostate cancer. We compare our results to their main competitors. Open source code written in R and Rcpp to run BART-BMA can be found at: https://github.com/BelindaHernandez/BART-BMA.git.
KeywordsBayesian Additive Regression Trees Bayesian model averaging Random forest Biomarker selection Small n large p
We would like to thank Drs Chris Watson, John Baugh, Mark Ledwidge and Professor Kenneth McDonald for kindly allowing us to use the cardiovascular dataset described. Hernández’s research was supported by the Irish Research Council. Raftery’s research was supported by NIH Grants Nos. R01-HD054511, R01-HD070936, and U54-HL127624, and by a Science Foundation Ireland E.T.S. Walton visitor award, Grant Reference 11/W.1/I2079. Protein biomarker discovery work in the Pennington Biomedical Proteomics Group is supported by grants from Science Foundation Ireland (for mass spectrometry instrumentation), the Irish Cancer Society (PCI11WAT), St Lukes Institute for Cancer Research, the Health Research Board (HRA_POR / 2011 / 125), Movember GAP1 and the EU FP7 (MIAMI). The UCD Conway Institute is supported by the Program for Research in Third Level Institutions as administered by the Higher Education Authority of Ireland.
- Chipman, H., McCulloch, R., Dorie, V.: Package dbarts (2014). https://cran.r-project.org/web/packages/dbarts/dbarts.pdf
- Cortes, I.: Package conformal (2014). https://cran.r-project.org/web/packages/conformal/conformal.pdf
- Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001). doi: 10.1214/aos/1013203451
- Harris, K., Girolami, M., Mischak, H.: Pattern Recognition in Bioinformatics, Lecture Notes in Computer Science, chap. Definition of Valid Proteomic Biomarkers: A Bayesian Solution, pp. 137–149. Springer, Berlin (2009)Google Scholar
- Kapelner, A., Bleich, J.: bartmachine: machine learning with Bayesian additive regression trees. ArXiv e-prints (2014a)Google Scholar
- Kapelner, A., Bleich, J.: Package bartMachine (2014b). http://cran.r-project.org/web/packages/bartMachine/bartMachine.pdf
- Killick, R., Eckley, I., Haynes, K., Fearnhead, P.: Package changepoint (2014). http://cran.r-project.org/web/packages/changepoint/changepoint.pdf
- Lakshminarayanan, B., Roy, D.M., Teh, Y.W.: Particle Gibbs for Bayesian additive regression trees. arXiv preprint arXiv:1502.04622 (2015)
- Lakshminarayanan, B., Roy, D.M., Teh, Y.W.: Mondrian forests for large-scale regression when uncertainty matters. In: Artificial Intelligence and Statistics, pp. 1478–1487. (arXiv:1506.03805, 2015) (2016)
- Liaw, A., Matthew, W.: Package randomForest (2015). http://cran.r-project.org/web/packages/randomForest/randomForest.pdf
- Morgan, J.N.: History and potential of binary segmentation for exploratory data analysis. J. Data Sci. 3, 123–136 (2005)Google Scholar
- Quinlan, J.R.: Discovering rules by induction from large collections of examples. In: Michie, D. (ed.) Expert Systems in the Micro Electronic Age. Edinburgh University Press, Edinburgh (1979)Google Scholar