Abstract
Decision trees (DT) are highly famous in machine learning and usually acquire state-of-the-art performance. Despite that, well-known variants like CART, ID3, random forest, and boosted trees miss a probabilistic version that encodes prior assumptions about tree structures and shares statistical strength between node parameters. Existing work on Bayesian DT depends on Markov Chain Monte Carlo (MCMC), which can be computationally slow, especially on high dimensional data and expensive proposals. In this study, we propose a method to parallelise a single MCMC DT chain on an average laptop or personal computer that enables us to reduce its run-time through multi-core processing while the results are statistically identical to conventional sequential implementation. We also calculate the theoretical and practical reduction in run time, which can be obtained utilising our method on multi-processor architectures. Experiments showed that we could achieve 18 times faster running time provided that the serial and the parallel implementation are statistically identical.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Data availability statement
Data that support the findings of this study are available in a public repository : https://archive.ics.uci.edu/ml/index.php under the name Pima Indians Diabetes, Dermatology, Wine
References
Altekar, G., Dwarkadas, S., Huelsenbeck, J.P., Ronquist, F.: Parallel metropolis coupled Markov chain Monte Carlo for Bayesian phylogenetic inference. Bioinformatics 20(3), 407–415 (2004)
Basse, G., Smith, A., Pillai, N.: Parallel Markov chain Monte Carlo via spectral clustering. In: Gretton, A., Robert, C.C. (eds.) Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 1318–1327. Cadiz, Spain, 09–11 May( 2016). PMLR
Blanes, S., Casas, F., Sanz-Serna, J.M.: Numerical integrators for the hybrid monte carlo method. SIAM J. Sci. Comput. 36(4), A1556–A1580 (2014)
Byrd, J.M.R., Jarvis, S.A, Bhalerao, A.H.: Reducing the run-time of mcmc programs by multithreading on smp architectures. In: 2008 IEEE International Symposium on Parallel and Distributed Processing, pp. 1–8. IEEE ( 2008)
Byrd, J.M.R., Jarvis, S.A, Bhalerao, A.H.: Speculative moves: multithreading markov chain monte carlo programs. High-Performance Medical Image Computing and Computer Aided Intervention (HP-MICCAI), pp. 1-8. (2008)
Chipman, H.A., George, E.I, McCulloch, R.E.: Bart: Bayesian additive regression trees. Ann. Appl. Stat. 266–298 (2010)
Denison, D.G.T., Mallick, B.K., Smith, A.F.M.: A bayesian cart algorithm. Biometrika 85(2), 363–377 (1998)
Drousiotis, E., Pentaliotis, P., Shi, L., Cristea, A.I.: Capturing fairness and uncertainty in student dropout prediction-a comparison study. In: International Conference on Artificial Intelligence in Education, pp. 139–144. Springer, (2021)
Drousiotis, E., Shi, L., Maskell, S.: Early predictor for student success based on behavioural and demographical indicators. In: Intelligent Tutoring Systems: 17th International Conference, ITS 2021, Virtual Event, June 7–11, 2021, Proceedings 17, pp. 161–172. Springer, (2021)
Drousiotis, E., Spirakis, P.G., Maskell, S.: Parallel approaches to accelerate bayesian decision trees. arXiv preprint arXiv:2301.09090 (2023)
Duane, S., Kennedy, A.D., Pendleton, B.J., Roweth, D.: Hybrid monte carlo. Phys. Lett. B 195(2), 216–222 (1987)
Le Brazidec, J.D., Bocquet, M., Saunier, O., Roustan, Y.: Quantification of uncertainties in the assessment of an atmospheric release source applied to the autumn 2017 106 ru event. Atmos. Chem. Phys. 21(17), 13247–13267 (2021)
Fisher, L., Mcdonald, J.: 3-two-sample t-test. Fixed effects analysis of variance. Probability and mathematical statistics: a series of monographs and textbooks, pp. 21–35 (1978)
Foster, I.: Designing and building parallel programs: concepts and tools for parallel software engineering. Addison-Wesley Longman Publishing Co., Inc., (1995)
Francom, D., Sansó, B., Kupresanin, A., Johannesson, G.: Sensitivity analysis and emulation for functional data using bayesian adaptive splines. Stat. Sin. 28(2), 791–816 (2018). http://www.jstor.org/stable/44841925
Hafych, V., Eller, P., Schulz, O., Caldwel, A.: Parallelizing mcmc sampling via space partitioning. Stat. Comput. 32(4), 56 (2022)
Hastings, W.K.: Monte carlo sampling methods using markov chains and their applications. Biometrika 57(1), 97–109 (1970)
Herlihy, M., Shavit, N.: The Art of Multiprocessor Programming, revised first edition. Morgan Kaufmann, (2012)
Jacob, P.E., O’Leary, J., Atchadé, Y.F.: Unbiased markov chain monte carlo methods with couplings. J. R. Stat. Soc. Ser. B Stat. Methodol. 82(3), (2020)
Jakaite, L., Schetinin, V., Maple, C., Schult, J.: Bayesian decision trees for eeg assessment of newborn brain maturity. In: 2010 UK Workshop on Computational Intelligence (UKCI), pp. 1–6. IEEE, (2010)
Kruijver, M., Kelly, H., Cheng, K., Lin, M.-H., Morawitz, J., Russell, L., Buckleton, J., Bright, J.-A.: Estimating the number of contributors to a dna profile using decision trees. Forensic Sci. Int. Genet. 50, 102407 (2021)
Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–1092 (1953)
Mohammadi, R., Pratola, M., Kapteinm, M.: Continuous-time birth-death mcmc for bayesian regression tree models. arXiv preprint arXiv:1904.09339 (2019)
Mohammadi, R., Pratola, M., Kaptein, M.: Continuous-time birth-death mcmc for bayesian regression tree models. J. Mach. Learn. Res. 21(1), jan (2020)
Peskun, P.H.: Optimum monte-carlo sampling using markov chains. Biometrika 60(3), 607–612 (1973)
Pratola, M.T., Chipman, H.A., Gattiker, J.R., Higdon, D.M., McCulloch, R., Rust, W.N.: Parallel bayesian additive regression trees. J. Comput. Graph. Stat. 23(3), 830–852 (2014). http://www.jstor.org/stable/43304924
Robert, C.P., Elvira, V., Tawn, N., Wu, C.: Accelerating mcmc algorithms. Wiley Interdiscip. Rev. Comput. Stat. 10(5), e1435 (2018)
Rockova, V., Saha, E.: On theory for bart. In: The 22nd international conference on artificial intelligence and statistics, pp. 2839–2848. PMLR, (2019)
Scarpone, C., Brinkmann, S.T., Grosse, T., Sonnenwald, D., Fuchs, M., Walker, B.B.: A multimethod approach for county-scale geospatial analysis of emerging infectious diseases: a cross-sectional case study of covid-19 incidence in germany. Int. J. Health Geogr. 19(1), 1–17 (2020)
Schetinin, V., Fieldsend, J.E., Partridge, D., Krzanowski, W.J., Everson, R.M., Bailey, T.C., Hernandez, A.: The bayesian decision tree technique with a sweeping strategy. arXiv preprint cs/0504042 (2005)
Taylor, D., Bright, J.-A., Buckleton, J.: Interpreting forensic dna profiling evidence without specifying the number of contributors. Forensic Sci. Int. Genet. 13, 269–280 (2014)
Valderrama-Bahamondez, G.I., Frohlich, H.: Mcmc techniques for parameter estimation of ode based models in systems biology. Front. Math. Stat. 5, 55 (2019)
Vijayarangam, J., Kamalakkannan, S., Smiles, A.: A novel comparison of neural network and decision tree as classifiers using r. In: 2021 Fifth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), pp. 712–715. IEEE (2021)
Wilkinson, D.J.: Parallel bayesian computation. Stat. Textbooks. Monogr. 184, 477 (2006)
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
This work was supported by the DA-CDT Doctoral Training Centre. The authors have no competing interests to declare that are relevant to the content of this article. Conflict of Interest for all authors - None
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
A preliminary version of this paper was presented in the LION16 Conference held in Milos, Greece on 8/6/2022.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Drousiotis, E., Spirakis, P. Single MCMC chain parallelisation on decision trees. Ann Math Artif Intell (2023). https://doi.org/10.1007/s10472-023-09876-9
Accepted:
Published:
DOI: https://doi.org/10.1007/s10472-023-09876-9