Machine Learning

, Volume 108, Issue 8–9, pp 1503–1522 | Cite as

On PAC-Bayesian bounds for random forests

  • Stephan S. Lorenzen
  • Christian IgelEmail author
  • Yevgeny Seldin
Part of the following topical collections:
  1. Special Issue of the ECML PKDD 2019 Journal Track


Existing guarantees in terms of rigorous upper bounds on the generalization error for the original random forest algorithm, one of the most frequently used machine learning methods, are unsatisfying. We discuss and evaluate various PAC-Bayesian approaches to derive such bounds. The bounds do not require additional hold-out data, because the out-of-bag samples from the bagging in the training process can be exploited. A random forest predicts by taking a majority vote of an ensemble of decision trees. The first approach is to bound the error of the vote by twice the error of the corresponding Gibbs classifier (classifying with a single member of the ensemble selected at random). However, this approach does not take into account the effect of averaging out of errors of individual classifiers when taking the majority vote. This effect provides a significant boost in performance when the errors are independent or negatively correlated, but when the correlations are strong the advantage from taking the majority vote is small. The second approach based on PAC-Bayesian \(C\)-bounds takes dependencies between ensemble members into account, but it requires estimating correlations between the errors of the individual classifiers. When the correlations are high or the estimation is poor, the bounds degrade. In our experiments, we compute generalization bounds for random forests on various benchmark data sets. Because the individual decision trees already perform well, their predictions are highly correlated and the \(C\)-bounds do not lead to satisfactory results. For the same reason, the bounds based on the analysis of Gibbs classifiers are typically superior and often reasonably tight. Bounds based on a validation set coming at the cost of a smaller training set gave better performance guarantees, but worse performance in most experiments.


PAC-Bayesian analysis Random forests Majority vote 



We acknowledge support by the Innovation Fund Denmark through the Danish Center for Big Data Analytics Driven Innovation (DABAI).

Supplementary material


  1. Andersen, M., & Vandenberghe, L. (2019). CVXOPT. Retrieved May, 2019 from
  2. Arlot, S., & Genuer, R. (2014). Analysis of purely random forests bias. arXiv:1407.3939.
  3. Biau, G. (2012). Analysis of a random forests model. Journal of Machine Learning Research, 13(1), 1063–1095.MathSciNetzbMATHGoogle Scholar
  4. Biau, G., Devroye, L., & Lugosi, G. (2008). Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research, 9, 2015–2033.MathSciNetzbMATHGoogle Scholar
  5. Breiman, L. (1996a). Bagging predictors. Machine Learning, 24(2), 123–140.zbMATHGoogle Scholar
  6. Breiman, L. (1996b). Out-of-bag estimation. Retrieved May, 2019 from
  7. Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5–32.CrossRefzbMATHGoogle Scholar
  8. Breiman, L. (2002). Some infinity theory for predictor ensembles. Journal of Combinatorial Theory A, 98, 175–191.CrossRefGoogle Scholar
  9. Denil, M., Matheson, D., & Freitas, N. D. (2014). Narrowing the gap: Random forests in theory and in practice. In Proceedings of the 31st international conference on machine learning (ICML), PMLR, Proceedings of machine learning research (Vol. 32, pp. 665–673).Google Scholar
  10. Fernández-Delgado, M., Cernadas, E., Barro, S., & Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research, 15, 3133–3181.MathSciNetzbMATHGoogle Scholar
  11. Genuer, R. (2010). Risk bounds for purely uniformly random forests. Technical report. France: Institut National de Recherche en Informatique et en Automatique.Google Scholar
  12. Germain, P., Lacasse, A., Laviolette, F., Marchand, M., & Roy, J. F. (2015). Risk bounds for the majority vote: From a PAC-Bayesian analysis to a learning algorithm. Journal of Machine Learning Research, 16, 787–860.MathSciNetzbMATHGoogle Scholar
  13. Geurts, P., Ernst, D., & Wehenkel, L. (2006). Extremely randomized trees. Machine Learning, 63(1), 3–42.CrossRefzbMATHGoogle Scholar
  14. Gieseke, F., & Igel, C. (2018). Training big random forests with little resources. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery and data mining (KDD) (pp. 1445–1454). ACM Press.Google Scholar
  15. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). Berlin: Springer.CrossRefzbMATHGoogle Scholar
  16. Langford, J., & Shawe-Taylor, J. (2002). PAC-Bayes & Margins. In Proceedings of the 15th international conference on neural information processing systems (pp. 439–446). MIT Press.Google Scholar
  17. Maurer, A. (2004). A note on the PAC-Bayesian theorem. arXiv:cs/0411099.
  18. Mcallester, D. (2003). Simplified PAC-Bayesian margin bounds. In Proceedings of the 16th annual conference on computational learning theory (COLT). LNCS (Vol. 2777, pp. 203–215). Springer.Google Scholar
  19. McAllester, D. A. (1998). Some PAC-Bayesian theorems. In Proceedings of the eleventh annual conference on computational learning theory (COLT) (pp. 230–234). ACM.Google Scholar
  20. McAllester, D. A. (1999). PAC-Bayesian model averaging. In Proceedings of the twelfth annual conference on computational learning theory (COLT) (pp. 164–170). ACM.Google Scholar
  21. Oneto, L., Cipollini, F., Ridella, S., & Anguita, D. (2018). Randomized learning: Generalization performance of old and new theoretically grounded algorithms. Neurocomputing, 298, 21–33.CrossRefGoogle Scholar
  22. Schapire, R. E., & Singer, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37(3), 297–336.CrossRefzbMATHGoogle Scholar
  23. Seeger, M. (2002). PAC-Bayesian generalization error bounds for Gaussian process classification. Journal of Machine Learning Research, 3, 233–269.CrossRefzbMATHGoogle Scholar
  24. Thiemann, N., Igel, C., Wintenberger, O., & Seldin, Y. (2017). A strongly quasiconvex PAC-Bayesian bound. In Proceedings of the international conference on algorithmic learning theory (ALT), PMLR, Proceedings of machine learning research (Vol. 76, pp. 466–492).Google Scholar
  25. Valiant, L. G. (1984). A theory of the learnable. Communications of the ACM, 27(11), 1134–1142.CrossRefzbMATHGoogle Scholar
  26. Wang, Y., Tang, Q., Xia, S.T., Wu, J., & Zhu, X. (2016). Bernoulli random forests: Closing the gap between theoretical consistency and empirical soundness. In Proceedings of the 17th international conference on machine learning (ICML) (pp. 2167–2173). Morgan Kaufmann.Google Scholar

Copyright information

© The Author(s), under exclusive licence to Springer Science+Business Media LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations