Dynamic Integration with Random Forests

  • Alexey Tsymbal
  • Mykola Pechenizkiy
  • Pádraig Cunningham
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4212)


Random Forests (RF) are a successful ensemble prediction technique that uses majority voting or averaging as a combination function. However, it is clear that each tree in a random forest may have a different contribution in processing a certain instance. In this paper, we demonstrate that the prediction performance of RF may still be improved in some domains by replacing the combination function with dynamic integration, which is based on local performance estimates. Our experiments also demonstrate that the RF Intrinsic Similarity is better than the commonly used Heterogeneous Euclidean/Overlap Metric in finding a neighbourhood for local estimates in the context of dynamic integration of classification random forests.


Random Forest Majority Vote Local Performance Base Classifier Combination Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bauer, E., Kohavi, R.: An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning 36(1,2), 105–139 (1999)CrossRefGoogle Scholar
  2. 2.
    Bingham, E., Mannila, H.: Random projection in dimensionality reduction: applications to image and text data. In: Proc. 7th ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining KDD 2001, pp. 245–250. ACM Press, New York (2001)CrossRefGoogle Scholar
  3. 3.
    Blake C.L., Keogh, E., Merz, C.J.: UCI repository of machine learning databases. Dept. of Information and Computer Science, University of California, Irvine, CA (1999),
  4. 4.
    Breiman, L.: Bias, Variance, and Arcing Classifiers, Tech. Report 486, Statistics Dept., University of California, Berkeley, USA (1996)Google Scholar
  5. 5.
    Breiman, L.: Random Forests. Machine Learning 45(1), 5–32 (2001)MATHCrossRefGoogle Scholar
  6. 6.
    Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proc. 13th Int. Conf. on Machine Learning, pp. 275–283. Morgan Kaufmann, San Francisco (1996)Google Scholar
  7. 7.
    Robnik-Šikonja, M.: Improving random forests. In: Boulicaut, J.-F., Esposito, F., Giannotti, F., Pedreschi, D. (eds.) ECML 2004. LNCS, vol. 3201, pp. 359–370. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Rooney, N., Patterson, D., Anand, S., Tsymbal, A.: Dynamic integration of regression models. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 164–173. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Schaffer, C.: Selecting a classification method by cross-validation. Machine Learning 13, 135–143 (1993)Google Scholar
  10. 10.
    Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Sequential genetic search for ensemble feature selection. In: Proc. 19th Int. Joint Conf. on Artificial Intelligence IJCAI 2005, pp. 877–882. Morgan Kaufmann, San Francisco (2005)Google Scholar
  11. 11.
    Tsymbal A., Pechenizkiy M., Cunningham P.: Dynamic integration with random forests. Tech. Report TCD-CS-2006-23, Dept. of Computer Science, Trinity College Dublin, Ireland (2006), available online at:
  12. 12.
    Tsymbal, A., Puuronen, S.: Bagging and boosting with dynamic integration of classifiers. In: Zighed, D.A., Komorowski, J., Żytkow, J.M. (eds.) PKDD 2000. LNCS, vol. 1910, pp. 116–125. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  13. 13.
    Wilson, D.R., Martinez, T.R.: Improved heterogeneous distance functions. Journal of Artificial Intelligence Research 6(1), 1–34 (1997)MATHMathSciNetGoogle Scholar
  14. 14.
    Witten, I., Frank, E.: Data Mining: Practical Machine Learning Tools With Java Implementations. Morgan Kaufmann, San Francisco (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Alexey Tsymbal
    • 1
  • Mykola Pechenizkiy
    • 2
  • Pádraig Cunningham
    • 1
  1. 1.Dept of Computer ScienceTrinity College DublinIreland
  2. 2.Dept of Math ITUniversity of JyväskyläFinland

Personalised recommendations