Ensemble Learning: A Study on Different Variants of the Dynamic Selection Approach

  • João Mendes-Moreira
  • Alipio Mario Jorge
  • Carlos Soares
  • Jorge Freire de Sousa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5632)


Integration methods for ensemble learning can use two different approaches: combination or selection. The combination approach (also called fusion) consists on the combination of the predictions obtained by different models in the ensemble to obtain the final ensemble prediction. The selection approach selects one (or more) models from the ensemble according to the prediction performance of these models on similar data from the validation set. Usually, the method to select similar data is the k-nearest neighbors with the Euclidean distance. In this paper we discuss other approaches to obtain similar data for the regression problem. We show that using similarity measures according to the target values improves results. We also show that selecting dynamically several models for the prediction task increases prediction accuracy comparing to the selection of just one model.


Random Forest Ensemble Learn Ensemble Generation Ensemble Prediction Binary Search Tree 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arya, S., Mount, D.M., Netanyahu, N.S., Silverman, R., Wu, A.Y.: An optimal algorithm for approximate nearest neighbor searching. Journal of the ACM 45(6), 891–923 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Communications of the ACM 18(9), 509–517 (1975)CrossRefzbMATHGoogle Scholar
  3. 3.
    Breiman, L.: Bagging predictors. Machine Learning 26, 123–140 (1996)zbMATHGoogle Scholar
  4. 4.
    Breiman, L.: Random forests. Machine Learning 45, 5–32 (2001)CrossRefzbMATHGoogle Scholar
  5. 5.
    Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and regression trees. Chapman and Hall/CRC, Boca Raton (1984)zbMATHGoogle Scholar
  6. 6.
    Caruana, R., Niculescu-Mozil, A., Crew, G., Ksikes, A.: Ensemble selection from libraries of models. In: International Conference on Machine Learning (2004)Google Scholar
  7. 7.
    Didaci, L., Giacinto, G.: Dynamic classifier selection by adaptive k-nearest neighbourhood rule. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 174–183. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural computation 10, 1895–1923 (1998)CrossRefGoogle Scholar
  9. 9.
    Freund, Y., Schapire, R.: Experiments with a new boosting algorithm. In: International Conference on Machine Learning, pp. 148–156 (1996)Google Scholar
  10. 10.
    García-Pedrajas, N., Hervás-Martínez, C., Ortiz-Boyer, D.: Cooperative coevolution of artificial neural network ensembles for pattern classification. IEEE Transactions on Evolutionary Computation 9(3), 271–302 (2005)CrossRefGoogle Scholar
  11. 11.
    Giacinto, G., Roli, F.: Adaptive selection of image classifiers. In: Del Bimbo, A. (ed.) ICIAP 1997. LNCS, vol. 1310, pp. 38–45. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  12. 12.
    Hastie, T., Tibshirani, R.: Discriminant adaptive nearest neighbor classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(6), 607–616 (1996)CrossRefGoogle Scholar
  13. 13.
    Kemp, S.E.: knnfinder: Fast near neighbour search. R package version 1.0Google Scholar
  14. 14.
    Ko, A.H.-R., Sabourin, R., Britto Jr., A.d.S.: From dynamic classifier selection to dynamic ensemble selection. Pattern Recognition 41, 1718–1731 (2008)CrossRefzbMATHGoogle Scholar
  15. 15.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Advances in Neural Information Processing Systems, vol. 7, pp. 231–238 (1995)Google Scholar
  16. 16.
    Kuncheva, L.I.: Switching between selection and fusion in combining classifiers: an experiment. IEEE Transactions on Systems, Man, and Cybernetics-Part B 32(2), 146–156 (2002)CrossRefGoogle Scholar
  17. 17.
    Lilliefors, H.W.: On the kolmogorov-smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association 62(318), 399–402 (1967)CrossRefGoogle Scholar
  18. 18.
    Liu, Y., Yao, X., Higuchi, T.: Evolutionary ensembles with negative correlation learning. IEEE Transactions on Evolutionary Computation 4(4), 380–387 (2000)CrossRefGoogle Scholar
  19. 19.
    Merz, C.J.: Dynamical selection of learning algorithms. In: Fisher, D., Lenz, H.-J. (eds.) International Workshop on Artificial Intelligence and Statistics. Learning from Data: Artificial Intelligence and Statistics, vol. V. Springer, Heidelberg (1996)Google Scholar
  20. 20.
    Merz, C.J.: Classification and regression by combining models. Phd thesis, University of California, USA (1998)Google Scholar
  21. 21.
    Prudêncio, R.B.C., Ludermir, T.B.: Meta-learning approaches to selecting time series models. Neurocomputing 61, 121–137 (2004)CrossRefGoogle Scholar
  22. 22.
    Puuronen, S., Terziyan, V., Tsymbal, A.: A dynamic integration algorithm for an ensemble of classifiers. In: Raś, Z.W., Skowron, A. (eds.) ISMIS 1999. LNCS, vol. 1609, pp. 592–600. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  23. 23.
    Robnik-Šikonja, M., Kononenko, I.: Theoretical and empirical analysis of relieff and rrelieff. Machine Learning 53(1-2), 23–69 (2003)CrossRefzbMATHGoogle Scholar
  24. 24.
    Rooney, N., Patterson, D., Anand, S., Tsymbal, A.: Dynamic integration of regression models. In: Roli, F., Kittler, J., Windeatt, T. (eds.) MCS 2004. LNCS, vol. 3077, pp. 164–173. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  25. 25.
    R. D. C. Team. R: A language and environment for statistical computing. Technical report, R Foundation for Statistical Computing (2006) ISBN 3-900051-07-0Google Scholar
  26. 26.
    Torgo, L.: Regression data repositoryGoogle Scholar
  27. 27.
    Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Dynamic integration with random forests. Tech. Report TCD-CS-2006-23, The University of Dublin, Trinity College (2006)Google Scholar
  28. 28.
    Tsymbal, A., Pechenizkiy, M., Cunningham, P.: Dynamic integration with random forests. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 801–808. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  29. 29.
    Tsymbal, A., Puuronen, S.: Bagging and boosting with dynamic integration of classifiers. In: Zighed, D.A., Komorowski, J., Żytkow, J.M. (eds.) PKDD 2000. LNCS (LNAI), vol. 1910, pp. 116–125. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  30. 30.
    Woods, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(4), 405–410 (1997)CrossRefGoogle Scholar
  31. 31.
    Xycoon. Statistics - econometrics - forecastingGoogle Scholar
  32. 32.
    Yankov, D., DeCoste, D., Keogh, E.: Ensembles of nearest neighbor forecasts. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds.) ECML 2006. LNCS (LNAI), vol. 4212, pp. 545–556. Springer, Heidelberg (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • João Mendes-Moreira
    • 1
    • 2
  • Alipio Mario Jorge
    • 2
    • 3
  • Carlos Soares
    • 2
    • 4
  • Jorge Freire de Sousa
    • 5
  1. 1.Faculdade de Engenharia, Universidade do Porto, DEIPortugal
  2. 2.LIAAD-INESC Porto L.A.Portugal
  3. 3.Faculdade de Ciências, Universidade do PortoPortugal
  4. 4.Faculdade de Economia, Universidade do PortoPortugal
  5. 5.Faculdade de Engenharia, Universidade do Porto, DEIGPortugal

Personalised recommendations