Random Oracles for Regression Ensembles

Part of the Studies in Computational Intelligence book series (SCI, volume 373)


This paper considers the use of Random Oracles in Ensembles for regression tasks. A Random Oracle model (Kuncheva and Rodríguez, 2007) consists of a pair of models and a fixed randomly created “oracle” (in the case of the Linear Random Oracle, it is a hyperplane that divides the dataset in two during training and, once the ensemble is trained, decides which model to use). They can be used as the base model for any ensemble method. Previously, they have been used for classification. Here, the use of Random Oracles for regression is studied using 61 datasets, Regression Trees as base models and several ensemble methods: Bagging , Random Subspaces, AdaBoost.R2 and Iterated Bagging. For all the considered methods and variants, ensembles with Random Oracles are better than the corresponding version without the Oracles.


Root Mean Square Error Regression Tree Ensemble Member Average Rank Random Oracle 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Breiman, L.: Bagging predictors. Machine Learning 24, 123–140 (1996)MathSciNetzbMATHGoogle Scholar
  2. 2.
    Breiman, L.: Using iterated bagging to Debias regressions. Machine Learning 45, 261–277 (2001)CrossRefzbMATHGoogle Scholar
  3. 3.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Machine Learning Research 7, 1–30 (2006)Google Scholar
  4. 4.
    Dietterich, T.G.: Approximate statistical test for comparing supervised classification learning algorithms. Neural Comp. 10, 1895–1923 (1998)CrossRefGoogle Scholar
  5. 5.
    Drucker, H.: Improving regressors using boosting techniques. In: Fisher, D.H. (ed.) Proc. the 14th Int. Conf. Machine Learning, Nashville, TN, pp. 107–115. Morgan Kaufmann, San Francisco (1997)Google Scholar
  6. 6.
    Elomaa, T., Kääriäinen, M.: An analysis of reduced error pruning. J. Artif. Intell. Research 15, 163–187 (2001)zbMATHGoogle Scholar
  7. 7.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Saitta, L. (ed.) Proc. the 13th Int. Conf. Machine Learning, Bari, Italy, pp. 148–156. Morgan Kaufmann, San Francisco (1996)Google Scholar
  8. 8.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comp. and Syst. Sciences 55, 119–139 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: An update. SIGKDD Explorations 11, 10–18 (2009)CrossRefGoogle Scholar
  10. 10.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Analysis and Machine Intell. 20, 832–844 (1998)CrossRefGoogle Scholar
  11. 11.
    Kuncheva, L.I., Whitaker, C.J.: Measures of diversity in classifier ensembles. Machine Learning 51, 181–207 (2003)CrossRefzbMATHGoogle Scholar
  12. 12.
    Kuncheva, L.I.: Combining pattern classifiers: Methods and algorithms. John Wiley & Sons, Hoboken (2004)CrossRefzbMATHGoogle Scholar
  13. 13.
    Kuncheva, L.I., Rodríguez, J.J.: Classifier ensembles with a random linear oracle. IEEE Trans. Knowledge and Data Engineering 19, 500–508 (2007)CrossRefGoogle Scholar
  14. 14.
    Margineantu, D.D., Dietterich, T.G.: Pruning adaptive boosting. In: Fisher, D.H. (ed.) Proc. 14th Int. Conf. Machine Learning, Nashville, TN, pp. 211–218. Morgan Kaufmann, San Francisco (1997)Google Scholar
  15. 15.
    Rodríguez, J.J., García-Osorio, C., Maudes, J.: Forests of nested dichotomies. Pattern Recognition Letters 31, 125–132 (2010)CrossRefGoogle Scholar
  16. 16.
    Shrestha, D.L., Solomatine, D.P.: Experiments with AdaBoost.RT: An improved boosting scheme for regression. Neural Comp. 18, 1678–1710 (2006)CrossRefzbMATHGoogle Scholar
  17. 17.
    Suen, Y., Melville, P., Mooney, R.: Combining bias and variance reduction techniques for regression trees. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 741–749. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  18. 18.
    Wang, Y., Witten, I.H.: Induction of model trees for predicting continuous classes. In: van Someren, M., Widmer, G. (eds.) ECML 1997. LNCS, vol. 1224, pp. 128–137. Springer, Heidelberg (1997)Google Scholar
  19. 19.
    Witten, I.H., Frank, E.: Data mining: Practical machine learning tools and techniques. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  20. 20.
    Zhang, C., Zhang, J., Wang, G.: An empirical study of using Rotation Forest to improve regressors. Applied Mathematics and Comp. 195, 618–629 (2008)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.University of BurgosSpain

Personalised recommendations