Progress in Artificial Intelligence

, Volume 5, Issue 2, pp 91–103 | Cite as

Random feature weights for regression trees

  • Álvar Arnaiz-González
  • José F. Díez-Pastor
  • César García-Osorio
  • Juan J. Rodríguez
Regular Paper


Ensembles are learning methods the operation of which relies on a combination of different base models. The diversity of ensembles is a fundamental aspect that conditions their operation. Random Feature Weights (\({\mathcal {RFW}}\)) was proposed as a classification-tree ensemble construction method in which diversity is introduced into each tree by means of a random weight associated with each attribute. These weights vary from one tree to another in the ensemble. In this article, the idea of \({\mathcal {RFW}}\) is adapted to decision-tree regression. A comparison is drawn with other ensemble construction methods: Bagging, Random Forest, Iterated Bagging, Random Subspaces and AdaBoost.R2 obtaining competitive results.


Regression trees Ensembles Bagging Decision trees Random feature weights 



This work was funded by the Ministry of Economy and Competitiveness, project TIN 2011-24046.


  1. 1.
    Asuncion, A., Newman, D.: UCI machine learning repository (2007)Google Scholar
  2. 2.
    Bhatnagar, V., Bhardwaj, M., Sharma, S., Haroon, S.: Accuracy diversity based pruning of classifier ensembles. Prog. Artif. Intell. 2(2–3), 97–111 (2014). doi: 10.1007/s13748-014-0042-9 CrossRefGoogle Scholar
  3. 3.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996). doi: 10.1007/BF00058655 MathSciNetzbMATHGoogle Scholar
  4. 4.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). doi: 10.1023/A:1010933404324 MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Breiman, L.: Using iterated bagging to debias regressions. Mach. Learn. 45(3), 261–277 (2001). doi: 10.1023/A:1017934522171 MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach. Learn. 40(2), 139–157 (2000). doi: 10.1023/A:1007607513941 CrossRefGoogle Scholar
  8. 8.
    Diez-Pastor, J.F., Garcıa-Osorio, C., Rodríguez, J.J.: GRASP forest for regression: grasp metaheuristic applied to the construction of ensembles of regression trees. In: 14 Conferencia de la Asociación Espaola para la Inteligencia Artificial (CAEPIA’11) (2011)Google Scholar
  9. 9.
    Diez-Pastor, J.F., García-Osorio, C., Rodríguez, J.J., Bustillo, A.: Grasp forest: a new ensemble method for trees. In: Sansone, C., Kittler, J., Roli, F. (eds.) Multiple classifier systems. Lecture Notes in Computer Science, vol. 6713, pp. 66–75. Springer, Berlin (2011). doi: 10.1007/978-3-642-21557-5_9
  10. 10.
    Drucker, H.: Improving regressors using boosting techniques. In: Proceedings of the 14th International Conference on Machine Learning (ICML 1997), pp. 107–115, Nashville, USA, 8–12 July 1997 (1997)Google Scholar
  11. 11.
    Elomaa, T., Kääriäinen, M.: An analysis of reduced error pruning. arXiv:1106.0668 (2011)
  12. 12.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine learning. Proceedings of the 13th International Conference (ICML ’96), pp. 148–156, Bari, 3–6 July 1996 (1996)Google Scholar
  13. 13.
    García-Osorio, C., de Haro-García, A., García-Pedrajas, N.: Democratic instance selection: a linear complexity instance selection algorithm based on classifier ensemble concepts. Artif. Intell. 174(5–6), 410–441 (2010). doi: 10.1016/j.artint.2010.01.001 MathSciNetCrossRefGoogle Scholar
  14. 14.
    Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural Comput. 4(1), 1–58 (1992)CrossRefGoogle Scholar
  15. 15.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, IH.: The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009). doi: 10.1145/1656274.1656278
  16. 16.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998). doi: 10.1109/34.709601 CrossRefGoogle Scholar
  17. 17.
    Hochberg, Y.: A sharper bonferroni procedure for multiple tests of significance. Biometrika 75(4), 800–802 (1988). doi: 10.1093/biomet/75.4.800 MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New York (2004)CrossRefzbMATHGoogle Scholar
  19. 19.
    Loh, W.Y.: Classification and regression trees. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 1(1), 14–23 (2011). doi: 10.1002/widm.8 CrossRefGoogle Scholar
  20. 20.
    Maudes, J., Rodríguez, J.J., García-Osorio, C., García-Pedrajas, N.: Random feature weights for decision tree ensemble construction. Inf. Fusion 13(1), 20–30 (2012). doi: 10.1016/j.inffus.2010.11.004 CrossRefGoogle Scholar
  21. 21.
    Mendes-Moreira, Ja, Soares, C., Jorge, A.M., Sousa, J.F.D.: Ensemble approaches for regression: a survey. ACM Comput. Surv. 45(1), 10:1–10:40 (2012). doi: 10.1145/2379776.2379786 CrossRefzbMATHGoogle Scholar
  22. 22.
    Rooney, N., Patterson, D.W., Anand, S.S., Tsymbal, A.: Random subspacing for regression ensembles. In: Proceedings of the 17th International Florida Artificial Intelligence Research Society Conference, Miami Beach, pp. 532–537 (2004)Google Scholar
  23. 23.
    Salzberg, S.L.: C4.5: programs for machine learning by J. Ross Quinlan (Morgan Kaufmann Publishers, Inc., 1993). Mach. Learn. 16(3), 235–240 (1994). doi: 10.1007/BF00993309 MathSciNetGoogle Scholar
  24. 24.
    Seiffert, C., Khoshgoftaar, T.M., Van Hulse, J., Napolitano, A.: Resampling or reweighting: a comparison of boosting implementations. In: 20th IEEE International Conference on Tools with Artificial Intelligence, 2008, ICTAI’08, vol. 1, pp. 445–451. IEEE, New York (2008)Google Scholar
  25. 25.
    Solomatine, D., Shrestha, D.: AdaBoost.RT: a boosting algorithm for regression problems. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, vol. 2, pp. 1163–1168 (2004). doi: 10.1109/IJCNN.2004.1380102

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Álvar Arnaiz-González
    • 1
  • José F. Díez-Pastor
    • 1
  • César García-Osorio
    • 1
  • Juan J. Rodríguez
    • 1
  1. 1.University of BurgosBurgosSpain

Personalised recommendations