Progress in Artificial Intelligence

, Volume 5, Issue 2, pp 91–103 | Cite as

Random feature weights for regression trees

  • Álvar Arnaiz-González
  • José F. Díez-Pastor
  • César García-Osorio
  • Juan J. Rodríguez
Regular Paper


Ensembles are learning methods the operation of which relies on a combination of different base models. The diversity of ensembles is a fundamental aspect that conditions their operation. Random Feature Weights (\({\mathcal {RFW}}\)) was proposed as a classification-tree ensemble construction method in which diversity is introduced into each tree by means of a random weight associated with each attribute. These weights vary from one tree to another in the ensemble. In this article, the idea of \({\mathcal {RFW}}\) is adapted to decision-tree regression. A comparison is drawn with other ensemble construction methods: Bagging, Random Forest, Iterated Bagging, Random Subspaces and AdaBoost.R2 obtaining competitive results.


Regression trees Ensembles Bagging Decision trees Random feature weights 


  1. 1.
    Asuncion, A., Newman, D.: UCI machine learning repository (2007)Google Scholar
  2. 2.
    Bhatnagar, V., Bhardwaj, M., Sharma, S., Haroon, S.: Accuracy diversity based pruning of classifier ensembles. Prog. Artif. Intell. 2(2–3), 97–111 (2014). doi:10.1007/s13748-014-0042-9 CrossRefGoogle Scholar
  3. 3.
    Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996). doi:10.1007/BF00058655 MathSciNetMATHGoogle Scholar
  4. 4.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). doi:10.1023/A:1010933404324 MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Breiman, L.: Using iterated bagging to debias regressions. Mach. Learn. 45(3), 261–277 (2001). doi:10.1023/A:1017934522171 MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)MathSciNetMATHGoogle Scholar
  7. 7.
    Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach. Learn. 40(2), 139–157 (2000). doi:10.1023/A:1007607513941 CrossRefGoogle Scholar
  8. 8.
    Diez-Pastor, J.F., Garcıa-Osorio, C., Rodríguez, J.J.: GRASP forest for regression: grasp metaheuristic applied to the construction of ensembles of regression trees. In: 14 Conferencia de la Asociación Espaola para la Inteligencia Artificial (CAEPIA’11) (2011)Google Scholar
  9. 9.
    Diez-Pastor, J.F., García-Osorio, C., Rodríguez, J.J., Bustillo, A.: Grasp forest: a new ensemble method for trees. In: Sansone, C., Kittler, J., Roli, F. (eds.) Multiple classifier systems. Lecture Notes in Computer Science, vol. 6713, pp. 66–75. Springer, Berlin (2011). doi:10.1007/978-3-642-21557-5_9
  10. 10.
    Drucker, H.: Improving regressors using boosting techniques. In: Proceedings of the 14th International Conference on Machine Learning (ICML 1997), pp. 107–115, Nashville, USA, 8–12 July 1997 (1997)Google Scholar
  11. 11.
    Elomaa, T., Kääriäinen, M.: An analysis of reduced error pruning. arXiv:1106.0668 (2011)
  12. 12.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine learning. Proceedings of the 13th International Conference (ICML ’96), pp. 148–156, Bari, 3–6 July 1996 (1996)Google Scholar
  13. 13.
    García-Osorio, C., de Haro-García, A., García-Pedrajas, N.: Democratic instance selection: a linear complexity instance selection algorithm based on classifier ensemble concepts. Artif. Intell. 174(5–6), 410–441 (2010). doi:10.1016/j.artint.2010.01.001 MathSciNetCrossRefGoogle Scholar
  14. 14.
    Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural Comput. 4(1), 1–58 (1992)CrossRefGoogle Scholar
  15. 15.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, IH.: The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009). doi:10.1145/1656274.1656278
  16. 16.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998). doi:10.1109/34.709601 CrossRefGoogle Scholar
  17. 17.
    Hochberg, Y.: A sharper bonferroni procedure for multiple tests of significance. Biometrika 75(4), 800–802 (1988). doi:10.1093/biomet/75.4.800 MathSciNetCrossRefMATHGoogle Scholar
  18. 18.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New York (2004)CrossRefMATHGoogle Scholar
  19. 19.
    Loh, W.Y.: Classification and regression trees. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 1(1), 14–23 (2011). doi:10.1002/widm.8 CrossRefGoogle Scholar
  20. 20.
    Maudes, J., Rodríguez, J.J., García-Osorio, C., García-Pedrajas, N.: Random feature weights for decision tree ensemble construction. Inf. Fusion 13(1), 20–30 (2012). doi:10.1016/j.inffus.2010.11.004 CrossRefGoogle Scholar
  21. 21.
    Mendes-Moreira, Ja, Soares, C., Jorge, A.M., Sousa, J.F.D.: Ensemble approaches for regression: a survey. ACM Comput. Surv. 45(1), 10:1–10:40 (2012). doi:10.1145/2379776.2379786 CrossRefMATHGoogle Scholar
  22. 22.
    Rooney, N., Patterson, D.W., Anand, S.S., Tsymbal, A.: Random subspacing for regression ensembles. In: Proceedings of the 17th International Florida Artificial Intelligence Research Society Conference, Miami Beach, pp. 532–537 (2004)Google Scholar
  23. 23.
    Salzberg, S.L.: C4.5: programs for machine learning by J. Ross Quinlan (Morgan Kaufmann Publishers, Inc., 1993). Mach. Learn. 16(3), 235–240 (1994). doi:10.1007/BF00993309 MathSciNetGoogle Scholar
  24. 24.
    Seiffert, C., Khoshgoftaar, T.M., Van Hulse, J., Napolitano, A.: Resampling or reweighting: a comparison of boosting implementations. In: 20th IEEE International Conference on Tools with Artificial Intelligence, 2008, ICTAI’08, vol. 1, pp. 445–451. IEEE, New York (2008)Google Scholar
  25. 25.
    Solomatine, D., Shrestha, D.: AdaBoost.RT: a boosting algorithm for regression problems. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, vol. 2, pp. 1163–1168 (2004). doi:10.1109/IJCNN.2004.1380102

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Álvar Arnaiz-González
    • 1
  • José F. Díez-Pastor
    • 1
  • César García-Osorio
    • 1
  • Juan J. Rodríguez
    • 1
  1. 1.University of BurgosBurgosSpain

Personalised recommendations