Abstract
Ensembles are learning methods the operation of which relies on a combination of different base models. The diversity of ensembles is a fundamental aspect that conditions their operation. Random Feature Weights (\({\mathcal {RFW}}\)) was proposed as a classification-tree ensemble construction method in which diversity is introduced into each tree by means of a random weight associated with each attribute. These weights vary from one tree to another in the ensemble. In this article, the idea of \({\mathcal {RFW}}\) is adapted to decision-tree regression. A comparison is drawn with other ensemble construction methods: Bagging, Random Forest, Iterated Bagging, Random Subspaces and AdaBoost.R2 obtaining competitive results.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Unstable refers to those methods that produce very different models with small changes in the training set or in the parameters of the method.
The recursive function is for numeric attributes, for nominal ones the algorithm creates as many children as different values has the attribute.
References
Asuncion, A., Newman, D.: UCI machine learning repository (2007)
Bhatnagar, V., Bhardwaj, M., Sharma, S., Haroon, S.: Accuracy diversity based pruning of classifier ensembles. Prog. Artif. Intell. 2(2–3), 97–111 (2014). doi:10.1007/s13748-014-0042-9
Breiman, L.: Bagging predictors. Mach. Learn. 24(2), 123–140 (1996). doi:10.1007/BF00058655
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001). doi:10.1023/A:1010933404324
Breiman, L.: Using iterated bagging to debias regressions. Mach. Learn. 45(3), 261–277 (2001). doi:10.1023/A:1017934522171
Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)
Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization. Mach. Learn. 40(2), 139–157 (2000). doi:10.1023/A:1007607513941
Diez-Pastor, J.F., Garcıa-Osorio, C., Rodríguez, J.J.: GRASP forest for regression: grasp metaheuristic applied to the construction of ensembles of regression trees. In: 14 Conferencia de la Asociación Espaola para la Inteligencia Artificial (CAEPIA’11) (2011)
Diez-Pastor, J.F., García-Osorio, C., Rodríguez, J.J., Bustillo, A.: Grasp forest: a new ensemble method for trees. In: Sansone, C., Kittler, J., Roli, F. (eds.) Multiple classifier systems. Lecture Notes in Computer Science, vol. 6713, pp. 66–75. Springer, Berlin (2011). doi:10.1007/978-3-642-21557-5_9
Drucker, H.: Improving regressors using boosting techniques. In: Proceedings of the 14th International Conference on Machine Learning (ICML 1997), pp. 107–115, Nashville, USA, 8–12 July 1997 (1997)
Elomaa, T., Kääriäinen, M.: An analysis of reduced error pruning. arXiv:1106.0668 (2011)
Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Machine learning. Proceedings of the 13th International Conference (ICML ’96), pp. 148–156, Bari, 3–6 July 1996 (1996)
García-Osorio, C., de Haro-García, A., García-Pedrajas, N.: Democratic instance selection: a linear complexity instance selection algorithm based on classifier ensemble concepts. Artif. Intell. 174(5–6), 410–441 (2010). doi:10.1016/j.artint.2010.01.001
Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural Comput. 4(1), 1–58 (1992)
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, IH.: The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009). doi:10.1145/1656274.1656278
Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 832–844 (1998). doi:10.1109/34.709601
Hochberg, Y.: A sharper bonferroni procedure for multiple tests of significance. Biometrika 75(4), 800–802 (1988). doi:10.1093/biomet/75.4.800
Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, New York (2004)
Loh, W.Y.: Classification and regression trees. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 1(1), 14–23 (2011). doi:10.1002/widm.8
Maudes, J., Rodríguez, J.J., García-Osorio, C., García-Pedrajas, N.: Random feature weights for decision tree ensemble construction. Inf. Fusion 13(1), 20–30 (2012). doi:10.1016/j.inffus.2010.11.004
Mendes-Moreira, Ja, Soares, C., Jorge, A.M., Sousa, J.F.D.: Ensemble approaches for regression: a survey. ACM Comput. Surv. 45(1), 10:1–10:40 (2012). doi:10.1145/2379776.2379786
Rooney, N., Patterson, D.W., Anand, S.S., Tsymbal, A.: Random subspacing for regression ensembles. In: Proceedings of the 17th International Florida Artificial Intelligence Research Society Conference, Miami Beach, pp. 532–537 (2004)
Salzberg, S.L.: C4.5: programs for machine learning by J. Ross Quinlan (Morgan Kaufmann Publishers, Inc., 1993). Mach. Learn. 16(3), 235–240 (1994). doi:10.1007/BF00993309
Seiffert, C., Khoshgoftaar, T.M., Van Hulse, J., Napolitano, A.: Resampling or reweighting: a comparison of boosting implementations. In: 20th IEEE International Conference on Tools with Artificial Intelligence, 2008, ICTAI’08, vol. 1, pp. 445–451. IEEE, New York (2008)
Solomatine, D., Shrestha, D.: AdaBoost.RT: a boosting algorithm for regression problems. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, vol. 2, pp. 1163–1168 (2004). doi:10.1109/IJCNN.2004.1380102
Acknowledgments
This work was funded by the Ministry of Economy and Competitiveness, project TIN 2011-24046.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Arnaiz-González, Á., Díez-Pastor, J.F., García-Osorio, C. et al. Random feature weights for regression trees. Prog Artif Intell 5, 91–103 (2016). https://doi.org/10.1007/s13748-016-0081-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13748-016-0081-5