Shared Ensemble Learning Using Multi-trees
Decision tree learning is a machine learning technique that allows accurate and comprehensible models to be generated. Accuracy can be improved by ensemble methods which combine the predictions of a set of different trees. However, a large amount of resources is necessary to generate the ensemble. In this paper, we introduce a new ensemble method that minimises the usage of resources by sharing the common parts of the components of the ensemble. For this purpose, we learn a decision multi-tree instead of a decision tree. We call this newapproac h shared ensembles. The use of a multi-tree produces an exponential number of hypotheses to be combined, which provides better results than boosting/bagging. We performed several experiments, showing that the technique allows us to obtain accurate models and improves the use of resources with respect to classical ensemble methods.
KeywordsDecision-tree learning Decision support systems Boosting Machine Learning Hypothesis Combination Randomisation
Unable to display preview. Download preview PDF.
- 3.T. G Dietterich. Ensemble methods in machine learning. In First International Workshop on Multiple Classifier Systems, pages 1–15, 2000.Google Scholar
- 5.C. Ferri, J. Hernández, and M. J. Ramírez. Induction of Decision Multi-trees using Levin Search. In Int. Conf. on Computational Science, ICCS’02, LNCS, 2002.Google Scholar
- 6.C. Ferri, J. Hernández, and M. J. Ramírez. Learning multiple and different hypotheses. Technical report, D.S.I.C., Universitat Politécnica de Valéncia, 2002.Google Scholar
- 7.C. Ferri, J. Hernández, and M. J. Ramírez. SMILES system, a multi-purpose learning system. http://www.dsic.upv.es/~flip/smiles/, 2002.
- 8.Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In the 13th Int. Conf. on Machine Learning (ICML’1996), pages 148–156, 1996.Google Scholar
- 9.Tim Kam Ho. Random decision forests. In Proc. of the 3rd International Conference on Document Analysis and Recognition, pages 278–282, 1995.Google Scholar
- 10.Tim Kam Ho. C4.5 decision forests. In Proc. of 14th Intl. Conf. on Pattern Recognition, Brisbane, Australia, pages 545–549, 1998.Google Scholar
- 12.Dragos D. Margineantu and Thomas G. Dietterich. Pruning adaptive boosting. In 14th Int. Conf. on Machine Learning, pages 211–218. Morgan Kaufmann, 1997.Google Scholar
- 13.N. J. Nilsson. Artificial Intelligence: a new synthesis. Morgan Kaufmann, 1998.Google Scholar
- 14.University of California. UCI Machine Learning Repository Content Summary. http://www.ics.uci.edu/~mlearn/MLSummary.html.
- 15.J. Pearl. Heuristics: Intelligence search strategies for computer problem solving. Addison Wesley, 1985.Google Scholar
- 16.J. R. Quinlan. Induction of Decision Trees. In Read. in Machine Learning. M. Kaufmann, 1990.Google Scholar
- 17.J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.Google Scholar
- 18.J. R. Quinlan. Bagging, Boosting, and C4.5. In Proc. of the 13th Nat. Conf. on A.I. and the 8th Innovative Applications of A.I. Conf., pages 725–730. AAAI/MIT Press, 1996.Google Scholar
- 20.Zijian Zheng and Geoffrey I. Webb. Stochastic attribute selection committees. In Australian Joint Conference on Artificial Intelligence, pages 321–332, 1998.Google Scholar
- 21.Zijian Zheng, Geoffrey I. Webb, and K. M. Ting. Integrating boosting and stochastic attribute selection committees for further improving the performance of decision tree learning. In Proc. of 10th Int. Conf. on Tools with Artificial Intelligence (ICTAI-98), IEEE Computer Society Press, pages 216–223, 1998.Google Scholar