Advertisement

Shared Ensemble Learning Using Multi-trees

  • Victor Estruch
  • Cesar Ferri
  • Jose Hernández-Orallo
  • Maria Jose Ramírez-Quintana
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2527)

Abstract

Decision tree learning is a machine learning technique that allows accurate and comprehensible models to be generated. Accuracy can be improved by ensemble methods which combine the predictions of a set of different trees. However, a large amount of resources is necessary to generate the ensemble. In this paper, we introduce a new ensemble method that minimises the usage of resources by sharing the common parts of the components of the ensemble. For this purpose, we learn a decision multi-tree instead of a decision tree. We call this newapproac h shared ensembles. The use of a multi-tree produces an exponential number of hypotheses to be combined, which provides better results than boosting/bagging. We performed several experiments, showing that the technique allows us to obtain accurate models and improves the use of resources with respect to classical ensemble methods.

Keywords

Decision-tree learning Decision support systems Boosting Machine Learning Hypothesis Combination Randomisation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, 1996.zbMATHMathSciNetGoogle Scholar
  2. 2.
    Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.zbMATHCrossRefGoogle Scholar
  3. 3.
    T. G Dietterich. Ensemble methods in machine learning. In First International Workshop on Multiple Classifier Systems, pages 1–15, 2000.Google Scholar
  4. 4.
    Thomas G. Dietterich. An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization. Machine Learning, 40(2):139–157, 2000.CrossRefGoogle Scholar
  5. 5.
    C. Ferri, J. Hernández, and M. J. Ramírez. Induction of Decision Multi-trees using Levin Search. In Int. Conf. on Computational Science, ICCS’02, LNCS, 2002.Google Scholar
  6. 6.
    C. Ferri, J. Hernández, and M. J. Ramírez. Learning multiple and different hypotheses. Technical report, D.S.I.C., Universitat Politécnica de Valéncia, 2002.Google Scholar
  7. 7.
    C. Ferri, J. Hernández, and M. J. Ramírez. SMILES system, a multi-purpose learning system. http://www.dsic.upv.es/~flip/smiles/, 2002.
  8. 8.
    Y. Freund and R. E. Schapire. Experiments with a new boosting algorithm. In the 13th Int. Conf. on Machine Learning (ICML’1996), pages 148–156, 1996.Google Scholar
  9. 9.
    Tim Kam Ho. Random decision forests. In Proc. of the 3rd International Conference on Document Analysis and Recognition, pages 278–282, 1995.Google Scholar
  10. 10.
    Tim Kam Ho. C4.5 decision forests. In Proc. of 14th Intl. Conf. on Pattern Recognition, Brisbane, Australia, pages 545–549, 1998.Google Scholar
  11. 11.
    Ludmila I. Kuncheva. A Theoretical Study on Six Classifier Fusion Strategies. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(2):281–286, 2002.CrossRefGoogle Scholar
  12. 12.
    Dragos D. Margineantu and Thomas G. Dietterich. Pruning adaptive boosting. In 14th Int. Conf. on Machine Learning, pages 211–218. Morgan Kaufmann, 1997.Google Scholar
  13. 13.
    N. J. Nilsson. Artificial Intelligence: a new synthesis. Morgan Kaufmann, 1998.Google Scholar
  14. 14.
    University of California. UCI Machine Learning Repository Content Summary. http://www.ics.uci.edu/~mlearn/MLSummary.html.
  15. 15.
    J. Pearl. Heuristics: Intelligence search strategies for computer problem solving. Addison Wesley, 1985.Google Scholar
  16. 16.
    J. R. Quinlan. Induction of Decision Trees. In Read. in Machine Learning. M. Kaufmann, 1990.Google Scholar
  17. 17.
    J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, 1993.Google Scholar
  18. 18.
    J. R. Quinlan. Bagging, Boosting, and C4.5. In Proc. of the 13th Nat. Conf. on A.I. and the 8th Innovative Applications of A.I. Conf., pages 725–730. AAAI/MIT Press, 1996.Google Scholar
  19. 19.
    David H. Wolpert. Stacked generalization. Neural Networks, 5(2):241–259, 1992.CrossRefGoogle Scholar
  20. 20.
    Zijian Zheng and Geoffrey I. Webb. Stochastic attribute selection committees. In Australian Joint Conference on Artificial Intelligence, pages 321–332, 1998.Google Scholar
  21. 21.
    Zijian Zheng, Geoffrey I. Webb, and K. M. Ting. Integrating boosting and stochastic attribute selection committees for further improving the performance of decision tree learning. In Proc. of 10th Int. Conf. on Tools with Artificial Intelligence (ICTAI-98), IEEE Computer Society Press, pages 216–223, 1998.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Victor Estruch
    • 1
  • Cesar Ferri
    • 1
  • Jose Hernández-Orallo
    • 1
  • Maria Jose Ramírez-Quintana
    • 1
  1. 1.DSICUPVValenciaSpain

Personalised recommendations