Advertisement

Feature Weighted Ensemble Classifiers – A Modified Decision Scheme

  • Thomas Martini Jørgensen
  • Christian Linneberg
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2096)

Abstract

In order to determine the output from an aggregated classifier a number of methods exists. A common approach is to apply the majority-voting scheme. If the performance of the classifiers can be ranked in some intelligent way, the voting process can be modified by assigning individual weights to each of the ensemble members. For some base classifiers, like decision trees, a given node or leaf is activated if the input lies within a well-defined region in input space. In other words, each leaf-node can be considered as defining a given feature in input space. In this paper, we present a method for adjusting the voting process of an ensemble by assigning individual weights to this set of features, implying that different nodes of the same decision tree can contribute differently to the overall voting process. By using a randomised “look-up technique” for the training examples the weights used in the decision process is determined using a perceptron-like learning rule. We present results obtained by applying such a technique to bagged ensembles of C4.5 trees and to the socalled PERT classifier, which is an ensemble of highly randomised decision trees. The proposed technique is compared to the majority-voting scheme on a number of data sets.

Keywords

Leaf Node Input Space Ensemble Member Individual Classifier Ensemble Classifier 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Kittler, J. and Roli, F. (eds.): Multiple Classifier Systems, First International Workshop, MCS 2000, Cagliari, Italy, June 2000. Proceedings, Berlin: Springer, (2000)Google Scholar
  2. 2.
    Hansen, L.K. and Salamon P.: Neural Network Ensembles IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 12, (1990) 993–1001CrossRefGoogle Scholar
  3. 3.
    Freund, Y. and Schapire, R.E.; A decision-theoretic generalization of on-line learning and an application to boosting Journal of Computer and System Sciences, vol. 55, (1997) 119–139MATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Breiman, L.: Bagging Predictors Machine Learning, vol. 24, (1996) 123–140MATHMathSciNetGoogle Scholar
  5. 5.
    Dietterich, T.G. and Kong, E.B.: Machine Learning Bias, Statistical Bias, and Statistical Variance of Decision Tree Algorithms, technical report, Department of Computer Science, Oregon State University. (1995)Google Scholar
  6. 6.
    Dietterich, T.G.: An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization Machine Learning, vol. 40, (2000) 139–157CrossRefGoogle Scholar
  7. 7.
    Dietterich, T.G.: Ensemble Methods in Machine Learning. In: Kittler, J. and Roli, F. (eds.): Multiple Classifier Systems, First International Workshop. Lecture Notes in Computer Science. Vol 1857. Springer Verlag. New York. (2000) 1–15CrossRefGoogle Scholar
  8. 8.
    Duin, R.P.W. and Tax, D.M.J.: Experiments with classifier combining rules. In: Kittler, J. and Roli, F. (eds.): Multiple Classifier Systems, First International Workshop. Lecture Notes in Computer Science. Vol 1857. Springer Verlag. New York. (2000) 16–29CrossRefGoogle Scholar
  9. 9.
    Jørgensen, T.M. and Linneberg, C.: Theoretical Analysis and Improved Decision Criteria for the n-Tuple Classifier IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, (1999) 336–347CrossRefGoogle Scholar
  10. 10.
    Jørgensen, T.M. and Linneberg, C.: Boosting the performance of weightless neural networks by using a post-processing transformation of the output scores, International Joint Conference on Neural Networks. IEEE. Washington, DC. (1999) paper no 126Google Scholar
  11. 11.
    Breiman, L: Some infinite theory for predictor ensembles, Technical report no 577, University of California. Berkerley, California. (2000)Google Scholar
  12. 12.
    Cutler, A. and Zhao, G.: Fast Classification Using Perfect Random Trees, Technical report no 5/99/99, Department of Mathematics and Statistics, Utah State University. (1999)Google Scholar
  13. 13.
    Cutler, A and Zhao, G.: Voting Perfect Random Trees, Technical report no 5/00/100, Department of Mathematics and Statistics, Utah State University. (2000)Google Scholar
  14. 14.
    Linneberg, C. and Jørgensen, T.M.; Improved Decision Scheme for Ensembles of Randomised Decision Trees submitted for publication (2001)Google Scholar
  15. 15.
    Jørgensen, T.M., Christensen, S.S. and Liisberg, C.: Crossvalidation and information measures for RAM based neural networks. In: Austin, J. (ed.): RAM-Based Neural Networks, Singapore: World Scientific, (1998) 78–88.Google Scholar
  16. 16.
    Burges, C.J.C.: A Tutorial on Support Vector Machines for pattern Recognition Data Mining and Knowledge Discovery, vol. 2, (1998) 121–167CrossRefGoogle Scholar
  17. 17.
    Vapnik, V.: The Nature of Statistical Learning Theory, New York: Springer-Verlag, (1995)MATHGoogle Scholar
  18. 18.
    Blake, C.L. and Merz, C.J.: UCI Repository of machine learning databases http://www.ics.uci.edu/~mlearn/MLRepository.html, Irvine, CA: University of California, Department of Information and Computer Science, (1998)Google Scholar
  19. 19.
    Michie, D., Spiegelhalter, D.J. and Tayler, C.C.: Machine Learning, Neural and Statistical Classification, Out of print, available at http://www.amsta.leeds.ac.uk/~charles/statlog/ Prentice-Hall, (1994)

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Thomas Martini Jørgensen
    • 1
  • Christian Linneberg
    • 1
    • 2
  1. 1.Risø National LaboratoryRoskildeDenmark
  2. 2.Intellix A/SFrederiksbergDenmark

Personalised recommendations