Bagging and Boosting with Dynamic Integration of Classifiers

  • Alexey Tsymbal
  • Seppo Puuronen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1910)

Abstract

One approach in classification tasks is to use machine learning techniques to derive classifiers using learning instances. The cooperation of several base classifiers as a decision committee has succeeded to reduce classification error. The main current decision committee learning approaches boosting and bagging use resampling with the training set and they can be used with different machine learning techniques which derive base classifiers. Boosting uses a kind of weighted voting and bagging uses equal weight voting as a combining method. Both do not take into account the local aspects that the base classifiers may have inside the problem space. We have proposed a dynamic integration technique to be used with ensembles of classifiers. In this paper, the proposed dynamic integration technique is applied with AdaBoost and bagging. The comparison results using several datasets of the UCI machine learning repository show that boosting and bagging with dynamic integration of classifiers results often better accuracy than boosting and bagging result with their original voting techniques.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bauer, E., Kohavi, R.: An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants. Machine Learning, Vol.36 (1999) 105–139.CrossRefGoogle Scholar
  2. 2.
    Blake, C.L., Merz, C.J.: UCI Repository of Machine Learning Databases [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Dept of Information and CS, Un-ty of California, Irvine CA (1998).
  3. 3.
    Breiman, L.: Bagging Predictors. Machine Learning, Vol. 24 (1996) 123–140.MATHMathSciNetGoogle Scholar
  4. 4.
    Breiman, L.: Arcing classifiers. The Annals of Statistics, Vol. 26, No. 3 (1998) 801–823.MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Cost, S., Salzberg, S.: A Weighted Nearest Neighbor Algorithm for Learning with Symbolic Features. Machine Learning, Vol. 10, No. 1 (1993) 57–78.Google Scholar
  6. 6.
    Fayyad, U., Piatetsky-Shapiro, G., Smyth, P., Uthurusamy, R.: Advances in Knowledge Discovery and Data Mining. AAAI/ MIT Press (1997).Google Scholar
  7. 7.
    Freund, Y., Schapire, R.E.: A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. In: Proc. 2nd European Conf. on Computational Learning Theory, Springer-Verlag (1995) 23–37.Google Scholar
  8. 8.
    Freund, Y., Schapire, R.E.: Discussion of the Paper “Additive Logistic Regression: a Statistical View of Boosting” by Jerome Friedman, Trevor Hastie and Robert Tibshirani. The Annals of Statistics, Vol. 28, No. 2 (2000).Google Scholar
  9. 9.
    Friedman, J., Hastie, T, Tibshirani, R.: Additive Logistic Regression: a Statistical View of Boosting. The Annals of Statistics, Vol. 28, No. 2 (2000).Google Scholar
  10. 10.
    Kohavi, R., Sommerfield, D., Dougherty, J.: Data Mining Using MLC++: A Machine Learning Library in C++. Tools with Artificial Intelligence, IEEE CS Press (1996) 234–245.Google Scholar
  11. 11.
    Koppel, M., Engelson, S.P.: Integrating Multiple Classifiers by Finding their Areas of Expertise. In: AAAI-96 Workshop On Integrating Multiple Learning Models (1996) 53–58.Google Scholar
  12. 12.
    Merz, C.: Dynamical Selection of Learning Algorithms. In: D. Fisher, H.-J. Lenz (eds.), Learning from Data, Artificial Intelligence and Statistics, Springer-Verlag, NY (1996).Google Scholar
  13. 13.
    Puuronen, S., Terziyan, V., Tsymbal, A.: A Dynamic Integration Algorithm for an Ensemble of Classifiers. In: Z.W. Ras, A. Skowron (eds.), Foundations of Intelligent Systems: ISMIS’99, Lecture Notes in AI, Vol. 1609, Springer-Verlag, Warsaw (1999) 592–600.CrossRefGoogle Scholar
  14. 14.
    Puuronen, S., Tsymbal, A., Terziyan, V.: Distance Functions in Dynamic Integration of Data Mining Techniques. In: B.V. Dasarathy (ed.), Data Mining and Knowledge Discovery: Theory, Tools, and Techniques, SPIE Press, USA (2000) to appear.Google Scholar
  15. 15.
    Quinlan, J.R.: C4.5 Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA (1993).Google Scholar
  16. 16.
    Schapire, R.E.: A Brief Introduction to Boosting. In: Proc. 16th Int. Joint Conf. AI (1999).Google Scholar
  17. 17.
    Schapire, R.E.: The Strength of Weak Learnability. Machine Learning, Vol. 5, No. 2 (1990) 197–227.Google Scholar
  18. 18.
    Schapire, E., Freaund, Y., Bartlett, P., Lee, W.S.: Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods. The Annals of Statistics, Vol. 26, No 5 (1998) 1651–1686.MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Skrypnik, I., Terziyan, V., Puuronen, S., Tsymbal, A.: Learning Feature Selection for Medical Databases. In: Proc. 12th IEEE Symp. on Computer-Based Medical Systems CBMS’99, IEEE CS Press, Stamford, CT (1999) 53–58.Google Scholar
  20. 20.
    Terziyan, V., Tsymbal, A., Puuronen, S.: The Decision Support System for Telemedicine Based on Multiple Expertise. Int. J. of Medical Informatics, Vol. 49, No. 2 (1998) 217–229.CrossRefGoogle Scholar
  21. 21.
    Tsymbal, A.: Decision Committee Learning with Dynamic Integration of Classifiers. In: Proc. 2000 ADBIS-DASFAA Symposium on Advances in Databases and Information Systems, Prague,September 2000, Lecture Notes in Computer Science, Springer-Verlag (to appear).Google Scholar
  22. 22.
    Tsymbal, A., Puuronen, S., Terziyan, V.: Advanced Dynamic Selection of Diagnostic Methods. In: Proceedings 11th IEEE Symp. on Computer-Based Medical Systems CBMS’98,IEEE CS Press, Lubbock, Texas, June (1998) 50–54.Google Scholar
  23. 23.
    Tsymbal, A., Puuronen, S., Terziyan, V.: Arbiter Meta-Learning with Dynamic Selection of Classifiers and its Experimental Investigation. In: J. Eder, I. Rozman, T. Welzer (eds.), Advances in Databases and Information Systems: 3rd East European Conference ADBIS’99, Lecture Notes in CS, Vol. 1691, Springer-Verlag, Maribor (1999) 205–217.Google Scholar
  24. 24.
    Webb, G.I.: MultiBoosting: A Technique for Combining Boosting and Wagging. Machine Learning (2000) in press.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2000

Authors and Affiliations

  • Alexey Tsymbal
    • 1
  • Seppo Puuronen
    • 1
  1. 1.Department of Computer Science and Information SystemsUniversity of JyväskyläJyväskyläFinland

Personalised recommendations