Skip to main content

Classifier Ensemble Using a Heuristic Learning with Sparsity and Diversity

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 7664)

Abstract

Classifier ensemble has been intensively studied with the aim of overcoming the limitations of individual classifier components in two prevalent directions, i.e., to diversely generate classifier components, and to sparsely combine multiple classifiers. Currently, most approaches are emphasized only on sparsity or on diversity. In this paper, we investigated classifier ensemble with learning both sparsity and diversity using a heuristic method. We formulated the sparsity and diversity learning problem in a general mathematical framework which is beneficial for learning sparsity and diversity while grouping classifiers. Moreover, we proposed a practical approach based on the genetic algorithm for the optimization process. In order to conveniently evaluate the diversity of component classifiers, we introduced the diversity contribution ability to select proper classifier components and evolve classifier weights. Experimental results on several UCI classification data sets confirm that our approach has a promising sparseness and the generalization performance.

Keywords

  • Classifier ensemble
  • Sparsity learning
  • Diversity learning
  • Bagging

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Dietterich, T.G.: Ensemble Methods in Machine Learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)

    CrossRef  Google Scholar 

  2. Ho, T., Hull, J., Srihari, S.: Decision Combination in Multiple Classifier Systems. IEEE T-PAMI 16(1), 66–75 (1994)

    CrossRef  Google Scholar 

  3. Kittler, J., Hatef, M., Duin, R., Matas, J.: On Combining Classifiers. IEEE T-PAMI 20(3), 226–239 (1998)

    CrossRef  Google Scholar 

  4. Kuncheva, L., Whitaker, C.: Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy. Mach. Learn. 51(2), 181–207 (2003)

    CrossRef  MATH  Google Scholar 

  5. Tang, E., Suganthan, P., Yao, X.: An Analysis of Diversity Measures. Mach. Learn. 1, 247–271 (2006)

    CrossRef  Google Scholar 

  6. Breiman, L.: Bagging Predictors. Mach. Learn. 24(1), 123–140 (1996)

    MathSciNet  MATH  Google Scholar 

  7. Freund, Y., Schapire, R.: A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting. J. Comput. Syst. Sci. 55(1), 119–139 (1997)

    CrossRef  MathSciNet  MATH  Google Scholar 

  8. Schapire, R.: The Strength of Weak Learnability. Mach. Learn. 5(2), 197–227 (1990)

    Google Scholar 

  9. Kim, H.C., Pang, S., Je, H.M., Kim, D.: Constructing Support Vector Machine Ensemble. Pattern Recogn. 36(12), 2757–2767 (2003)

    CrossRef  MATH  Google Scholar 

  10. Breiman, L.: Random Forests. Mach. Learn. 45(1), 15–32 (2001)

    CrossRef  Google Scholar 

  11. Liu, C.L.: Classifier Combination Based on Confidence Transformation. Pattern Recogn. 38(1), 11–28 (2005)

    CrossRef  MATH  Google Scholar 

  12. Yin, X.C., Liu, C.P., Han, Z.: Feature Combination Using Boosting. Pattern Recogn. Lett. 26(13), 2195–2205 (2005)

    CrossRef  Google Scholar 

  13. Yu, Y., Li, Y.F., Zhou, Z.H.: Diversity Regularized Machine. In: IJCAI, pp. 1603–1608 (2011)

    Google Scholar 

  14. Li, N., Yu, Y., Zhou, Z.-H.: Diversity Regularized Ensemble Pruning. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012, Part I. LNCS, vol. 7523, pp. 330–345. Springer, Heidelberg (2012)

    CrossRef  Google Scholar 

  15. Grove, A., Schuurmans, D.: Boosting in the Limit: Maximizing the Margin of Learned Ensembles. In: AAAI, pp. 692–699 (1998)

    Google Scholar 

  16. Margineantu, D., Dietterich, T.: Pruning Adaptive Boosting. In: ICML, pp. 211–218 (1997)

    Google Scholar 

  17. Martinez-Munoz, G., Hernandez-Lobato, D., Suarez, A.: An Analysis of Ensemble Pruning Techniques Based on Ordered Aggregation. IEEE T-PAMI 31(2), 245–259 (2009)

    CrossRef  Google Scholar 

  18. Yao, X., Liu, Y.: Making Use of Population Information in Evolutionary Artificial Neural Networks. IEEE T-SMC Part B 28(3), 417–425 (1998)

    MathSciNet  Google Scholar 

  19. Zhang, L., Zhou, W.D.: Sparse Ensembles Using Weighted Combination Methods Based on Linear Programming. Pattern Recogn. 44(1), 97–106 (2011)

    CrossRef  MATH  Google Scholar 

  20. Zhou, Z.H., Wu, J.X., Tang, W.: Ensembling Neural Networks: Many Could Be Better than All? Artif. Intell. 137(1-2), 239–263 (2002)

    CrossRef  MathSciNet  MATH  Google Scholar 

  21. Chen, H., Tino, P., Yao, X.: Predictive Ensemble Pruning by Expectation Propagation. IEEE T-TDE 21(7), 999–1013 (2009)

    Google Scholar 

  22. Chen, H., Yao, X.: Multiobjective Neural Network Ensembles Based on Regularized Negative Correlation Learning. IEEE T-KDE 22(12), 1738–1751 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yin, XC., Huang, K., Hao, HW., Iqbal, K., Wang, ZB. (2012). Classifier Ensemble Using a Heuristic Learning with Sparsity and Diversity. In: Huang, T., Zeng, Z., Li, C., Leung, C.S. (eds) Neural Information Processing. ICONIP 2012. Lecture Notes in Computer Science, vol 7664. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-34481-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-34481-7_13

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-34480-0

  • Online ISBN: 978-3-642-34481-7

  • eBook Packages: Computer ScienceComputer Science (R0)