Skip to main content
SpringerLink
Log in

Exploiting unlabeled data to enhance ensemble diversity

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

Ensemble learning learns from the training data by generating an ensemble of multiple base learners. It is well-known that to construct a good ensemble with strong generalization ability, the base learners are deemed to be accurate as well as diverse. In this paper, unlabeled data is exploited to facilitate ensemble learning by helping augment the diversity among the base learners. Specifically, a semi-supervised ensemble method named udeed, i.e. Unlabeled Data to Enhance Ensemble Diversity, is proposed. In contrast to existing semi-supervised ensemble methods which utilize unlabeled data by estimating error-prone pseudo-labels on them to enlarge the labeled data to improve base learners’ accuracies, udeed works by maximizing accuracies of base learners on labeled data while maximizing diversity among them on unlabeled data. Extensive experiments on 20 regular-scale and five large-scale data sets are conducted under the setting of either few or abundant labeled data. Experimental results show that udeed can effectively utilize unlabeled data for ensemble learning via diversity augmentation, and is highly competitive to well-established semi-supervised ensemble methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bennett K, Demiriz A, Maclin R (2002) Exploiting unlabeled data in ensemble methods. In: Proceedings of the 8th ACM SIGKDD international conference on knowledge discovery and data mining, Edmonton, Canada, pp 289–296

  • Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings of the 11th annual conference on computational learning theory, Madison, WI, pp 92–100

  • Breiman L (1996) Bagging predictors. Mach Learn 24(2): 123–140

    MathSciNet  MATH  Google Scholar 

  • Brown G, Wyatt JL (2003) The use of the ambiguity decomposition in neural network ensemble learning methods. In: Proceedings of the 20th international conference on machine learning, Washington, DC, pp 67–74

  • Chapelle O, Schölkopf B, Zien A (2006) Semi-supervised learning. MIT Press, Cambridge, MA

    Google Scholar 

  • Chen HH, Yao X (2009) Regularized negative correlation learning for neural network ensemble. IEEE Trans Neural Netw 20(12): 1962–1979

    Article  Google Scholar 

  • Chen K, Wang S (2008) Regularized boost for semi-supervised learning. In: Platt JC, Koller D, Singer Y, Roweis S (eds) Advances in neural information processing systems, vol 20. MIT Press, Cambridge, MA, pp 281–288

    Google Scholar 

  • Chen K, Wang S (2011) Semi-supervised learning via regularized boosting working on multiple semi-supervised assumptions. IEEE Trans Pattern Anal Mach Intell 33(1): 129–143

    Article  Google Scholar 

  • Cunningham P, Carney J (2000) Diversity versus quality in classification ensembles based on feature selection. Technical Report TCD-CS-2000-02, Department of Computer Science, Trinity College Dublin, Dublin, Ireland

  • d’Alché Buc F, Grandvalet Y, Ambroise C (2002) Semi-supervised marginboost. In: Dietterich TG, Becker S, Ghahramani Z (eds) Advances in neural information processing systems, vol 14. MIT Press, Cambridge, MA, pp 553–560

    Google Scholar 

  • Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7: 1–30

    MathSciNet  MATH  Google Scholar 

  • Dietterich TG (2000) Ensemble methods in machine learning. In: Proceedings of the 1st international workshop on multiple classifier systems, Cagliari, Italy, pp 1–15

  • Efron B, Tibshirani R (1993) An introduction to the boostrap. Chapman & Hall, New York

    Google Scholar 

  • Frank A, Asuncion A (2010) UCI machine learning repository. Technical report, School of Information and Computer Science, University of California, Irvine, CA. http://archive.ics.uci.edu/ml

  • Freund Y, Schapire RE (1995) A decision-theoretic generalization of on-line learning and an application to boosting. In: Vitányi PMB (ed) Lecture notes in computer science, vol 904. Springer, Berlin, pp 23–37

    Google Scholar 

  • Giacinto G, Roli F (2001) Design of effective neural network ensembles for image classification processes. Image Vis Comput 19(9/10): 699–707

    Article  Google Scholar 

  • Hettich S, Bay SD (1998) The UCI KDD archive. Technical report, Department of Information and Computer Science, University of California, Irvine, CA. http://kdd.ics.uci.edu

  • Krogh A, Vedelsby J (1995) Neural network ensembles, cross validation, and active learning. In: Tesauro G, Touretzky DS, Leen TK (eds) Advances in neural information processing systems, vol 7. MIT Press, Cambridge, MA, pp 231–238

    Google Scholar 

  • Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2): 181–207

    Article  MATH  Google Scholar 

  • Li M, Zhou ZH (2005) SETRED: self-training with editing. In: Proceedings of the 9th Pacific-Asia conference on knowledge discovery and data mining, Hanoi, Vietnam, pp 611–621

  • Li M, Zhou ZH (2007) Improve computer-aided diagnosis with machine learning techniques using undiagnosed samples. IEEE Trans Syst Man Cybern A Syst Hum 37(6): 1088–1098

    Article  Google Scholar 

  • Liu Y, Yao X (1999a) Ensemble learning via negative correlation. Neural Netw 12(10): 1399–1404

    Article  Google Scholar 

  • Liu Y, Yao X (1999b) Simultaneous training of negatively correlated neural networks in an ensemble. IEEE Trans Syst Man Cybern B Cybern 29(6): 716–725

    Article  Google Scholar 

  • Lu X, Jain AK (2004) Ethnicity identification from face images. In: Proceedings of SPIE international symposium on defense and security, Kissimmee, FL, pp 114–123

  • Mallapragada PK, Jin R, Jain AK, Liu Y (2009) Semiboost: boosting for semi-supervised learning. IEEE Trans Pattern Anal Mach Intell 31(11): 2000–2014

    Article  Google Scholar 

  • Mason L, Bartlett P, Baxter J, Frean M (2000) Functional gradient techniques for combining hypotheses. In: Smola A, Bartlett P, Schölkopf B, Schuurmans D (eds) Advances in large margin classifiers. MIT Press, Cambridge, pp 221–246

    Google Scholar 

  • McKay R, Abbass H (2001) Analyzing anticorrelation in ensemble learning. In: Proceedings of 2001 conference on artificial neural networks and expert systems, Otago, New Zealand, pp 22–27

  • Melville P (2005) Creating diverse ensemble classifiers to reduce supervision. PhD thesis, Department of Computer Sciences, University of Texas at Austin, Austin, TX

  • Melville P, Mooney RJ (2003) Constructing diverse classifier ensembles using artificial training examples. In: Proceedings of the 18th international joint conference on artificial intelligence, Acapulco, Mexico, pp 505–510

  • Opitz DW (1999) Feature selection for ensembles. In: Proceedings of the 16th national conference on artificial intelligence, Orlando, FL, pp 379–384

  • Opitz DW, Shavlik JW (1996) Actively searching for an effective neural network ensemble. Connection Science 8(3–4): 337–353

    Article  Google Scholar 

  • Partridge D, Krzanowski WJ (1997) Software diversity: practical statistics for its measurement and exploitation. Inf Softw Technol 39(10): 707–717

    Article  Google Scholar 

  • Rosen BE (1996) Ensemble learning using decorrelated neural networks. Connect Sci 8(3): 373–383

    Article  Google Scholar 

  • Saffari A, Grabner H, Bischof H (2008) SERboost: Semi-supervised boosting with expectation regularization. In: Proceedings of the 10th European conference on computer vision, Marseille, France, pp 588–601

  • Saffari A, Leistner C, Bischof H (2009) Regularized multi-class semi-supervised boosting. In: Proceedings of the 2009 IEEE computer society conference on computer vision and pattern recognition, Miami, FL, pp 967–974

  • Skalak D (1996) The sources of increased accuracy for two proposed boosting algorithms. In: Working notes of AAAI-96 workshop on integrating multiple learned models for improving & scaling machine learning algorithms, Portland, OR

  • Valizadegan H, Jin R, Jain AK (2008) Semi-supervised boosting for multi-class classification. In: Proceedings of the 19th European conference on machine learning, Antwerp, Belgium, pp 522–537

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1: 80–83

    Article  Google Scholar 

  • Wolpert DH (1992) Stacked generalization. Neural Netw 5(2): 241–259

    Article  MathSciNet  Google Scholar 

  • Zhang ML, Zhou ZH (2010) Exploiting unlabeled data to enhance ensemble diversity. In: Proceedings of the 10th IEEE international conference on data mining, Sydney, Australia, pp 619–628

  • Zhang ML, Zhou ZH (in press) CoTrade: confident co-training with data editing. IEEE Trans Syst Man Cybern B Cybern

  • Zhou ZH (2009a) Ensemble learning. In: Li SZ (ed) Encyclopedia of biometrics. Springer, Berlin

    Google Scholar 

  • Zhou ZH (2009b) When semi-supervised learning meets ensemble learning. In: Proceedings of the 8th international workshop on multiple classifier systems, Reykjavik, Iceland, pp 529–538

  • Zhou ZH, Jiang Y (2004) NeC4.5: neural ensemble based C4.5. IEEE Trans Knowl Data Eng 16(6): 770–773

    Article  MathSciNet  Google Scholar 

  • Zhou ZH, Li M (2005) Tri-training: exploiting unlabeled data using three classifiers. IEEE Trans Knowl Data Eng 17(11): 1529–1541

    Article  Google Scholar 

  • Zhou ZH, Li M (2010) Semi-supervised learning by disagreement. Knowl Inf Syst 24(3): 415–439

    Article  Google Scholar 

  • Zhou ZH, Zhan DC, Yang Q (2007) Semi-supervised learning with very few labeled training examples. In: Proceedings of the 22nd AAAI conference on artificial intelligence, Vancouver, Canada, pp 675–680

  • Zhu X (2006) Semi-supervised learning literature survey. Technical report 1530, Department of Computer Science, University of Wisconsin at Madison, Madison, WI

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, 210096, China

    Min-Ling Zhang

  2. National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, 210093, China

    Min-Ling Zhang & Zhi-Hua Zhou

Authors
  1. Min-Ling Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi-Hua Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhi-Hua Zhou.

Additional information

Responsible editor: Johannes Fürnkranz.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, ML., Zhou, ZH. Exploiting unlabeled data to enhance ensemble diversity. Data Min Knowl Disc 26, 98–129 (2013). https://doi.org/10.1007/s10618-011-0243-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10618-011-0243-9

Keywords

Search

Navigation