The effect of varying levels of class distribution on bagging for different algorithms: An empirical study

Original Article


Many real world applications involve highly imbalanced class distribution. Research into learning from imbalanced class distribution is considered to be one of ten challenging problems in data mining research, and it has increasingly captured the attention of both academia and industry. In this work, we study the effects of different levels of imbalanced class distribution on bagging predictors by using under-sampling techniques. Despite the popularity of bagging in many real-world applications, some questions have not been clearly answered in the existing research, such as the effect of varying the levels of class distribution on different bagging predictors, e.g., whether bagging is superior to single learners when the levels of class distribution change. Most classification learning algorithms are designed to maximize the overall accuracy rate and assume that training instances are uniformly distributed; however, the overall accuracy does not represent correct prediction on the minority class, which is the class of interest to users. The overall accuracy metric is therefore ineffective for evaluating the performance of classifiers in extremely imbalanced data. This study investigates the effect of varying levels of class distribution on different bagging predictors based on the Area Under the Receiver Operating Characteristic (ROC) Curve (AUC) as a performance metric, using an under-sampling technique on 14 data-sets with imbalanced class distributions. Our experimental results indicate that Decision Table (DTable) and RepTree are the learning algorithms with the best bagging AUC performance. The AUC performances of bagging predictors are statistically superior to single learners, with the exception of Support Vector Machines (SVM) and Decision Stump (DStump).


Imbalanced class distribution AUC performance Bagging 


  1. 1.
    Batista GE, Prati RC, Monard MC (2004) A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor Newsl 6(1):20–29CrossRefGoogle Scholar
  2. 2.
    Boehm O, Hardoon DR, Manevitz LM (2011) Classifying cognitive states of brain activity via one-class neural networks with feature selection by genetic algorithms. Int J Mach Learn Cybern 2:1–10Google Scholar
  3. 3.
    Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn 30(7):1145–1159CrossRefGoogle Scholar
  4. 4.
    Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140MATHMathSciNetGoogle Scholar
  5. 5.
    Bunkhumpornpat C, Sinapiromsaran K, Lursinsap C (2009) Safe-level-SMOTE: Safe-Level-Synthetic Minority Over-Sampling Technique for handling the class imbalanced problem. In: Advances in Knowledge Discovery and Data Mining, vol 5476, pp 475–482Google Scholar
  6. 6.
    Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16(1):321–357MATHGoogle Scholar
  7. 7.
    Chawla NV, Lazarevic A, Hall LO, Bowyer KW (2003) SMOTEBoost: improving prediction of the minority class in boosting. Paper presented at the PKDD 2003:107–119Google Scholar
  8. 8.
    Chawla NV, Japkowicz N, Kotcz A (2004) Editorial: special issue on learning from imbalanced data sets. ACM SIGKDD Explor Newsl 6(1):1–6CrossRefGoogle Scholar
  9. 9.
    Cieslak D, Chawla N (2008) Learning decision trees for unbalanced data. Paper presented at the ECML PKDD 2008, pp 241–256Google Scholar
  10. 10.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MATHMathSciNetGoogle Scholar
  11. 11.
    Estabrooks A, Jo T, Japkowicz N (2004) A multiple resampling method for learning from imbalanced data sets. Comput Intell 20(1):18–36CrossRefMathSciNetGoogle Scholar
  12. 12.
    Fawcett T (2004) ROC graphs: Notes and practical considerations for researchers. Mach Learn 31 (HPL-2003-4):1–38Google Scholar
  13. 13.
    Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874CrossRefMathSciNetGoogle Scholar
  14. 14.
    Guoxun H, Hui H, Wenyuan W (2005) An Over-sampling expert system for learning from imbalanced data sets. Paper presented at the International Conference on Neural Networks and Brain, 13–15 Oct, 537–541 Google Scholar
  15. 15.
    Han H, Wang WY, Mao BH (2005) Borderline-SMOTE: A new over-sampling method in imbalanced data sets learning. In: Advances in Intelligent Computing, vol 3644. Lecture notes in Computer Science, pp 878–887Google Scholar
  16. 16.
    He H, Garcia AE (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284CrossRefGoogle Scholar
  17. 17.
    Hido S, Kashima H, Takahashi Y (2009) Roughly balanced bagging for imbalanced data. Stat Anal Data Min 2(5–6):412–426CrossRefMathSciNetGoogle Scholar
  18. 18.
    Jun W, Shitong W, Chung F (2011) Positive and negative fuzzy rule system, extreme learning machine and image classification. Int J Mach Learn Cybern: 1–11Google Scholar
  19. 19.
    Kang P, Cho S (2006) EUS SVMs: Ensemble of under-sampled SVMs for data imbalance problems. Paper presented at the Neural Information Processing, pp 837–846Google Scholar
  20. 20.
    Koknar-Tezel S, Latecki LJ (2009) Improving SVM classification on imbalanced data sets in distance spaces. Paper presented at the 9th IEEE International Conference on Data Mining, 6–9 Dec, pp 259–267Google Scholar
  21. 21.
    Kotsiantis S, Kanellopoulos D, Pintelas P (2006) Handling imbalanced datasets: a review. Int Trans Comput Sci Eng 30(1):25–36Google Scholar
  22. 22.
    Liang G (2012) An investigation of sensitivity on bagging predictors: An empirical approach. Paper presented at the 26th AAAI Conference on Artificial Intelligence, AAAI 2012, Toronto, 22–26 July, 2439–2440Google Scholar
  23. 23.
    Liang G, Zhang C (2011) Empirical study of bagging predictors on medical data. Paper presented at the 9th Australian Data Mining Conference, AusDM 2011, Ballarat, Australia, pp 31–40Google Scholar
  24. 24.
    Liang G, Zhang C (2011) An empirical evaluation of bagging with different learning algorithms on imbalanced data. In: Proceedings of the 7th International Conference on Advanced Data Mining and Applications, ADMA 2011. 339–352Google Scholar
  25. 25.
    Liang G, Zhang C (2012) An efficient and simple under-sampling technique for imbalanced time series classification. Paper presented at the ACM 21st Conference on Information and Knowledge Management, CIKM 2012, Maui Hawaii, 29th October – 2nd NovemberGoogle Scholar
  26. 26.
    Liang G, Zhang C (2012) A comparative study of sampling methods and algorithms for imbalanced time series classification. Paper presented at the 25th Australasian Joint Conference on Artificial Intelligence, AI 2012, Sydney, 4th–7th DecemberGoogle Scholar
  27. 27.
    Liang G, Zhu X, Zhang C (2011a) An empirical study of bagging predictors for different learning algorithms. Paper presented at the 25th AAAI Conference on Artificial Intelligence, AAAI 2011, San Francisco, 7–11 August, 1802–1803Google Scholar
  28. 28.
    Liang G, Zhu X, Zhang C (2011b) An empirical study of bagging predictors for imbalanced data with different levels of class distribution. Paper presented at the 24th Australasian Joint Conference on Artificial Intelligence, AI 2011, Perth, 5th–8th December, 213–222Google Scholar
  29. 29.
    Ling C, Huang J, Zhang H (2003) AUC: A better measure than accuracy in comparing learning algorithms. Paper presented at the AI 2003, pp 329–341Google Scholar
  30. 30.
    Liu XY, Wu J, Zhou ZH (2009) Exploratory undersampling for class imbalance learning. IEEE Trans Syst Man Cybern B Cybern 39(2):539–550CrossRefGoogle Scholar
  31. 31.
    Liu W, Chawla S, Cieslak DA, Chawla NV (2010) A robust decision tree algorithm for imbalanced data sets. Paper presented at the SIAM International Conference on Data Mining, SDM 2010, Columbus, Ohio, USA, April 29–May 1, pp 766–777Google Scholar
  32. 32.
    Maloof M (2003) Learning when data sets are imbalanced and when costs are unequal and unknown. In: Proceedings of the ICML-2003 Workshop on Learning from Imbalanced Data Sets II, WashingtonGoogle Scholar
  33. 33.
    Mazurowski MA, Habas PA, Zurada JM, Lo JY, Baker JA, Tourassi GD (2008) Training neural network classifiers for medical decision making: the effects of imbalanced datasets on classification performance. Neural Netw 21(2–3):427–436CrossRefGoogle Scholar
  34. 34.
    Melville P, Mooney RJ (2005) Creating diversity in ensembles using artificial data. Inform Fusion 6(1):99–111CrossRefGoogle Scholar
  35. 35.
    Mena L, Gonzalez J (2006) Machine learning for imbalanced datasets: Application in medical diagnostic. Paper presented at the Proceedings of the 19th International FLAIRS Conference, pp 574–579Google Scholar
  36. 36.
    Merz C, Murphy P (2006) UCI Repository of Machine Learning Databases.
  37. 37.
    Phua C, Alahakoon D, Lee V (2004) Minority report in fraud detection: classification of skewed data. ACM SIGKDD Explor Newsl 6(1):50–59CrossRefGoogle Scholar
  38. 38.
    Provost F, Fawcett T, Kohavi R (1998) The case against accuracy estimation for comparing induction algorithms. Paper presented at the 15th International Conference on Machine Learning, pp 445–453Google Scholar
  39. 39.
    Rao RB, Krishnan S, Niculescu RS (2006) Data mining for improved cardiac care. ACM SIGKDD Explor Newsl 8(1):3–10CrossRefGoogle Scholar
  40. 40.
    Su CT, Hsiao YH (2007) An evaluation of the robustness of MTS for imbalanced data. IEEE Trans Knowl Data Eng:1321–1332Google Scholar
  41. 41.
    Sun Y, Kamel M, Wong A, Wang Y (2007) Cost-sensitive boosting for classification of imbalanced data. Pattern Recogn 40(12):3358–3378CrossRefMATHGoogle Scholar
  42. 42.
    Wang X, Wang Y, Xu X, Ling W, Yeung DS (2001) A new approach to fuzzy rule generation: fuzzy extension matrix. Fuzzy Sets Syst 123(3):291–306CrossRefMATHMathSciNetGoogle Scholar
  43. 43.
    Wang XZ, He Q, Chen DG, Yeung D (2005) A genetic algorithm for solving the inverse problem of support vector machines. Neurocomputing 68:225–238CrossRefGoogle Scholar
  44. 44.
    Weiss GM (2004) Mining with rarity: a unifying framework. ACM SIGKDD Explor Newsl 6(1):7–19CrossRefGoogle Scholar
  45. 45.
    Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscoGoogle Scholar
  46. 46.
    Wu X, Kumar V, Ross Quinlan J, Ghosh J, Yang Q, Motoda H, McLachlan GJ, Ng A, Liu B, Yu PS (2008) Top 10 algorithms in data mining. Knowl Inf Syst 14(1):1–37CrossRefGoogle Scholar
  47. 47.
    Yang Q, Wu X (2006) 10 challenging problems in data mining research. Int J Inform Technol Decis Mak 5(4):597–604CrossRefGoogle Scholar
  48. 48.
    Zeng-Chang Q (2005) ROC analysis for predictions made by probabilistic classifiers. Paper presented at the International Conference on Machine Learning and Cybernetics 18–21 Aug. 2005, pp 3119–3124Google Scholar

Copyright information

© Springer-Verlag 2012

Authors and Affiliations

  1. 1.Centre for Quantum Computation & Intelligent SystemsFEIT, University of TechnologySydneyAustralia
  2. 2.Centre for Cardiovascular and Chronic Care, Faculty of HealthUniversity of TechnologySydneyAustralia

Personalised recommendations