Clustering-Based Ensemble Pruning and Multistage Organization Using Diversity

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11734)


The purpose of ensemble pruning is to reduce the number of predictive models in order to improve efficiency and predictive performance of the ensemble. In clustering-based approach, we are looking for groups of similar models, and then we prune each of them separately in order to increase overall diversity of the ensemble. In this paper we propose two methods for this purpose using classifier clustering on the basis of a criterion based on diversity measure. In the first method we select from each cluster the model with the best predictive performance to form the final ensemble, while the second one employs the multistage organization, where instead of removing the classifiers from the ensemble each classifier group makes the decision independently. The final answer of the proposed framework is the result of the majority voting of the decisions returned by each group. Experimentation results validated through statistical tests confirmed the usefulness of the proposed approaches.


Ensemble pruning Classifier ensemble Clustering Multistage organization 



This work was supported by the Polish National Science Centre under the grant No. 2017/27/B/ST6/01325 as well as by the statutory funds of the Department of Systems and Computer Networks, Faculty of Electronics, Wroclaw University of Science and Technology.


  1. 1.
    Bakker, B., Heskes, T.: Clustering ensembles of neural network models. Neural Netw. 16(2), 261–269 (2003)CrossRefGoogle Scholar
  2. 2.
    Fu, Q., Hu, S.X., Zhao, S.: Clustering-based selective neural network ensemble. J. Zhejiang Univ. Sci. 6(5), 387–392 (2005)CrossRefGoogle Scholar
  3. 3.
    Giacinto, G., Roli, F., Fumera, G.: Design of effective multiple classifier systems by clustering of classifiers. In: 15th International Conference on Pattern Recognition, ICPR 2000 (2000)Google Scholar
  4. 4.
    Ho, T.K.: The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell. 20, 832–844 (1998)CrossRefGoogle Scholar
  5. 5.
    Ho, T.K., Hull, J.J., Srihari, S.N.: Decision combination in multiple classifier systems. IEEE Trans. Pattern Anal. Mach. Intell. 16(1), 66–75 (1994)CrossRefGoogle Scholar
  6. 6.
    Kohavi, R., Wolpert, D.: Bias plus variance decomposition for zero-one loss functions. In: Proceedings of the Thirteenth International Conference on International Conference on Machine Learning, ICML 1996, pp. 275–283. Morgan Kaufmann Publishers Inc., San Francisco (1996)Google Scholar
  7. 7.
    Krawczyk, B., Cyganek, B.: Selecting locally specialised classifiers for one-class classification ensembles. Pattern Anal. Appl. 20(2), 427–439 (2017)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Krawczyk, B., Minku, L.L., Gama, J., Stefanowski, J., Wozniak, M.: Ensemble learning for data stream analysis: a survey. Inf. Fusion 37, 132–156 (2017)CrossRefGoogle Scholar
  9. 9.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Hoboken (2004)CrossRefGoogle Scholar
  10. 10.
    Lazarevic, A., Obradovic, Z.: The effective pruning of neural network classifiers. In: 2001 IEEE/INNS International Conference on Neural Networks, IJCNN 2001 (2001)Google Scholar
  11. 11.
    Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Ruta, D., Gabrys, B.: A theoretical analysis of the limits of majority voting errors for multiple classifier systems. Pattern Anal. Appl. 2(4), 333–350 (2002)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Topolski, M.: Algorithm of principal component analysis PCA with fuzzy observation of facility features detection of carcinoma cells multiple myeloma. In: Burduk, R., Kurzynski, M., Wozniak, M. (eds.) Progress in Computer Recognition Systems (2019)Google Scholar
  14. 14.
    Woźniak, M., Graña, M.: A survey of multiple classifier systems as hybrid systems. Inf. Fusion 16, 3–17 (2014)CrossRefGoogle Scholar
  15. 15.
    Zhang, H., Cao, L.: A spectral clustering based ensemble pruning approach. Neurocomputing 139, 289–297 (2014)CrossRefGoogle Scholar
  16. 16.
    Zhou, Z.H.: Ensemble Methods: Foundations and Algorithms. Chapman & Hall CRC, Boca Raton (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Systems and Computer Networks, Faculty of ElectronicsWrocław University of Science and TechnologyWrocławPoland

Personalised recommendations