Advertisement

Dynamically building diversified classifier pruning ensembles via canonical correlation analysis

  • Zhong-Qiu Jiang
  • Xiang-Jun Shen
  • Jian-Ping Gou
  • Liangjun Wang
  • Zheng-Jun Zha
Article
  • 54 Downloads

Abstract

Empirical studies on ensemble learning that combines multiple classifiers have shown that, it is an effective technique to improve accuracy and stability of a single classifier. In this paper, we propose a novel method of dynamically building diversified sparse ensembles. We first apply a technique known as the canonical correlation to model the relationship between the input data variables and output base classifiers. The canonical (projected) output classifiers and input training data variables are encoded globally through a multi-linear projection of CCA, to decrease the impacts of noisy input data and incorrect classifiers to a minimum degree in such a global view. Secondly, based on the projection, a sparse regression method is used to prune representative classifiers by combining classifier diversity measurement. Based on the above methods, we evaluate the proposed approach by several datasets, such as UCI and handwritten digit recognition. Experimental results of the study show that, the proposed approach achieves better accuracy as compared to other ensemble methods such as QFWEC, Simple Vote Rule, Random Forest, Drep and Adaboost.

Keywords

Ensemble learning Classifier ensemble Classifier combination Classifier fusion 

Notes

Acknowledgements

This work was funded in part by the National Natural Science Foundation of China(No.61572240, 61601202,61502208), the Open Project Program of the National Laboratory of Pattern Recognition(NLPR)(No.201600005), Natural Science Foundation of Jiangsu Province (Grant No. BK20140571).

References

  1. 1.
    Bao BK, Zhu G, Shen J, Yan S (2013) Robust image analysis with sparse representation on quantized visual features. IEEE Trans Image Process 22(3):860–871MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140MathSciNetzbMATHGoogle Scholar
  3. 3.
    Britto AS, Sabourin R, Oliveira LE (2014) Dynamic selection of classifiers-a comprehensive review. Pattern Recogn 47(11):3665–3680CrossRefGoogle Scholar
  4. 4.
    Chen H, Tiňo P, Yao X (2009) Predictive ensemble pruning by expectation propagation. IEEE Trans Knowl Data Eng 21(7):999–1013CrossRefGoogle Scholar
  5. 5.
    Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–451MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Franc V, Hlavác V (2002) Multi-class support vector machine. In: 2002. Proceedings. 16th international conference on pattern recognition, vol 2, pp 236–239Google Scholar
  7. 7.
    Freund Y (1996) Experiments with a new boosting algorithm. In: Thirteenth international conference on machine learning, pp 148–156Google Scholar
  8. 8.
    Fumera G, Roli F (2005) A theoretical and experimental analysis of linear combiners for multiple classifier systems. IEEE Trans Pattern Anal Mach Intell 27 (6):942–956CrossRefGoogle Scholar
  9. 9.
    Gao X, Sun Q, Xu H (2017) Multiple-rank supervised canonical correlation analysis for feature extraction, fusion and recognition. Pergamon Press Inc.,Google Scholar
  10. 10.
    Ghorai S, Mukherjee A, Sengupta S, Dutta PK (2011) Cancer classification from gene expression data by nppc ensemble. IEEE/ACM Trans Comput Biol Bioinform 8(3):659–671CrossRefGoogle Scholar
  11. 11.
    Giacinto G, Roli F (2001) Design of effective neural network ensembles for image classification purposes. Image Vis Comput 19(9):699–707CrossRefGoogle Scholar
  12. 12.
    Hardoon DR, Szedmak SR, Shawe-Taylor JR (2004) Canonical correlation analysis: an overview with application to learning methods. Neural Comput 16 (12):2639–2664CrossRefzbMATHGoogle Scholar
  13. 13.
    Ho TK (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20(8):832–844CrossRefGoogle Scholar
  14. 14.
    Ko AH, Sabourin R, Britto JrAS (2008) From dynamic classifier selection to dynamic ensemble selection. Pattern Recogn 41(5):1718–1731CrossRefzbMATHGoogle Scholar
  15. 15.
    Krogh A, Vedelsby J et al. (1995) Neural network ensembles, cross validation, and active learning. Adv Neural Inf Proces Syst 7:231–238Google Scholar
  16. 16.
    Kuncheva LI (2004) Combining Pattern Classifiers: Methods and Algorithms. Wiley-InterscienceGoogle Scholar
  17. 17.
    Kuncheva LI (2013) A bound on kappa-error diagrams for analysis of classifier ensembles. IEEE Trans Knowl Data Eng 25(3):494–501CrossRefGoogle Scholar
  18. 18.
    Kuncheva LI, Rodriguez JJ (2007) Classifier ensembles with a random linear oracle. IEEE Trans Knowl Data Eng 19(4):500–508CrossRefGoogle Scholar
  19. 19.
    Kuncheva LI, Rodríguez JJ (2014) A weighted voting framework for classifiers ensembles. Knowl Inf Syst 38(2):259–275CrossRefGoogle Scholar
  20. 20.
    Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning 51(2):181–207CrossRefzbMATHGoogle Scholar
  21. 21.
    Li N, Yu Y, Zhou ZH (2012) Diversity regularized ensemble pruning. In: European conference on machine learning and knowledge discovery in databases, pp 330–345Google Scholar
  22. 22.
    Liu L, Shao L, Rockett P (2013) Boosted key-frame selection and correlated pyramidal motion-feature representation for human action recognition. Pattern Recogn 46(7):1810–1818CrossRefGoogle Scholar
  23. 23.
    Liu J, Shang S, Zheng K, Wen J-R (2016) Multi-view ensemble learning for dementia diagnosis from neuroimaging: an artificial neural network approach. Neurocomputing 195:112–116CrossRefGoogle Scholar
  24. 24.
    Liu W, Zha ZJ, Wang Y, Lu K, Tao D (2016) P-laplacian regularized sparse coding for human activity recognition. IEEE Trans Ind Electron 63(8):5120–5129Google Scholar
  25. 25.
    Mao S, Jiao L, Xiong L, Gou S, Chen B, Yeung S-K (2015) Weighted classifier ensemble based on quadratic form. Pattern Recogn 48(5):1688–1706CrossRefzbMATHGoogle Scholar
  26. 26.
    Martínez-Muñoz G., Hernández-Lobato D., Suárez A. (2009) An analysis of ensemble pruning techniques based on ordered aggregation. IEEE Trans Pattern Anal Mach Intell 31(2):245–259CrossRefGoogle Scholar
  27. 27.
    Needell D, Tropp JA (2009) Cosamp iterative signal recovery from incomplete and inaccurate samples. Appl Comput Harmon Anal 26(3):301–321MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Quan Y, Xu Y, Sun Y, Huang Y (2016) Supervised dictionary learning with multiple classifier integration. Pattern Recogn 55:247–260CrossRefGoogle Scholar
  29. 29.
    Saitta L (2006) Hypothesis diversity in ensemble classification. In: International symposium on methodologies for intelligent systems, pp 662–670Google Scholar
  30. 30.
    Sim J, Wright CC (2005) The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 85(3):257Google Scholar
  31. 31.
    Skalak DB et al (1996) The sources of increased accuracy for two proposed boosting algorithms. In: Proceedings American association for artificial intelligence, AAAI-96, Integrating Multiple Learned Models Workshop, pp 120–125Google Scholar
  32. 32.
    Tang S, Zhang Y-D, Xu Z-X, Li H-J, Zheng Y-T, Li J-T (2015) An efficient concept detection system via sparse ensemble learning. Neurocomputing 169:124–133CrossRefGoogle Scholar
  33. 33.
    Tao D, Tang X, Li X, Wu X (2006) Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval. IEEE Trans Pattern Anal Mach Intell 28(7):1088–1099CrossRefGoogle Scholar
  34. 34.
    Ueda N (2000) Optimal linear combination of neural networks for improving classification performance. IEEE Trans Pattern Anal Mach Intell 22(2):207–215CrossRefGoogle Scholar
  35. 35.
    Via J, Santamaria I, Perez J (2005) Canonical correlation analysis (cca) algorithms for multiple data sets: application to blind simo equalization. In: Signal processing conference, 2005 European, pp 1–4Google Scholar
  36. 36.
    Wahlberg B, Boyd S, Annergren M, Wang Y (2012) An admm algorithm for a class of total variation regularized estimation problems. IFAC Proceedings Volumes 45(16):83–88CrossRefGoogle Scholar
  37. 37.
    Wang X-Z, Xing H-J, Li Y, Hua Q, Dong C-R, Pedrycz W (2015) A study on relationship between generalization abilities and fuzziness of base classifiers in ensemble learning. IEEE Trans Fuzzy Syst 23(5):1638–1654CrossRefGoogle Scholar
  38. 38.
    Woloszynski T, Kurzynski M (2011) A probabilistic model of classifier competence for dynamic ensemble selection. Pattern Recogn 44(10):2656–2668CrossRefzbMATHGoogle Scholar
  39. 39.
    Wright J, Ma Y, Mairal J, Sapiro G, Huang TS, Yan S (2010) Sparse representation for computer vision and pattern recognition. Proc IEEE 98(6):1031–1044CrossRefGoogle Scholar
  40. 40.
    Yang X, Liu W, Tao D, Cheng J (2017) Canonical correlation analysis networks for two-view image recognition. Information Sciences An International Journal 385(C):338–352CrossRefGoogle Scholar
  41. 41.
    Yin XC, Yang C, Hao HW (2014) Learning to diversify via weighted kernels for classifier ensemble. Eprint ArxivGoogle Scholar
  42. 42.
    Yin X-C, Huang K, Hao H-W (2015) De 2: dynamic ensemble of ensembles for learning nonstationary data. Neurocomputing 165:14–22CrossRefGoogle Scholar
  43. 43.
    Zhang L, Zhou W (2010) On the sparseness of 1-norm support vector machines. Neural Netw 23(3):373–385CrossRefGoogle Scholar
  44. 44.
    Zhang L, Zhou W-D (2011) Sparse ensembles using weighted combination methods based on linear programming. Pattern Recogn 44(1):97–106MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Zhang Y, Burer S, Street WN (2006) Ensemble pruning via semi-definite programming. J Mach Learn Res 7:1315–1338MathSciNetzbMATHGoogle Scholar
  46. 46.
    Zhang E, Zhang X, Jiao L, Li L, Hou B (2016) Spectral–spatial hyperspectral image ensemble classification via joint sparse representation. Pattern Recogn 59:42–54CrossRefGoogle Scholar
  47. 47.
    Zhao Z, Jiao L, Liu F, Zhao J, Chen P (2016) Semisupervised discriminant feature learning for sar image category via sparse ensemble. IEEE Trans Geosci Remote Sens 54(6):3532–3547CrossRefGoogle Scholar
  48. 48.
    Zhou ZH, Wu JX, Jiang Y, Chen SF (2001) Genetic algorithm based selective neural network ensemble. In: International joint conference on artificial intelligence, pp 797–802Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Zhong-Qiu Jiang
    • 1
  • Xiang-Jun Shen
    • 1
  • Jian-Ping Gou
    • 1
  • Liangjun Wang
    • 1
  • Zheng-Jun Zha
    • 2
  1. 1.School of Computer Science and Telecommunication EngineeringJiangsu UniversityZhenjiangChina
  2. 2.School of Information Science and TechnologyUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations