Skip to main content
SpringerLink
Log in

Stability investigation of multi-objective heuristic ensemble classifiers

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Stability analysis of heuristic ensemble classifiers, which are designed by using heuristic methods, is a significant topic due to the stochastic nature of heuristic algorithms. Considering the importance of this issue, the novelty of this paper is stability analysis of heuristic ensemble classifiers. So, in this paper, at first, two multi-objective heuristic ensemble classifiers by using a new multi-objective heuristic approach called multi-objective inclined planes optimization (MOIPO) algorithm and a conventional one called multi-objective particle swarm optimization (MOPSO) algorithm are designed and then, two-level factorial designs, as a statistical approach, are applied to investigate the stability of the best ensemble classifier from two designed ensemble classifiers for the first time; for this purpose, the effects of three structural parameters of winner algorithm i.e. inflation rate, leader selection pressure and deletion selection pressure on the performance of designed heuristic ensemble classifier for three datasets as a representative of simple data, overlapped data and data with huge number of features are investigated. Extensive experimental and comparative results on different kinds of benchmarks with nonlinear, overlapping class boundaries and different feature space dimensions not only show the supremacy of MOIPO for designing ensemble classifiers but also the important parameters and important interactions for each objective function.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

References

  1. Ab Ghani S, Muhamad NA, Zainuddin H, Noorden ZA, Mohamad N (2017) Application of response surface methodology for optimizing the oxidative stability of natural ester oil using mixed antioxidants. IEEE Trans Dielectr Electr Insul 24(2):974–983

    Article  Google Scholar 

  2. Barmuta P, Ferranti F, Gibiino GP, Lewandowski A, Schreurs DM (2015) Compact behavioral models of nonlinear active devices using response surface methodology. IEEE Trans Microw Theory Techn 63(1):56–64

    Article  Google Scholar 

  3. Bhardwaj M, Bhatnagar V (2015) Towards an optimally pruned classifier ensemble. Int J Mach Learn Cyb 6(5):699–718

    Article  Google Scholar 

  4. Chen YS (2015) Application of multi-objective fractional factorial design for ultra-wideband antennas with uniform gain and high fidelity. IET Microw Antennas Propag 9(15):1667–1672

    Article  Google Scholar 

  5. Chen Z, Xu Y, Wang C, Wen Z, Wu Y, Xu R (2016) A large-signal statistical model and yield estimation of GaN HEMTs based on response surface methodology. IEEE Microw Compon Lett 26(9):690–692

    Article  Google Scholar 

  6. Coello CAC, Pulido GT, Lechuga MS (2004) Handling multiple objectives with particle swarm optimization. IEEE Trans Evol Comput 8(3):256–279

    Article  Google Scholar 

  7. De Araujo DRB, Joaquim FM, Carmelo JABF. (2015) New graph model to design optical networks. IEEE Commun Lett 19(12):2130–2133

    Article  Google Scholar 

  8. Dos Santos EM, Sabourin R, Maupin P (2008) Pareto analysis for the selection of classifier ensembles. In: Genetic and evolutionary computation, proceedings of the 10th annual conference on. Atlanta, USA, pp 681–688

  9. Dos Santos EM, Sabourin R, Maupin P (2009) Overfitting cautious selection of classifier ensembles with genetic algorithms. Inform Fusion 10(2):150–162

    Article  Google Scholar 

  10. Fan X, Hu S, He J (2017) A dynamic selection ensemble method for target recognition based on clustering and randomized reference classifier. Int J Mach Learn Cyb 1–11

  11. Gigerenzer G, Gaissmaier W (2011) Heuristic decision making. Annu Rev Psychol 62:451–482

    Article  Google Scholar 

  12. Gupta A, Thakkar AR (2014) Optimization of stacking ensemble configuration based on various metahueristic algorithms. IEEE international advance computing conference, pp 444–451

  13. He YC, Wang XZ, He YL, Zhao SL, Li WB (2016) Exact and approximate algorithms for discounted {0–1} knapsack problem. Inform Sci 369:634–647

    Article  MathSciNet  Google Scholar 

  14. Jairo V, Luis A (2015) Factorial design for robustness evaluation of fractional PID controllers. IEEE Latin Am Trans 13(5):1286–1293

    Article  Google Scholar 

  15. Kennedy J, Eberhurt R (1995) Particle swarm optimization. IEEE 1995 neural networks conference, pp 1942–1948

  16. Kim MJ, Kang DK (2012) Classifiers selection in ensembles using genetic algorithms for bankruptcy prediction. Expert Syst Appl 39(10):9308–9314

    Article  Google Scholar 

  17. Kuhn HW, Tucker AW (1951) Nonlinear programming. In: Neyman J (ed) Proceedings of the second Berkeley symposium on mathematical statistics and probability. California, University of California Press, Berkeley, pp 481–492

    Google Scholar 

  18. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  19. Mahfouf M, Chen MY, Linkens D (2004) Adaptive weighted particle swarm optimization for multi-objective optimal design of alloy steels. In: Parallel problem solving from nature-ppsn viii. Springer, Heidelberg, pp 762–771

    Chapter  Google Scholar 

  20. Miettinen K (1999) Nonlinear multiobjective optimization. Kluwer Academic Publishers, Boston

    MATH  Google Scholar 

  21. Mousavi R, Eftekhari M (2015) A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput 37:652–666

    Article  Google Scholar 

  22. Mozaffari MH, Abdy H, Zahiri SH (2013) Application of inclined planes system optimization on data clustering. In: First Iranian Conference on Pattern Recognition and Image Analysis, Proceedings of the IEEE, pp 1–3

  23. Myers RH, Montgomery DC, Anderson-Cook CM (2016) Response surface methodology: process and product optimization using designed experiments. Wiley, USA

    MATH  Google Scholar 

  24. Nanda SJ, Panda G (2014) A survey on nature inspired metaheuristic algorithms for partitional clustering. Swarm Evol Comput 16:1–18

    Article  Google Scholar 

  25. Polikar R (2006) Ensemble based systems in decision making. IEEE Circ Syst Mag 6(3):21–45

    Article  Google Scholar 

  26. Rahman A, Verma B (2013) Ensemble classifier generation using non-uniform layered clustering and genetic algorithm. Knowl Based Syst 43:30–42

    Article  Google Scholar 

  27. Rayal R, Khanna D, Sandhu JK, Hooda N, Rana PS (2017) N-semble: neural network based ensemble approach. Int J Mach Learn Cyb 1–9

  28. Reyes-Sierra M, Coello CAC (2006) Multi-objective particle swarm optimizers: a survey of the state-of-the-art. Int J Comput Intell Research 2(3):287–308

    MathSciNet  Google Scholar 

  29. Shahraki H, Zahiri SH (2017) Fuzzy decision function estimation using fuzzified particle swarm optimization. Int J Mach Learn Cyb 8(6):1827–1838

    Article  Google Scholar 

  30. Sharkey AJ, Sharkey NE, Gerecke U, Chandroth GO (2000) The test and select approach to ensemble combination. In: Multiple classifier system, vol 1857. Springer, Berlin, pp 30–44

    Chapter  Google Scholar 

  31. Shi L, Xi L, Ma X, Weng M, Hu X (2011) A novel ensemble algorithm for biomedical classification based on ant colony optimization. Appl Soft Comput 11(8):5674–5683

    Article  Google Scholar 

  32. Shunmugapriya P, Kanmani S (2013) Optimization of stacking ensemble configurations through artificial bee colony algorithm. Swarm Evol Comput 12:24–32

    Article  Google Scholar 

  33. Srivastava B, Srivastava R, Jangid M (2014) Filter vs. wrapper approach for optimum gene selection of high dimensional gene expression dataset: an analysis with cancer datasets. In: International conference on high performance computing and applications, IEEE, pp 1–6

  34. Sushanta P, Ward T (2014) Importance of voltage reduction and optimal voltage setting during reactive power compensation. IEEE Trans Power Del 29(4):1999–2007

    Article  Google Scholar 

  35. Tan CJ, Lim CP, Cheah YN (2014) A multi-objective evolutionary algorithm-based ensemble optimizer for feature selection and classification with neural network models. Neurocomputing 125:217–228

    Article  Google Scholar 

  36. Tanha J, Van Someren M, Afsarmanesh H (2014) Boosting for multiclass semi-supervised learning. Pattern Recogn Lett 37:63–77

    Article  Google Scholar 

  37. Wang XZ, Xing HJ, Li Y, Hua Q, Dong CR, Pedrycz W (2015) A study on relationship between generalization abilities and fuzziness of base classifiers in ensemble learning. IEEE Trans Fuzzy Syst 23(5):1638–1654

    Article  Google Scholar 

  38. Xiao-Hua Z, Hong-yun M, Li-cheng J (2005) Intelligent particle swarm optimization in multiobjective optimization. IEEE Evol Comput 1:714–719

    Google Scholar 

  39. Yule G (1900) On the association of attributes in statistics. Philos T R Soc Lond 194:257–319

    Article  MATH  Google Scholar 

  40. Zhao J, Zhang Z, Han C, Sun L (2014) Experiments with feature-prior hybrid ensemble method for classification. In: Tenth international IEEE conference on computational intelligence and security, pp 223–227

  41. Zhu H, He Y, Wang XZ, Tsang ECC (2017) Discrete differential evolutions for the discounted {0–1} knapsack problem. Int J Bio Inspired Comput 10(4):219–238

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mechatronic Engineering, Faculty of Engineering, Higher Education Complex of Bam, Bam, Iran

    Zeinab Khatoun Pourtaheri

  2. Department of Electrical Engineering, Faculty of Electrical and Computer Engineering, University of Birjand, Birjand, Iran

    Seyed Hamid Zahiri & Seyed Mohammad Razavi

Authors
  1. Zeinab Khatoun Pourtaheri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seyed Hamid Zahiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Seyed Mohammad Razavi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zeinab Khatoun Pourtaheri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pourtaheri, Z.K., Zahiri, S.H. & Razavi, S.M. Stability investigation of multi-objective heuristic ensemble classifiers. Int. J. Mach. Learn. & Cyber. 10, 1109–1121 (2019). https://doi.org/10.1007/s13042-018-0789-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-018-0789-6

Keywords

Search

Navigation