Stochastic ensemble pruning method via simulated quenching walking

  • Zahra Sadat Taghavi
  • Seyed Taghi Akhavan Niaki
  • Amir Hossein NiknamfarEmail author
Original Article


Inspired by an upward stochastic walking idea, a new ensemble pruning method called simulated quenching walking (SQWALKING) is developed in this paper. The rationale behind this method is to give values to stochastic movements as well as to accept unvalued solutions during the investigation of search spaces. SQWALKING incorporates simulated quenching and forward selection methods to choose the models through the ensemble using probabilistic steps. Two versions of SQWALKING are introduced based on two different evaluation measures; SQWALKINGA that is based on an accuracy measure and SQWALKINGH that is based on a human-like foresight measure. The main objective is to construct a proper architecture of ensemble pruning, which is independent of ensemble construction and combination phases. Extensive comparisons between the proposed method and competitors in terms of heterogeneous and homogeneous ensembles are performed using ten datasets. The comparisons on the heterogeneous ensemble show that SQWALKINGH and SQWALKINGA can lead respectively to 5.13% and 4.22% average accuracy improvement. One reason for these promising results is the pruning phase that takes additional time to find the best models compared to rivals. Finally, the proposed SQWALKINGs are also evaluated on a real-world dataset.


Ensemble pruning method Simulated quenching algorithm Forward selection method Simulated annealing algorithm Genetic algorithm 



The authors would like to acknowledge the efforts and the consideration of the editor and all reviewers for their valuable comments and suggestions to improve the quality of the paper.


  1. 1.
    Kuncheva LI (2004) Combining pattern classifiers, methods and algorithms. Wiley, HobokenzbMATHGoogle Scholar
  2. 2.
    Tan PN, Steinbach M, Kumar V (2006) Introduction to data mining. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  3. 3.
    Dietterich TG (2000) Ensemble methods in machine learning. Springer, BerlinGoogle Scholar
  4. 4.
    Prodromidis A, Chan P (2000) Meta-learning in distributed data mining systems: issues and approaches. In: Advances of distributed data mining. AAAI Press, Palo AltoGoogle Scholar
  5. 5.
    Caruana R et al (2004) Ensemble selection from libraries of models. In: Proceedings of the 21st international conference on machine learning, ACM: Banff, Canada. p 137–144Google Scholar
  6. 6.
    Margineantu D, Dietterich T (1997) Pruning adaptive boosting. In: Proceedings of the 14th international conference on machine learning. Morgan Kaufmann Publishers Inc. p 211–218Google Scholar
  7. 7.
    Zhou ZH, Wu J, Tang W (2002) Ensemble neural networks: many could be better than all. Artif Intell 137(1–2):239–263zbMATHGoogle Scholar
  8. 8.
    Ekbal A, Saha S (2011) A multiobjective simulated annealing approach for classifier ensemble: named entity recognition in Indian languages as case studies. Expert Syst Appl 38(12):14760–14772Google Scholar
  9. 9.
    Ekbal A, Saha S (2011) Weighted vote-based classifier ensemble for named entity recognition: a genetic algorithm-based approach. ACM Trans Asian Lang Inf Process 10(2):1–37Google Scholar
  10. 10.
    Ekbal A, Saha S (2013) Simulated annealing based classifier ensemble techniques: application to part of speech tagging. Inf Fusion 14(3):288–300Google Scholar
  11. 11.
    Saha S, Ekbal A (2013) Combining multiple classifiers using vote based classifier ensemble technique for named entity recognition. Data Knowl Eng 85:15–39Google Scholar
  12. 12.
    Deb K (2001) Multi-objective optimization using evolutionary algorithms. Wiley, Hoboken, p 518zbMATHGoogle Scholar
  13. 13.
    Banfield RE et al (2005) Ensemble diversity measures and their application to thinning. Inf Fusion 6(1):49–62Google Scholar
  14. 14.
    Partalas I, Tsoumakas G, Vlahavas I (2010) An ensemble uncertainty aware measure for directed hill climbing ensemble pruning. Mach Learn 81(3):257–282MathSciNetGoogle Scholar
  15. 15.
    Taghavi ZS, Sajedi H (2013) Human-inspired ensemble pruning using hill climbing algorithm. In: The 5th Robocup Iranopen International Symposium and the 3rd Joint Conference of Robotices and AI. IEEE: Qazvin-Iran. p 1–7Google Scholar
  16. 16.
    Taghavi ZS, Sajedi H (2014) Ensemble selection using simulated annealing walking. In: The proceedings of the international conference on advances in computing, electronics and electrical technology. Institute of research engineers and doctors: Kuala Lumpur, MalaysiaGoogle Scholar
  17. 17.
    Taghavi ZS, Sajedi H (2015) Ensemble pruning based on oblivious Chained Tabu searches. Int J Hybrid Intell Syst 12(3):131–143Google Scholar
  18. 18.
    Zhou ZH, Tang W (2003) Selective ensemble of decision trees. Springer, BerlinzbMATHGoogle Scholar
  19. 19.
    Tamon C, Xiang J (2000) On the boosting pruning problem. In: The 11th European conference on machine learning, ECML. Springer, Berlin, p 404–412Google Scholar
  20. 20.
    Ingber L (1993) Simulated annealing: Practice versus theory. Math Comput Model 18(11):29–57MathSciNetzbMATHGoogle Scholar
  21. 21.
    Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing. Science 220(4598):671–680MathSciNetzbMATHGoogle Scholar
  22. 22.
    Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140zbMATHGoogle Scholar
  23. 23.
    Schapire RE (1990) The strength of weak learnability. Mach Learn 5(2):197–227Google Scholar
  24. 24.
    Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139MathSciNetzbMATHGoogle Scholar
  25. 25.
    Parmanto B, Munro P, Doyle H (1996) Improving committee diagnosis with resampling techniques. Adv Neural Inf Process Syst 8:882–888Google Scholar
  26. 26.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32zbMATHGoogle Scholar
  27. 27.
    Zhou ZH (2012) Ensemble methods: foundations and algorithms. CRC Press, Boca RatonGoogle Scholar
  28. 28.
    Xia X, Lin T, Chen Z (2018) Maximum relevancy maximum complementary based ordered aggregation for ensemble pruning. Appl Intell 48:2568–2579Google Scholar
  29. 29.
    Guo H et al (2018) Margin and diversity based ordering ensemble pruning. Neurocomputing 275:237–246Google Scholar
  30. 30.
    Li D et al (2018) RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl Inf Syst. Google Scholar
  31. 31.
    Onan A, Korukoğlu S, Bulut H (2017) A hybrid ensemble pruning approach based on consensus clustering and multi-objective evolutionary algorithm for sentiment classification. Inf Process Manag 53(4):814–833Google Scholar
  32. 32.
    Lin C et al (2014) LibD3C: ensemble classifiers with a clustering and dynamic selection strategy. Neurocomputing 123:424–435Google Scholar
  33. 33.
    Zhang H, Cao L (2014) A spectral clustering based ensemble pruning approach. Neurocomputing 139:289–297Google Scholar
  34. 34.
    Sheen S, Anitha R, Sirish P (2013) Malware detection by pruning of parallel ensembles using harmony search. Pattern Recognit Lett Innov Knowl-based Tech 34(14):1679–1686Google Scholar
  35. 35.
    Narassiguin A, Elghazel H, Aussem A (2017) Dynamic ensemble selection with probabilistic classifier chains. In: Machine learning and knowledge discovery in databases, p 169–186Google Scholar
  36. 36.
    Mozaffari A et al (2017) A hierarchical selective ensemble randomized neural network hybridized with heuristic feature selection for estimation of sea-ice thickness. Appl Intell 46(1):16–33Google Scholar
  37. 37.
    Ye R, Dai Q (2018) A novel greedy randomized dynamic ensemble selection algorithm. Neural Process Lett 47(2):565–599Google Scholar
  38. 38.
    Bandyopadhyay S et al (2008) A simulated annealing based multi-objective optimization algorithm: AMOSA. IEEE Trans Evol Comput 12(3):269–283Google Scholar
  39. 39.
    Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Addison-Wesley Longman Publishing Co., Inc., Boston, p 372Google Scholar
  40. 40.
    Deb K et al. (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2): 181–197Google Scholar
  41. 41.
    Wang X et al (2015) A study on relationship between generalization abilities and fuzziness of base classifiers in ensemble learning. IEEE Trans Fuzzy Syst 23(5):1638–1654Google Scholar
  42. 42.
    Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: Proceedings of the 13th International conference on international conference on machine learning. Morgan Kaufmann Publishers Inc., Bari, Italy. p 148–156Google Scholar
  43. 43.
    Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetzbMATHGoogle Scholar
  44. 44.
    Wei L et al (2014) Improved and promising identification of human micrornas by incorporating a high-quality negative set. IEEE/ACM Trans Comput Biol Bioinform 11(1):192–201Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Young Researchers and Elite Club, Qazvin BranchIslamic Azad UniversityQazvinIran
  2. 2.Department of Industrial EngineeringSharif University of TechnologyTehranIran

Personalised recommendations