Abstract
The problem being investigated in this paper concerns the generation of an optimal ensemble (i.e. subset) of classifiers (picked up from set of classifiers applied on a specific classification task) that maximizes the classification performance of the voting ensemble method. The design of an algorithmic framework to meet the above goal would benefit the voting process as far as its complexity is concerned. The methodology employed here treats the classifiers as objects represented by binary vectors, and quantifies the dissimilarities between pairs of classifiers. Then, a multidimensional scaling approach is put in place to transform the classifiers into points in a low-dimensional Euclidean space. The set of the resulting points is processed by a neuro-fuzzy competitive learning network trained by a hybrid procedure that combines the merits of fuzzy clustering and particle swarm optimization. The network’s outcome is a set of homogenous groups of classifiers. To this end, the optimal ensemble is obtained by selecting from each group the best classifier. The method was successfully applied to a number of simulation experiments that involved many data sets and classifiers. Comparative analysis with other relative algorithms took place. The results were very promising as the proposed method appeared to be significantly superior in all of the experiments.
References
Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66
Aliev RA, Guirimov BG, Fazlollahi B, Aliev RR (2009) Evolutionary algorithm-based learning of fuzzy neural networks. Part 2: recurrent fuzzy neural networks. Fuzzy Sets Syst 160(17):2553–2566
Almaksour A, Anquetil E (2011) Improving premise structure in evolving Takagi-Sugeno neuro-fuzzy classifiers. Evolving Systems 2(1):25–33
AlZoubi O, Fossati D, D’Mello S, Calvo RA (2015) Affect detection from non-stationary physiological data using ensemble classifiers. Evolv Syst 6(2):79–92
Anagnostopoulos CN, Iliou T, Giannoukos I (2015) Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artif Intell Rev 43(2):155–177
Berkhin P (2006) A survey of clustering data mining techniques. In: Kogan J, Nicholas C, Teboulle M (eds) Grouping multidimensional data. Springer, Berlin Heidelberg, pp 25–71
Breiman L (2001) Random Forests. Mach Learn 45(1):5–32
Chatterjee A, Siarry A (2007) A PSO-aided neuro-fuzzy classifier employing linguistic hedge concepts. Expert Syst Appl 33(4):1097–1109
Cleary JG, Trigg LE (1995) K*: an instance-based learner using an entropic distance measure. In: Proceedings of the 12th International Conference on Machine Learning, pp 108–114
Clerc M, Kennedy J (2002) The particle swarm—explosion. Stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20:37–46
Cox TF, Cox MAA (2001) Multidimensional Scaling. Chapman & Hall/CRC
Dietterich TG (1997) Machine learning research: four current directions. AI Magazine 18(4):97–136
Dietterich TG (2000a) Ensemble methods in machine learning. Multiple classifier systems. Springer, Berlin Heidelberg, pp 1–15
Dietterich TG (2000b) An experimental comparison of three methods for constructing ensembles of decision trees: bagging. Boosting and randomization. Mach Learn 40(2):139–157
Eberhart RC, Shi Y (2001) Tracking and optimizing dynamic systems with particle swarms. In: Proceedings of the IEEE Congress on Evolutionary Computation, Seoul, Korea pp 94–100
Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: 13th International Conference on Machine Learning, San Francisco, pp 148–156
Ghosh A, Shankar BU, Meher SK (2009) A novel approach to neuro-fuzzy classification. Neural Networks 22(1):100–109
Giacinto G, Roli F (2001) An approach to the automatic design of multiple classifier systems. Pattern Recogn Lett 22(1):25–33
Gower JC, Ross GJS (1969) Minimum spanning trees and single linkage cluster analysis. J Roy Stat Soc 18:54–64
Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11:10–18
Huang Z (1998) Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Min Knowl Disc 2:283–304
Iglesias JA, Angelov P, Ledezma A, Sanchis A (2010) Evolving classification of agents’ behaviors: a general approach. Evolving Systems 1(3):161–171
Iliou T, Anagnostopoulos CN, Stephanakis I, Anastassopoulos G (2013) Combined classification of risk factors for appendicitis prediction in childhood. In: Iliadis L, Papadopoulos H, Jayne C (eds) Engineering Applications of Neural Networks. Springer, Berlin Heidelberg, pp 203–211
Jang J (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Transactions on Systems Man and Cybernetics 23:665–685
John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence, pp 338–345
Juang CF, Chung IF, Hsu CH (2007) Automatic construction of feedforward/recurrent fuzzy systems by clustering-aided simplex particle swarm optimization. Fuzzy Sets Syst 158:1979–1996
Kanungo T, Mount DM, Netanyahu NS, Piatko CD, Silverman R, Wu AY (2002) An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans Pattern Anal Mach Intell 24(7):881–892
Kar S, Das S, Ghosh PK (2014) Applications of neuro fuzzy systems: a brief review and future outline. Appl Soft Comput 15:243–259
Karayiannis NB, Bezdek JC (1997) An integrated approach to fuzzy learning vector quantization and fuzzy c-means clustering. IEEE Trans Fuzzy Syst 5(4):622–628
Keles A, Hasiloglu AS, Keles A, Aksoy Y (2007) Neuro-fuzzy classification of prostate cancer using NEFCLASS-J. Comput Biol Med 37(11):1617–1628
Kim YS, Street WN, Menczer F (2006) Optimal ensemble construction via meta-evolutionary ensembles. Expert Syst Appl 30:705–714
Kittler J, Hatef M, Duin RWD, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239
Kotsiantis SB (2007) Supervised learning: a review of classification techniques. Informatica 31:249–268
Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207
Le Cassie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Appl Stat 41:191–201
Leng G, Zeng XJ, Keane JA (2012) An improved approach of self-organising fuzzy neural network based on similarity measures. Evolving Systems 3(1):19–30
Lin CJ, Hong SJ (2007) The design of neuro-fuzzy networks using particle swarm optimization and recursive singular value decomposition. Neurocomputing 71(1–3):297–310
Mateos-García D, García-Gutiérrez J, Riquelme-Santos JC (2016) An evolutionary voting for k-nearest neighbours. Expert Syst Appl 43:9–14
Mazinan AH, Sagharichiha F (2015) A novel hybrid PSO-ACO approach with its application to SPP. Evolving Systems 6(4):293–302
Mendialdua I, Arruti A, Jauregi E, Lazkano E, Sierra B (2015) Classifier subset selection to construct multi-classifiers by means of estimation of distribution algorithms. Neurocomputing 157:46–60
Morrison D, Wang R, De Silva LC (2007) Ensemble methods for spoken emotion recognition in call-centers. Speech Commun 49:98–112
Mousavia R, Eftekhari M (2015) A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput 37:652–666
Nanni L, Lumini A (2009) A genetic encoding approach for learning methods for combining classifiers. Expert Syst Appl 36:7510–7514
Nauck D, Kruse R (1999) Neuro-fuzzy systems for function approximation. Fuzzy Sets Syst 101:261–271
Nguyen SD, Choi SB (2015) Design of a new adaptive neuro-fuzzy inference system based on a solution for clustering in a data potential field. Fuzzy Sets Syst 279:64–86
Niros AD, Tsekouras GE, Tsolakis D, Manousakis-Kokorakis A, Kyriazis D (2015) Hierarchical fuzzy clustering in conjunction with particle swarm optimization to efficiently design RBF neural networks. J Intell Rob Syst 78(1):105–125
Oliveira MV, Schirru R (2009) Applying particle swarm optimization algorithm for tuning a neuro-fuzzy inference system for sensor monitoring. Prog Nucl Energy 51(1):177–183
Pal NR, Bezdek JC (1995) On clustering validity for the fuzzy c-means model. IEEE Trans Fuzzy Syst 3:70–3799
Pal SK, Mitra S (1992) Multilayer perceptron, fuzzy sets, and classification. IEEE Trans Neural Networks 3:683–697
Partalas I, Tsoumakas G, Katakis I, Vlahavas I (2006) Ensemble pruning using reinforcement learning. In: Advances in artificial intelligence, Springer, pp 301–310
Peng CR, Liu L, Niu B, Lv YL, Li MJ, Yuan YL, Zhu YB, Cai YD (2011a) Prediction of RNA-binding proteins by voting systems. J Biomed Biotechnol 2011:8. doi:10.1155/2011/506205
Peng Y, Wang G, Kou G, Shi Y (2011b) An empirical study of classification algorithm evaluation for financial risk prediction. Appl Soft Comput 11(2):2906–2915
Platt JC (1998) Fast training of support vector machines using sequential minimal optimization. In: Burges CJC, Smola A, Scholkopf B (eds) Advances in kernel methods-support vector learning. MIT Press, USA, pp 185–208
Quinlan JR (1993) C4.5: Programs for Machine Learning, Morgan Kaufman
Rodd SF, Kulkarni UP, Yardi AR (2013) Adaptive neuro-fuzzy technique for performance tuning of database management systems. Evolving Systems 4(2):133–143
Ruta D, Gabrys B (2005) Classifier selection for majority voting. Information Fusion 6:63–81
Sammon JW (1969) A nonlinear mapping for data structure analysis. IEEE Trans Comput 18(5):401–409
Strehl A, Ghosh J (2003) Cluster ensembles-a knowledge reuse framework for combining multiple partitions. The Journal of Machine Learning Research 3:583–617
Tamvakis A, Miritzis J, Tsirtsis G, Spyropoulou A, Spatharis S (2012) Effects of meteorological forcing on coastal eutrophication: modeling with model trees. Estuar Coast Shelf Sci 115:210–217
Tamvakis A, Anagnostopoulos CN, Tsekouras G, Anastassopoulos G (2015) Optimizing voting classification using cluster analysis on medical diagnosis data. In: EANN’15, Proceedings of the 16th International conference on engineering applications of neural networks, Rhodes, Greece, pp 12 (1–7)
Tong M, Liu KH, Xu C, Ju W (2013) An ensemble of SVM classifiers based on gene pairs. Comput Biol Med 43:729–737
Tsao ECK, Bezdek JC, Pal NR (1994) Fuzzy Kohonen clustering networks. Pattern Recogn 27(5):757–764
Tsekouras GE (2013) A simple and effective algorithm for implementing particle swarm optimization in RBF network’s design using input-output fuzzy clustering. Neurocomputing 108:36–44
Tsekouras GE, Tsimikas J (2013) On training RBF neural networks using input–output fuzzy clustering and particle swarm optimization. Fuzzy Sets Syst 221:65–89
Tsekouras G, Sarimveis H, Bafas G (2003) A simple algorithm for training fuzzy systems using input–output data. Adv Eng Softw 34:247–259
Tsekouras GE, Rigos A, Chatzipavlis A, Velegrakis A (2015) A neural-fuzzy network based on Hermite polynomials to predict the coastal erosion. Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (EANN). Rhodes Island, Greece, pp 195–205
Tsolakis D, Tsekouras GE, Niros AD, Rigos A (2012) On the systematic development of fast fuzzy vector quantization for grayscale image compression. Neural Networks 36:83–96
Ulas A, Semerci M, Yildiz OT, Alpaydin E (2009) Incremental construction of classifier and discriminant ensembles. Inf Sci 179(9):1298–1318
Weiss SM, Kapouleas I (1990) An empirical comparison of pattern recognition, neural nets and machine learning classification methods. In: Shanlik JW, Dietterich TG (Eds) Readings in Machine Learning Morgan Kaufmann pp 177–183
Witten IH, Frank E (1998) Generating accurate rule sets without global optimization. In: Proceedings of the 15th International Conference on Machine Learning, Morgan Kaufmann, pp 144–151
Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, Morgan Kaufmann
Zait M, Messatfa H (1997) A comparative study of clustering methods. Future generation computer systems 13(2–3):149–159
Acknowledgments
This study was financially supported by the research project (code number 2522) “Synergy for the sustainable development and safe use of the Greek tourist beaches – Beachtour”, which is implemented within the framework of the Action “Cooperation 2011 - Partnerships of Production and Research Institutions in Focused Research and Technology” of the Operational Programme “Competitiveness and Entrepreneurship” (OPCE ΙΙ), (Action’s Beneficiary: General Secretariat for Research and Technology- MIA-RTDI), and is co-financed by the European Regional Development Fund (ERDF) and the Greek State.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tamvakis, A., Tsekouras, G.E., Rigos, A. et al. A methodology to carry out voting classification tasks using a particle swarm optimization-based neuro-fuzzy competitive learning network. Evolving Systems 8, 49–69 (2017). https://doi.org/10.1007/s12530-016-9160-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-016-9160-x