Evolving Systems

, Volume 8, Issue 1, pp 49–69 | Cite as

A methodology to carry out voting classification tasks using a particle swarm optimization-based neuro-fuzzy competitive learning network

  • Androniki Tamvakis
  • George E. Tsekouras
  • Anastasios Rigos
  • Christos Kalloniatis
  • Christos-Nikolaos Anagnostopoulos
  • George Anastassopoulos
Original Paper


The problem being investigated in this paper concerns the generation of an optimal ensemble (i.e. subset) of classifiers (picked up from set of classifiers applied on a specific classification task) that maximizes the classification performance of the voting ensemble method. The design of an algorithmic framework to meet the above goal would benefit the voting process as far as its complexity is concerned. The methodology employed here treats the classifiers as objects represented by binary vectors, and quantifies the dissimilarities between pairs of classifiers. Then, a multidimensional scaling approach is put in place to transform the classifiers into points in a low-dimensional Euclidean space. The set of the resulting points is processed by a neuro-fuzzy competitive learning network trained by a hybrid procedure that combines the merits of fuzzy clustering and particle swarm optimization. The network’s outcome is a set of homogenous groups of classifiers. To this end, the optimal ensemble is obtained by selecting from each group the best classifier. The method was successfully applied to a number of simulation experiments that involved many data sets and classifiers. Comparative analysis with other relative algorithms took place. The results were very promising as the proposed method appeared to be significantly superior in all of the experiments.


Voting classification Multidimensional scaling Particle swarm optimization Fuzzy clustering Neuro-fuzzy network Competitive learning 


  1. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66Google Scholar
  2. Aliev RA, Guirimov BG, Fazlollahi B, Aliev RR (2009) Evolutionary algorithm-based learning of fuzzy neural networks. Part 2: recurrent fuzzy neural networks. Fuzzy Sets Syst 160(17):2553–2566MathSciNetCrossRefMATHGoogle Scholar
  3. Almaksour A, Anquetil E (2011) Improving premise structure in evolving Takagi-Sugeno neuro-fuzzy classifiers. Evolving Systems 2(1):25–33CrossRefGoogle Scholar
  4. AlZoubi O, Fossati D, D’Mello S, Calvo RA (2015) Affect detection from non-stationary physiological data using ensemble classifiers. Evolv Syst 6(2):79–92CrossRefGoogle Scholar
  5. Anagnostopoulos CN, Iliou T, Giannoukos I (2015) Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artif Intell Rev 43(2):155–177CrossRefGoogle Scholar
  6. Berkhin P (2006) A survey of clustering data mining techniques. In: Kogan J, Nicholas C, Teboulle M (eds) Grouping multidimensional data. Springer, Berlin Heidelberg, pp 25–71CrossRefGoogle Scholar
  7. Breiman L (2001) Random Forests. Mach Learn 45(1):5–32CrossRefMATHGoogle Scholar
  8. Chatterjee A, Siarry A (2007) A PSO-aided neuro-fuzzy classifier employing linguistic hedge concepts. Expert Syst Appl 33(4):1097–1109CrossRefGoogle Scholar
  9. Cleary JG, Trigg LE (1995) K*: an instance-based learner using an entropic distance measure. In: Proceedings of the 12th International Conference on Machine Learning, pp 108–114Google Scholar
  10. Clerc M, Kennedy J (2002) The particle swarm—explosion. Stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73CrossRefGoogle Scholar
  11. Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20:37–46CrossRefGoogle Scholar
  12. Cox TF, Cox MAA (2001) Multidimensional Scaling. Chapman & Hall/CRCGoogle Scholar
  13. Dietterich TG (1997) Machine learning research: four current directions. AI Magazine 18(4):97–136Google Scholar
  14. Dietterich TG (2000a) Ensemble methods in machine learning. Multiple classifier systems. Springer, Berlin Heidelberg, pp 1–15CrossRefGoogle Scholar
  15. Dietterich TG (2000b) An experimental comparison of three methods for constructing ensembles of decision trees: bagging. Boosting and randomization. Mach Learn 40(2):139–157CrossRefGoogle Scholar
  16. Eberhart RC, Shi Y (2001) Tracking and optimizing dynamic systems with particle swarms. In: Proceedings of the IEEE Congress on Evolutionary Computation, Seoul, Korea pp 94–100Google Scholar
  17. Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: 13th International Conference on Machine Learning, San Francisco, pp 148–156Google Scholar
  18. Ghosh A, Shankar BU, Meher SK (2009) A novel approach to neuro-fuzzy classification. Neural Networks 22(1):100–109CrossRefGoogle Scholar
  19. Giacinto G, Roli F (2001) An approach to the automatic design of multiple classifier systems. Pattern Recogn Lett 22(1):25–33CrossRefMATHGoogle Scholar
  20. Gower JC, Ross GJS (1969) Minimum spanning trees and single linkage cluster analysis. J Roy Stat Soc 18:54–64MathSciNetGoogle Scholar
  21. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11:10–18CrossRefGoogle Scholar
  22. Huang Z (1998) Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Min Knowl Disc 2:283–304CrossRefGoogle Scholar
  23. Iglesias JA, Angelov P, Ledezma A, Sanchis A (2010) Evolving classification of agents’ behaviors: a general approach. Evolving Systems 1(3):161–171CrossRefGoogle Scholar
  24. Iliou T, Anagnostopoulos CN, Stephanakis I, Anastassopoulos G (2013) Combined classification of risk factors for appendicitis prediction in childhood. In: Iliadis L, Papadopoulos H, Jayne C (eds) Engineering Applications of Neural Networks. Springer, Berlin Heidelberg, pp 203–211CrossRefGoogle Scholar
  25. Jang J (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Transactions on Systems Man and Cybernetics 23:665–685CrossRefGoogle Scholar
  26. John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence, pp 338–345Google Scholar
  27. Juang CF, Chung IF, Hsu CH (2007) Automatic construction of feedforward/recurrent fuzzy systems by clustering-aided simplex particle swarm optimization. Fuzzy Sets Syst 158:1979–1996CrossRefMATHGoogle Scholar
  28. Kanungo T, Mount DM, Netanyahu NS, Piatko CD, Silverman R, Wu AY (2002) An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans Pattern Anal Mach Intell 24(7):881–892CrossRefMATHGoogle Scholar
  29. Kar S, Das S, Ghosh PK (2014) Applications of neuro fuzzy systems: a brief review and future outline. Appl Soft Comput 15:243–259CrossRefGoogle Scholar
  30. Karayiannis NB, Bezdek JC (1997) An integrated approach to fuzzy learning vector quantization and fuzzy c-means clustering. IEEE Trans Fuzzy Syst 5(4):622–628CrossRefGoogle Scholar
  31. Keles A, Hasiloglu AS, Keles A, Aksoy Y (2007) Neuro-fuzzy classification of prostate cancer using NEFCLASS-J. Comput Biol Med 37(11):1617–1628CrossRefGoogle Scholar
  32. Kim YS, Street WN, Menczer F (2006) Optimal ensemble construction via meta-evolutionary ensembles. Expert Syst Appl 30:705–714CrossRefGoogle Scholar
  33. Kittler J, Hatef M, Duin RWD, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239CrossRefGoogle Scholar
  34. Kotsiantis SB (2007) Supervised learning: a review of classification techniques. Informatica 31:249–268MathSciNetMATHGoogle Scholar
  35. Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207CrossRefMATHGoogle Scholar
  36. Le Cassie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Appl Stat 41:191–201CrossRefMATHGoogle Scholar
  37. Leng G, Zeng XJ, Keane JA (2012) An improved approach of self-organising fuzzy neural network based on similarity measures. Evolving Systems 3(1):19–30CrossRefGoogle Scholar
  38. Lin CJ, Hong SJ (2007) The design of neuro-fuzzy networks using particle swarm optimization and recursive singular value decomposition. Neurocomputing 71(1–3):297–310CrossRefGoogle Scholar
  39. Mateos-García D, García-Gutiérrez J, Riquelme-Santos JC (2016) An evolutionary voting for k-nearest neighbours. Expert Syst Appl 43:9–14CrossRefGoogle Scholar
  40. Mazinan AH, Sagharichiha F (2015) A novel hybrid PSO-ACO approach with its application to SPP. Evolving Systems 6(4):293–302CrossRefGoogle Scholar
  41. Mendialdua I, Arruti A, Jauregi E, Lazkano E, Sierra B (2015) Classifier subset selection to construct multi-classifiers by means of estimation of distribution algorithms. Neurocomputing 157:46–60CrossRefGoogle Scholar
  42. Morrison D, Wang R, De Silva LC (2007) Ensemble methods for spoken emotion recognition in call-centers. Speech Commun 49:98–112CrossRefGoogle Scholar
  43. Mousavia R, Eftekhari M (2015) A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput 37:652–666CrossRefGoogle Scholar
  44. Nanni L, Lumini A (2009) A genetic encoding approach for learning methods for combining classifiers. Expert Syst Appl 36:7510–7514CrossRefGoogle Scholar
  45. Nauck D, Kruse R (1999) Neuro-fuzzy systems for function approximation. Fuzzy Sets Syst 101:261–271MathSciNetCrossRefMATHGoogle Scholar
  46. Nguyen SD, Choi SB (2015) Design of a new adaptive neuro-fuzzy inference system based on a solution for clustering in a data potential field. Fuzzy Sets Syst 279:64–86MathSciNetCrossRefGoogle Scholar
  47. Niros AD, Tsekouras GE, Tsolakis D, Manousakis-Kokorakis A, Kyriazis D (2015) Hierarchical fuzzy clustering in conjunction with particle swarm optimization to efficiently design RBF neural networks. J Intell Rob Syst 78(1):105–125CrossRefGoogle Scholar
  48. Oliveira MV, Schirru R (2009) Applying particle swarm optimization algorithm for tuning a neuro-fuzzy inference system for sensor monitoring. Prog Nucl Energy 51(1):177–183CrossRefGoogle Scholar
  49. Pal NR, Bezdek JC (1995) On clustering validity for the fuzzy c-means model. IEEE Trans Fuzzy Syst 3:70–3799CrossRefGoogle Scholar
  50. Pal SK, Mitra S (1992) Multilayer perceptron, fuzzy sets, and classification. IEEE Trans Neural Networks 3:683–697CrossRefGoogle Scholar
  51. Partalas I, Tsoumakas G, Katakis I, Vlahavas I (2006) Ensemble pruning using reinforcement learning. In: Advances in artificial intelligence, Springer, pp 301–310Google Scholar
  52. Peng CR, Liu L, Niu B, Lv YL, Li MJ, Yuan YL, Zhu YB, Cai YD (2011a) Prediction of RNA-binding proteins by voting systems. J Biomed Biotechnol 2011:8. doi:10.1155/2011/506205
  53. Peng Y, Wang G, Kou G, Shi Y (2011b) An empirical study of classification algorithm evaluation for financial risk prediction. Appl Soft Comput 11(2):2906–2915CrossRefGoogle Scholar
  54. Platt JC (1998) Fast training of support vector machines using sequential minimal optimization. In: Burges CJC, Smola A, Scholkopf B (eds) Advances in kernel methods-support vector learning. MIT Press, USA, pp 185–208Google Scholar
  55. Quinlan JR (1993) C4.5: Programs for Machine Learning, Morgan KaufmanGoogle Scholar
  56. Rodd SF, Kulkarni UP, Yardi AR (2013) Adaptive neuro-fuzzy technique for performance tuning of database management systems. Evolving Systems 4(2):133–143CrossRefGoogle Scholar
  57. Ruta D, Gabrys B (2005) Classifier selection for majority voting. Information Fusion 6:63–81CrossRefMATHGoogle Scholar
  58. Sammon JW (1969) A nonlinear mapping for data structure analysis. IEEE Trans Comput 18(5):401–409CrossRefGoogle Scholar
  59. Strehl A, Ghosh J (2003) Cluster ensembles-a knowledge reuse framework for combining multiple partitions. The Journal of Machine Learning Research 3:583–617MathSciNetMATHGoogle Scholar
  60. Tamvakis A, Miritzis J, Tsirtsis G, Spyropoulou A, Spatharis S (2012) Effects of meteorological forcing on coastal eutrophication: modeling with model trees. Estuar Coast Shelf Sci 115:210–217CrossRefGoogle Scholar
  61. Tamvakis A, Anagnostopoulos CN, Tsekouras G, Anastassopoulos G (2015) Optimizing voting classification using cluster analysis on medical diagnosis data. In: EANN’15, Proceedings of the 16th International conference on engineering applications of neural networks, Rhodes, Greece, pp 12 (1–7)Google Scholar
  62. Tong M, Liu KH, Xu C, Ju W (2013) An ensemble of SVM classifiers based on gene pairs. Comput Biol Med 43:729–737CrossRefGoogle Scholar
  63. Tsao ECK, Bezdek JC, Pal NR (1994) Fuzzy Kohonen clustering networks. Pattern Recogn 27(5):757–764CrossRefGoogle Scholar
  64. Tsekouras GE (2013) A simple and effective algorithm for implementing particle swarm optimization in RBF network’s design using input-output fuzzy clustering. Neurocomputing 108:36–44CrossRefGoogle Scholar
  65. Tsekouras GE, Tsimikas J (2013) On training RBF neural networks using input–output fuzzy clustering and particle swarm optimization. Fuzzy Sets Syst 221:65–89MathSciNetCrossRefMATHGoogle Scholar
  66. Tsekouras G, Sarimveis H, Bafas G (2003) A simple algorithm for training fuzzy systems using input–output data. Adv Eng Softw 34:247–259CrossRefMATHGoogle Scholar
  67. Tsekouras GE, Rigos A, Chatzipavlis A, Velegrakis A (2015) A neural-fuzzy network based on Hermite polynomials to predict the coastal erosion. Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (EANN). Rhodes Island, Greece, pp 195–205CrossRefGoogle Scholar
  68. Tsolakis D, Tsekouras GE, Niros AD, Rigos A (2012) On the systematic development of fast fuzzy vector quantization for grayscale image compression. Neural Networks 36:83–96CrossRefMATHGoogle Scholar
  69. Ulas A, Semerci M, Yildiz OT, Alpaydin E (2009) Incremental construction of classifier and discriminant ensembles. Inf Sci 179(9):1298–1318CrossRefGoogle Scholar
  70. Weiss SM, Kapouleas I (1990) An empirical comparison of pattern recognition, neural nets and machine learning classification methods. In: Shanlik JW, Dietterich TG (Eds) Readings in Machine Learning Morgan Kaufmann pp 177–183Google Scholar
  71. Witten IH, Frank E (1998) Generating accurate rule sets without global optimization. In: Proceedings of the 15th International Conference on Machine Learning, Morgan Kaufmann, pp 144–151Google Scholar
  72. Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, Morgan KaufmannGoogle Scholar
  73. Zait M, Messatfa H (1997) A comparative study of clustering methods. Future generation computer systems 13(2–3):149–159CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Marine SciencesUniversity of the AegeanMytileneGreece
  2. 2.Department of Cultural Technology and CommunicationUniversity of the AegeanMytileneGreece
  3. 3.Medical SchoolDemocritus University of ThraceAlexandroupolisGreece

Personalised recommendations