Skip to main content
Log in

A methodology to carry out voting classification tasks using a particle swarm optimization-based neuro-fuzzy competitive learning network

Evolving Systems Aims and scope Submit manuscript

Abstract

The problem being investigated in this paper concerns the generation of an optimal ensemble (i.e. subset) of classifiers (picked up from set of classifiers applied on a specific classification task) that maximizes the classification performance of the voting ensemble method. The design of an algorithmic framework to meet the above goal would benefit the voting process as far as its complexity is concerned. The methodology employed here treats the classifiers as objects represented by binary vectors, and quantifies the dissimilarities between pairs of classifiers. Then, a multidimensional scaling approach is put in place to transform the classifiers into points in a low-dimensional Euclidean space. The set of the resulting points is processed by a neuro-fuzzy competitive learning network trained by a hybrid procedure that combines the merits of fuzzy clustering and particle swarm optimization. The network’s outcome is a set of homogenous groups of classifiers. To this end, the optimal ensemble is obtained by selecting from each group the best classifier. The method was successfully applied to a number of simulation experiments that involved many data sets and classifiers. Comparative analysis with other relative algorithms took place. The results were very promising as the proposed method appeared to be significantly superior in all of the experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Notes

  1. http://archive.ics.uci.edu/ml/.

References

  • Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6(1):37–66

    Google Scholar 

  • Aliev RA, Guirimov BG, Fazlollahi B, Aliev RR (2009) Evolutionary algorithm-based learning of fuzzy neural networks. Part 2: recurrent fuzzy neural networks. Fuzzy Sets Syst 160(17):2553–2566

    Article  MathSciNet  MATH  Google Scholar 

  • Almaksour A, Anquetil E (2011) Improving premise structure in evolving Takagi-Sugeno neuro-fuzzy classifiers. Evolving Systems 2(1):25–33

    Article  Google Scholar 

  • AlZoubi O, Fossati D, D’Mello S, Calvo RA (2015) Affect detection from non-stationary physiological data using ensemble classifiers. Evolv Syst 6(2):79–92

    Article  Google Scholar 

  • Anagnostopoulos CN, Iliou T, Giannoukos I (2015) Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artif Intell Rev 43(2):155–177

    Article  Google Scholar 

  • Berkhin P (2006) A survey of clustering data mining techniques. In: Kogan J, Nicholas C, Teboulle M (eds) Grouping multidimensional data. Springer, Berlin Heidelberg, pp 25–71

    Chapter  Google Scholar 

  • Breiman L (2001) Random Forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  • Chatterjee A, Siarry A (2007) A PSO-aided neuro-fuzzy classifier employing linguistic hedge concepts. Expert Syst Appl 33(4):1097–1109

    Article  Google Scholar 

  • Cleary JG, Trigg LE (1995) K*: an instance-based learner using an entropic distance measure. In: Proceedings of the 12th International Conference on Machine Learning, pp 108–114

  • Clerc M, Kennedy J (2002) The particle swarm—explosion. Stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73

    Article  Google Scholar 

  • Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Measur 20:37–46

    Article  Google Scholar 

  • Cox TF, Cox MAA (2001) Multidimensional Scaling. Chapman & Hall/CRC

  • Dietterich TG (1997) Machine learning research: four current directions. AI Magazine 18(4):97–136

    Google Scholar 

  • Dietterich TG (2000a) Ensemble methods in machine learning. Multiple classifier systems. Springer, Berlin Heidelberg, pp 1–15

    Chapter  Google Scholar 

  • Dietterich TG (2000b) An experimental comparison of three methods for constructing ensembles of decision trees: bagging. Boosting and randomization. Mach Learn 40(2):139–157

    Article  Google Scholar 

  • Eberhart RC, Shi Y (2001) Tracking and optimizing dynamic systems with particle swarms. In: Proceedings of the IEEE Congress on Evolutionary Computation, Seoul, Korea pp 94–100

  • Freund Y, Schapire RE (1996) Experiments with a new boosting algorithm. In: 13th International Conference on Machine Learning, San Francisco, pp 148–156

  • Ghosh A, Shankar BU, Meher SK (2009) A novel approach to neuro-fuzzy classification. Neural Networks 22(1):100–109

    Article  Google Scholar 

  • Giacinto G, Roli F (2001) An approach to the automatic design of multiple classifier systems. Pattern Recogn Lett 22(1):25–33

    Article  MATH  Google Scholar 

  • Gower JC, Ross GJS (1969) Minimum spanning trees and single linkage cluster analysis. J Roy Stat Soc 18:54–64

    MathSciNet  Google Scholar 

  • Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The WEKA data mining software: an update. SIGKDD Explor 11:10–18

    Article  Google Scholar 

  • Huang Z (1998) Extensions to the k-means algorithm for clustering large data sets with categorical values. Data Min Knowl Disc 2:283–304

    Article  Google Scholar 

  • Iglesias JA, Angelov P, Ledezma A, Sanchis A (2010) Evolving classification of agents’ behaviors: a general approach. Evolving Systems 1(3):161–171

    Article  Google Scholar 

  • Iliou T, Anagnostopoulos CN, Stephanakis I, Anastassopoulos G (2013) Combined classification of risk factors for appendicitis prediction in childhood. In: Iliadis L, Papadopoulos H, Jayne C (eds) Engineering Applications of Neural Networks. Springer, Berlin Heidelberg, pp 203–211

    Chapter  Google Scholar 

  • Jang J (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Transactions on Systems Man and Cybernetics 23:665–685

    Article  Google Scholar 

  • John GH, Langley P (1995) Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the 11th Conference on Uncertainty in Artificial Intelligence, pp 338–345

  • Juang CF, Chung IF, Hsu CH (2007) Automatic construction of feedforward/recurrent fuzzy systems by clustering-aided simplex particle swarm optimization. Fuzzy Sets Syst 158:1979–1996

    Article  MATH  Google Scholar 

  • Kanungo T, Mount DM, Netanyahu NS, Piatko CD, Silverman R, Wu AY (2002) An efficient k-means clustering algorithm: analysis and implementation. IEEE Trans Pattern Anal Mach Intell 24(7):881–892

    Article  MATH  Google Scholar 

  • Kar S, Das S, Ghosh PK (2014) Applications of neuro fuzzy systems: a brief review and future outline. Appl Soft Comput 15:243–259

    Article  Google Scholar 

  • Karayiannis NB, Bezdek JC (1997) An integrated approach to fuzzy learning vector quantization and fuzzy c-means clustering. IEEE Trans Fuzzy Syst 5(4):622–628

    Article  Google Scholar 

  • Keles A, Hasiloglu AS, Keles A, Aksoy Y (2007) Neuro-fuzzy classification of prostate cancer using NEFCLASS-J. Comput Biol Med 37(11):1617–1628

    Article  Google Scholar 

  • Kim YS, Street WN, Menczer F (2006) Optimal ensemble construction via meta-evolutionary ensembles. Expert Syst Appl 30:705–714

    Article  Google Scholar 

  • Kittler J, Hatef M, Duin RWD, Matas J (1998) On combining classifiers. IEEE Trans Pattern Anal Mach Intell 20(3):226–239

    Article  Google Scholar 

  • Kotsiantis SB (2007) Supervised learning: a review of classification techniques. Informatica 31:249–268

    MathSciNet  MATH  Google Scholar 

  • Kuncheva LI, Whitaker CJ (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51(2):181–207

    Article  MATH  Google Scholar 

  • Le Cassie S, Van Houwelingen JC (1992) Ridge estimators in logistic regression. Appl Stat 41:191–201

    Article  MATH  Google Scholar 

  • Leng G, Zeng XJ, Keane JA (2012) An improved approach of self-organising fuzzy neural network based on similarity measures. Evolving Systems 3(1):19–30

    Article  Google Scholar 

  • Lin CJ, Hong SJ (2007) The design of neuro-fuzzy networks using particle swarm optimization and recursive singular value decomposition. Neurocomputing 71(1–3):297–310

    Article  Google Scholar 

  • Mateos-García D, García-Gutiérrez J, Riquelme-Santos JC (2016) An evolutionary voting for k-nearest neighbours. Expert Syst Appl 43:9–14

    Article  Google Scholar 

  • Mazinan AH, Sagharichiha F (2015) A novel hybrid PSO-ACO approach with its application to SPP. Evolving Systems 6(4):293–302

    Article  Google Scholar 

  • Mendialdua I, Arruti A, Jauregi E, Lazkano E, Sierra B (2015) Classifier subset selection to construct multi-classifiers by means of estimation of distribution algorithms. Neurocomputing 157:46–60

    Article  Google Scholar 

  • Morrison D, Wang R, De Silva LC (2007) Ensemble methods for spoken emotion recognition in call-centers. Speech Commun 49:98–112

    Article  Google Scholar 

  • Mousavia R, Eftekhari M (2015) A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput 37:652–666

    Article  Google Scholar 

  • Nanni L, Lumini A (2009) A genetic encoding approach for learning methods for combining classifiers. Expert Syst Appl 36:7510–7514

    Article  Google Scholar 

  • Nauck D, Kruse R (1999) Neuro-fuzzy systems for function approximation. Fuzzy Sets Syst 101:261–271

    Article  MathSciNet  MATH  Google Scholar 

  • Nguyen SD, Choi SB (2015) Design of a new adaptive neuro-fuzzy inference system based on a solution for clustering in a data potential field. Fuzzy Sets Syst 279:64–86

    Article  MathSciNet  Google Scholar 

  • Niros AD, Tsekouras GE, Tsolakis D, Manousakis-Kokorakis A, Kyriazis D (2015) Hierarchical fuzzy clustering in conjunction with particle swarm optimization to efficiently design RBF neural networks. J Intell Rob Syst 78(1):105–125

    Article  Google Scholar 

  • Oliveira MV, Schirru R (2009) Applying particle swarm optimization algorithm for tuning a neuro-fuzzy inference system for sensor monitoring. Prog Nucl Energy 51(1):177–183

    Article  Google Scholar 

  • Pal NR, Bezdek JC (1995) On clustering validity for the fuzzy c-means model. IEEE Trans Fuzzy Syst 3:70–3799

    Article  Google Scholar 

  • Pal SK, Mitra S (1992) Multilayer perceptron, fuzzy sets, and classification. IEEE Trans Neural Networks 3:683–697

    Article  Google Scholar 

  • Partalas I, Tsoumakas G, Katakis I, Vlahavas I (2006) Ensemble pruning using reinforcement learning. In: Advances in artificial intelligence, Springer, pp 301–310

  • Peng CR, Liu L, Niu B, Lv YL, Li MJ, Yuan YL, Zhu YB, Cai YD (2011a) Prediction of RNA-binding proteins by voting systems. J Biomed Biotechnol 2011:8. doi:10.1155/2011/506205

  • Peng Y, Wang G, Kou G, Shi Y (2011b) An empirical study of classification algorithm evaluation for financial risk prediction. Appl Soft Comput 11(2):2906–2915

    Article  Google Scholar 

  • Platt JC (1998) Fast training of support vector machines using sequential minimal optimization. In: Burges CJC, Smola A, Scholkopf B (eds) Advances in kernel methods-support vector learning. MIT Press, USA, pp 185–208

    Google Scholar 

  • Quinlan JR (1993) C4.5: Programs for Machine Learning, Morgan Kaufman

  • Rodd SF, Kulkarni UP, Yardi AR (2013) Adaptive neuro-fuzzy technique for performance tuning of database management systems. Evolving Systems 4(2):133–143

    Article  Google Scholar 

  • Ruta D, Gabrys B (2005) Classifier selection for majority voting. Information Fusion 6:63–81

    Article  MATH  Google Scholar 

  • Sammon JW (1969) A nonlinear mapping for data structure analysis. IEEE Trans Comput 18(5):401–409

    Article  Google Scholar 

  • Strehl A, Ghosh J (2003) Cluster ensembles-a knowledge reuse framework for combining multiple partitions. The Journal of Machine Learning Research 3:583–617

    MathSciNet  MATH  Google Scholar 

  • Tamvakis A, Miritzis J, Tsirtsis G, Spyropoulou A, Spatharis S (2012) Effects of meteorological forcing on coastal eutrophication: modeling with model trees. Estuar Coast Shelf Sci 115:210–217

    Article  Google Scholar 

  • Tamvakis A, Anagnostopoulos CN, Tsekouras G, Anastassopoulos G (2015) Optimizing voting classification using cluster analysis on medical diagnosis data. In: EANN’15, Proceedings of the 16th International conference on engineering applications of neural networks, Rhodes, Greece, pp 12 (1–7)

  • Tong M, Liu KH, Xu C, Ju W (2013) An ensemble of SVM classifiers based on gene pairs. Comput Biol Med 43:729–737

    Article  Google Scholar 

  • Tsao ECK, Bezdek JC, Pal NR (1994) Fuzzy Kohonen clustering networks. Pattern Recogn 27(5):757–764

    Article  Google Scholar 

  • Tsekouras GE (2013) A simple and effective algorithm for implementing particle swarm optimization in RBF network’s design using input-output fuzzy clustering. Neurocomputing 108:36–44

    Article  Google Scholar 

  • Tsekouras GE, Tsimikas J (2013) On training RBF neural networks using input–output fuzzy clustering and particle swarm optimization. Fuzzy Sets Syst 221:65–89

    Article  MathSciNet  MATH  Google Scholar 

  • Tsekouras G, Sarimveis H, Bafas G (2003) A simple algorithm for training fuzzy systems using input–output data. Adv Eng Softw 34:247–259

    Article  MATH  Google Scholar 

  • Tsekouras GE, Rigos A, Chatzipavlis A, Velegrakis A (2015) A neural-fuzzy network based on Hermite polynomials to predict the coastal erosion. Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (EANN). Rhodes Island, Greece, pp 195–205

    Chapter  Google Scholar 

  • Tsolakis D, Tsekouras GE, Niros AD, Rigos A (2012) On the systematic development of fast fuzzy vector quantization for grayscale image compression. Neural Networks 36:83–96

    Article  MATH  Google Scholar 

  • Ulas A, Semerci M, Yildiz OT, Alpaydin E (2009) Incremental construction of classifier and discriminant ensembles. Inf Sci 179(9):1298–1318

    Article  Google Scholar 

  • Weiss SM, Kapouleas I (1990) An empirical comparison of pattern recognition, neural nets and machine learning classification methods. In: Shanlik JW, Dietterich TG (Eds) Readings in Machine Learning Morgan Kaufmann pp 177–183

  • Witten IH, Frank E (1998) Generating accurate rule sets without global optimization. In: Proceedings of the 15th International Conference on Machine Learning, Morgan Kaufmann, pp 144–151

  • Witten IH, Frank E (2005) Data mining: practical machine learning tools and techniques, Morgan Kaufmann

  • Zait M, Messatfa H (1997) A comparative study of clustering methods. Future generation computer systems 13(2–3):149–159

    Article  Google Scholar 

Download references

Acknowledgments

This study was financially supported by the research project (code number 2522) “Synergy for the sustainable development and safe use of the Greek tourist beaches – Beachtour”, which is implemented within the framework of the Action “Cooperation 2011 - Partnerships of Production and Research Institutions in Focused Research and Technology” of the Operational Programme “Competitiveness and Entrepreneurship” (OPCE ΙΙ), (Action’s Beneficiary: General Secretariat for Research and Technology- MIA-RTDI), and is co-financed by the European Regional Development Fund (ERDF) and the Greek State.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to George E. Tsekouras.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tamvakis, A., Tsekouras, G.E., Rigos, A. et al. A methodology to carry out voting classification tasks using a particle swarm optimization-based neuro-fuzzy competitive learning network. Evolving Systems 8, 49–69 (2017). https://doi.org/10.1007/s12530-016-9160-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12530-016-9160-x

Keywords

Navigation