Soft Computing

, Volume 20, Issue 10, pp 3927–3946 | Cite as

New mechanism for archive maintenance in PSO-based multi-objective feature selection

  • Hoai Bach NguyenEmail author
  • Bing Xue
  • Ivy Liu
  • Peter Andreae
  • Mengjie Zhang


In classification problems, a large number of features are typically used to describe the problem’s instances. However, not all of these features are useful for classification. Feature selection is usually an important pre-processing step to overcome the problem of “curse of dimensionality”. Feature selection aims to choose a small number of features to achieve similar or better classification performance than using all features. This paper presents a particle swarm Optimization (PSO)-based multi-objective feature selection approach to evolving a set of non-dominated feature subsets which achieve high classification performance. The proposed algorithm uses local search techniques to improve a Pareto front and is compared with a pure multi-objective PSO algorithm, three well-known evolutionary multi-objective algorithms and a current state-of-the-art PSO-based multi-objective feature selection approach. Their performances are examined on 12 benchmark datasets. The experimental results show that in most cases, the proposed multi-objective algorithm generates better Pareto fronts than all other methods.


Multi-objective Feature selection Classification  Particle Swarm Optimization 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.


  1. Asuncion A, Newman D (2007) Uci machine learning repositoryGoogle Scholar
  2. Bhowan U, McCloskey D (2015) Genetic programming for feature selection and question-answer ranking in ibm watson. In: Genetic Programming. Springer, New York, pp 153–166Google Scholar
  3. Bin W, Qinke P, Jing Z, Xiao C (2012) A binary particle swarm optimization algorithm inspired by multi-level organizational learning behavior. Eur J Oper Res 219(2):224–233MathSciNetCrossRefzbMATHGoogle Scholar
  4. Boubezoul A, Paris S (2012) Application of global optimization methods to model and feature selection. Pattern Recogn 45(10):3676–3686CrossRefzbMATHGoogle Scholar
  5. Cervante L, Xue B, Zhang M, Shang L (2012) Binary particle swarm optimisation for feature selection: a filter based approach. In: 2012 IEEE Congress on Evolutionary Computation (CEC), pp 1–8Google Scholar
  6. Chaaraoui AA, Flórez-Revuelta F (2013) Human action recognition optimization based on evolutionary feature subset selection. In: Proceedings of the 15th annual conference on Genetic and evolutionary computation (GECCO). ACM, pp 1229–1236Google Scholar
  7. Chakraborty B (2002) Genetic algorithm with fuzzy fitness function for feature selection. In: Proceedings of the 2002 IEEE International Symposium on Industrial Electronics (ISIE), vol 1, pp 315–319Google Scholar
  8. Chakraborty B, Chakraborty G (2013) Fuzzy consistency measure with particle swarm optimization for feature selection. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp 4311–4315Google Scholar
  9. Chuang LY, Chang HW, Tu CJ, Yang CH (2008) Improved binary pso for feature selection using gene expression data. Comput Biol Chem 32(1):29–38CrossRefzbMATHGoogle Scholar
  10. Dash M, Liu H (1997) Feature selection for classification. Intell Data Anal 1(3):131–156CrossRefGoogle Scholar
  11. Deb K, Agrawal S, Pratap A, Meyarivan T (2000) A fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: Nsga-ii. Lect Notes Comput Sci 1917:849–858CrossRefGoogle Scholar
  12. Gheyas IA, Smith LS (2010) Feature subset selection in large dimensionality domains. Pattern Recogn 43(1):5–13CrossRefzbMATHGoogle Scholar
  13. Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182zbMATHGoogle Scholar
  14. Hamdani TM, Won JM, Alimi AM, Karray F (2007) Multi-objective feature selection with nsga ii. In: Adaptive and Natural Computing Algorithms. Springer, New York, pp 240–247Google Scholar
  15. Hancer E, Xue B, Karaboga D, Zhang M (2015) A binary abc algorithm based on advanced similarity scheme for feature selection. Appl Soft Comput 36:334–348CrossRefGoogle Scholar
  16. Huang CL, Dun JF (2008) A distributed pso-svm hybrid system with feature selection and parameter optimization. Appl Soft Comput 8(4):1381–1391CrossRefGoogle Scholar
  17. Huang CL, Wang CJ (2006) A ga-based feature selection and parameters optimizationfor support vector machines. Expert Syst Appl 31(2):231–240CrossRefGoogle Scholar
  18. Kennedy J, Eberhart R et al (1995) Particle swarm optimization. Proc IEEE Int Conf Neural Netw 4:1942–1948CrossRefGoogle Scholar
  19. Knowles J, Corne D (1999) The pareto archived evolution strategy: A new baseline algorithm for pareto multiobjective optimisation. In: Congress on Evolutionary Computation, vol 1. IEEEGoogle Scholar
  20. Lane MC, Xue B, Liu I, Zhang M (2013) Particle swarm optimisation and statistical clustering for feature selection. In: AI 2013: Advances in Artificial Intelligence. Springer, New York, pp 214–220Google Scholar
  21. Lane MC, Xue B, Liu I, Zhang M (2014) Gaussian based particle swarm optimisation and statistical clustering for feature selection. In: Evolutionary computation in combinatorial optimisation. Springer, New York, pp 133–144Google Scholar
  22. Lee S, Soak S, Oh S, Pedrycz W, Jeon M (2008) Modified binary particle swarm optimization. Prog Nat Sci 18(9):1161–1166MathSciNetCrossRefGoogle Scholar
  23. Li X (2003) A non-dominated sorting particle swarm optimizer for multiobjective optimization. In: Proceedings of the 5th annual conference on Genetic and Evolutionary Computation (GECCO). Springer, New York, pp 37–48Google Scholar
  24. Liang D, Tsai CF, Wu HT (2015) The effect of feature selection on financial distress prediction. Knowl-Based Syst 73:289–297CrossRefGoogle Scholar
  25. Lin F, Liang D, Yeh CC, Huang JC (2014) Novel feature selection methods to financial distress prediction. Expert Syst Appl 41(5):2472–2483CrossRefGoogle Scholar
  26. Lin SW, Ying KC, Chen SC, Lee ZJ (2008) Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Syst Appl 35(4):1817–1824CrossRefGoogle Scholar
  27. Liu Y, Wang G, Chen H, Dong H, Zhu X, Wang S (2011) An improved particle swarm optimization for feature selection. J Bionic Eng 8(2):191–200CrossRefGoogle Scholar
  28. Marill T, Green DM (1963) On the effectiveness of receptors in recognition systems. IEEE Trans Inf Theory 9(1):11–17CrossRefGoogle Scholar
  29. Mohemmed AW, Zhang M, Johnston M (2009) Particle swarm optimization based adaboost for face detection. In: IEEE Congress on Evolutionary Computation (CEC), pp 2494–2501Google Scholar
  30. Neshatian K, Zhang M (2009a) Dimensionality reduction in face detection: a genetic programming approach. In: IEEE 24th International Conference on Image and Vision Computing New Zealand (IVCNZ’09), pp 391–396Google Scholar
  31. Neshatian K, Zhang M (2009b) Genetic programming for feature subset ranking in binary classification problems. In: Genetic programming. Springer, New York, pp 121–132Google Scholar
  32. Nguyen H, Xue B, Liu I, Zhang M (2014a) Filter based backward elimination in wrapper based pso for feature selection in classification. In: 2014 IEEE Congress on Evolutionary Computation (CEC), pp 3111–3118Google Scholar
  33. Nguyen HB, Xue B, Liu I, Zhang M (2014b) Pso and statistical clustering for feature selection: a new representation. In: Simulated evolution and learning. Springer, New York, pp 569–581Google Scholar
  34. Nguyen HB, Xue B, Liu I, Andreae P, Zhang M (2015) Gaussian transformation based representation in particle swarm optimisation for feature selection. In: Applications of evolutionary computation. Springer, New York, pp 541–553Google Scholar
  35. Oreski S, Oreski G (2014) Genetic algorithm-based heuristic for feature selection in credit risk assessment. Expert Syst Appl 41(4):2052–2064CrossRefGoogle Scholar
  36. Pudil P, Novovičová J, Kittler J (1994) Floating search methods in feature selection. Pattern Recogn Lett 15(11):1119–1125CrossRefGoogle Scholar
  37. Seo JH, Lee YH, Kim YH (2014) Feature selection for very short-term heavy rainfall prediction using evolutionary computation. Adv Meteorol 2014:203545. doi: 10.1155/2014/203545
  38. Stearns SD (1976) On selecting features for pattern classifiers. In: Proceedings of the 3rd International Conference on Pattern Recognition (ICPR 1976), pp 71–75Google Scholar
  39. Tran B, Xue B, Zhang M (2014) Improved pso for feature selection on high-dimensional datasets. In: Simulated evolution and learning. Springer, New York, pp 503–515Google Scholar
  40. Unler A, Murat A (2010) A discrete particle swarm optimization method for feature selection in binary classification problems. Eur J Oper Res 206(3):528–539CrossRefzbMATHGoogle Scholar
  41. Van Den Bergh F (2006) An analysis of particle swarm optimizers. Ph.D. thesis, University of PretoriaGoogle Scholar
  42. Vieira SM, Mendonça LF, Farinha GJ, Sousa JM (2013) Modified binary pso for feature selection using svm applied to mortality prediction of septic patients. Appl Soft Comput 13(8):3494–3504CrossRefGoogle Scholar
  43. Wang L (2005) A hybrid genetic algorithm-neural network strategy for simulation optimization. Appl Math Comput 170(2):1329–1343MathSciNetzbMATHGoogle Scholar
  44. Wang X, Yang J, Teng X, Xia W, Jensen R (2007) Feature selection based on rough sets and particle swarm optimization. Pattern Recogn Lett 28(4):459–471CrossRefGoogle Scholar
  45. Whitney AW (1971) A direct method of nonparametric measurement selection. IEEE Trans Comput 100(9):1100–1103MathSciNetCrossRefzbMATHGoogle Scholar
  46. Xue B, Cervante L, Shang L, Browne WN, Zhang M (2012a) A multi-objective particle swarm optimisation for filter-based feature selection in classification problems. Connect Sci 2–3:91–116CrossRefGoogle Scholar
  47. Xue B, Zhang M, Browne W (2012b) New fitness functions in binary particle swarm optimisation for feature selection. In: 2012 IEEE Congress on Evolutionary Computation (CEC), pp 2145–2152Google Scholar
  48. Xue B, Zhang M, Browne WN (2013) Particle swarm optimization for feature selection in classification: a multi-objective approach. IEEE Trans Cybern 43(6):1656–1671CrossRefGoogle Scholar
  49. Xue B, Zhang M, Browne WN (2014) Particle swarm optimisation for feature selection in classification: novel initialisation and updating mechanisms. Appl Soft Comput 18:261–276CrossRefGoogle Scholar
  50. Xue B, Zhang M, Browne W, Yao X (2015a) A survey on evolutionary computation approaches to feature selection. IEEE Trans Evol Comput (99):1–1. doi: 10.1109/TEVC.2015.2504420
  51. Xue B, Zhang M, Browne WN (2015b) A comprehensive comparison on evolutionary feature selection approaches to classification. Int J Comput Intell Appl 14(02):1550008Google Scholar
  52. Yang CS, Chuang LY, Ke CH, Yang CH (2008) Boolean binary particle swarm optimization for feature selection. In: 2008 IEEE Congress on Evolutionary Computation (CEC), pp 2093–2098Google Scholar
  53. Yuan H, Tseng SS, Gangshan W, Fuyan Z (1999) A two-phase feature selection method using both filter and wrapper. In: 1999 IEEE International Conference on Systems, Man, and Cybernetics (SMC), vol 2, pp 132–136Google Scholar
  54. Zhang Y, Gong D, Hu Y, Zhang W (2015) Feature selection algorithm based on bare bones particle swarm optimization. Neurocomputing 148:150–157CrossRefGoogle Scholar
  55. Zhao H, Sinha AP, Ge W (2009) Effects of feature construction on classification performance: an empirical study in bank failure prediction. Expert Syst Appl 36(2):2633–2644Google Scholar
  56. Zhu Z, Ong YS, Dash M (2007) Wrapper-filter feature selection algorithm using a memetic framework. IEEE Trans Syst Man Cybern 37(1):70–76Google Scholar
  57. Zitzler E, Laumanns M, Thiele L (2001) Spea 2: improving the strength pareto evolutionary algorithmGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Hoai Bach Nguyen
    • 1
    Email author
  • Bing Xue
    • 1
  • Ivy Liu
    • 2
  • Peter Andreae
    • 1
  • Mengjie Zhang
    • 1
  1. 1.School of Engineering and Computer ScienceVictoria University of WellingtonWellingtonNew Zealand
  2. 2.School of Mathematics and StatisticsVictoria University of WellingtonWellingtonNew Zealand

Personalised recommendations