Advertisement

Knowledge and Information Systems

, Volume 21, Issue 2, pp 249–266 | Cite as

Parameter determination and feature selection for back-propagation network by particle swarm optimization

  • Shih-Wei LinEmail author
  • Shih-Chieh Chen
  • Wen-Jie Wu
  • Chih-Hsien Chen
Regular Paper

Abstract

The back-propagation network (BPN) is a popular tool with applications in a variety of fields. Nevertheless, different problems may require different parameter settings for a given network architecture. A dataset may contain many features, but not all features are beneficial for classification by the BPN. Therefore, a particle-swarm-optimization-based approach, denoted as PSOBPN, is proposed to obtain the suitable parameter settings for BPN and to select the beneficial subset of features which result in a better classification accuracy rate. A set of 23 problems with a range of examples and features drawn from the UCI (University of California, Irvine) machine learning repository is adopted to test the performance of the proposed algorithm. The results are compared with several well-known published algorithms. The comparative study shows that the proposed approach improves the classification accuracy rate in most test problems. Furthermore, when the feature selection is taken into consideration, the classification accuracy rates of most datasets are increased. The proposed algorithm should thus be useful to both practitioners and researchers.

Keywords

Back-propagation network Particle swarm optimization Feature selection Parameter determination 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Abe N, Kudo M (2006) Non-parametric classifier-independent feature selection. Pattern Recognit 39: 737–746zbMATHCrossRefGoogle Scholar
  2. 2.
    Ahluwalia M, Bull L (1999) Coevolving functions in genetic programming: classification using k-nearest-neighbour, In: Banzhaf W, Daida J, Eiben G et al (eds) GECCO-99. In: Proceedings of the genetic and evolutionary computation conference. Morgan Kaufmann, Florida, pp 947–952Google Scholar
  3. 3.
    Barletta M, Gisario A, Guarino S (2007) Modeling of electrostatic fluidized bed (EFB) coating process using artificial neural networks. Eng Appl Artif Intell 20: 721–733CrossRefGoogle Scholar
  4. 4.
    Berry MJA, Linoff G (eds) (2001) Data mining techniques: for marking, sales and customer support. Wiley, NYGoogle Scholar
  5. 5.
    Castillo PA, Carpio J, Merelo JJ et al (2000a) Evolving multilayer perceptrons. Neural Process Lett 12: 115–127zbMATHCrossRefGoogle Scholar
  6. 6.
    Castillo PA, Merelo JJ, Prieto A et al (2000b) G-prop: global optimization of multilayer perceptrons using GAs. Neurocomputing 35: 149–163zbMATHCrossRefGoogle Scholar
  7. 7.
    Chen W-C, Fu GL, Tai P-H et al (2009) Process parameter optimization for MIMO plastic injection molding via soft computing. Expert Syst Appl 36: 1114–1122CrossRefGoogle Scholar
  8. 8.
    Choi YS (2001) Discovering text databases with neural nets. Knowl Inf Syst 3: 356–373zbMATHCrossRefGoogle Scholar
  9. 9.
    Danaher S, Datta S, Waddle I et al (2004) Erosion modeling using bayesian regulated artificial neural networks. Wear 256: 879–888CrossRefGoogle Scholar
  10. 10.
    Dixon PW, Corne DW, Oates MJ (2002) A preliminary investigation of modified XCS as a generic data mining tool. In: Lanai PL, Stolzmann W, Wilson S (eds) Advertences in learning classifier systems. Springer, Berlin, pp 257–269Google Scholar
  11. 11.
    Ebrahimzadeh A, Ranjbar A (2008) Intelligent digital signal-type identification. Eng Appl Artif Intell 21: 569–577CrossRefGoogle Scholar
  12. 12.
    Fung G, Mangasarian OL (2003) Finite Newton method for Lagrangian support vector machine classification. Neurocomputing 55: 39–55CrossRefGoogle Scholar
  13. 13.
    Ghosh A, Jain LC (2004) Evolutionary computation in data mining. Springer, GermanyGoogle Scholar
  14. 14.
    Ghosh R, Verma B (2003) A hierarchical method for finding optimal architecture and weights using evolutionary least square based learning. Int J Neural Syst 13: 13–24CrossRefGoogle Scholar
  15. 15.
    Gupta JND, Sexton RS (1999) Comparing backpropagation with a genetic algorithm for neural network training. Int J Manag Sci 27: 679–684Google Scholar
  16. 16.
    Gusikhin O, Rychtyckyj N, Filev D (2007) Intelligent systems in the automotive industry: applications and trends. Knowl Inf Syst 12: 147–168CrossRefGoogle Scholar
  17. 17.
    Han J, Kamber M (2007) Data mining: concepts and techniques. Morgan Kaufmann, San FranciscoGoogle Scholar
  18. 18.
    He Q, Wang L (2007) An effective co-evolutionary particle swarm optimization for constrained engineering design problems. Eng Appl Artif Intell 20: 89–99CrossRefGoogle Scholar
  19. 19.
    Heo G-S, Oh I-S (2008) Simultaneous node pruning of input and hidden layers using genetic algorithms. In: Proceedings of IEEE the seventh international conference on machine learning and cybernetics. pp 3428–3433Google Scholar
  20. 20.
    Hettich S, Blake CL, Merz CJ (1998) UCI repository of machine learning databases, Department of Information and Computer Sciences, University of California, Irvine, CA. http://www.ics.uci.edu/~mlearn/MLRepository.html
  21. 21.
    Hou T-H, Su C-H, Chang H-Z (2008) Using neural networks and immune algorithms to find the optimal parameters for an IC wire bonding process. Expert Syst Appl 34: 427–436CrossRefGoogle Scholar
  22. 22.
    Huang C-Y, Chen L-H, Chen Y-L et al (2009) Evaluating the process of a genetic algorithm to improve the back-propagation network: a Monte Carlo study. Expert Syst Appl 36: 1459–1465CrossRefGoogle Scholar
  23. 23.
    Javadi AA, Farmani R, Tan TP (2005) A hybrid intelligent genetic algorithm. Adv Eng Inform 19: 225–262CrossRefGoogle Scholar
  24. 24.
    Karpenko M, Sepehri N (2002) Neural network classifiers applied to condition monitoring of a pneumatic process valve actuator. Eng Appl Artif Intell 15: 273–283CrossRefGoogle Scholar
  25. 25.
    Kennedy J, Eberhart R (1995) Particle swarm optimization. Proc IEEE Conf Neural Netw 4: 1942–1948CrossRefGoogle Scholar
  26. 26.
    Khaw JFC, Lim BS, Lim LEN (1995) Optimal design of neural networks using the Taguchi method. Neurocomputing 7: 225–245zbMATHCrossRefGoogle Scholar
  27. 27.
    Kim K-J, Han I (2000) Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index. Expert Syst Appl 19: 125–132CrossRefGoogle Scholar
  28. 28.
    Ko Y-D, Moon P, Kim CE et al (2009) Modeling and optimization of the growth rate for ZnO thin films using neural networks and genetic algorithms. Expert Syst Appl 36: 4061–4066CrossRefGoogle Scholar
  29. 29.
    Kohavi R, John GH (1997) Wrappers for feature subset selection. Artif Intell 97: 273–324zbMATHCrossRefGoogle Scholar
  30. 30.
    Kwok NM, Liu DK, Dissanayake G (2006) Evolutionary computing based mobile robot localization. Eng Appl Artif Intell 19: 857–868CrossRefGoogle Scholar
  31. 31.
    Lee J, Kang S (2007) GA based meta-modeling of BPN architecture for constrained approximate optimization. Int J Solids Struct 44: 5980–5993zbMATHCrossRefGoogle Scholar
  32. 32.
    Lezoray O, Cardot H (2001) A neural network architecture for data classification. Int J Neural Syst 11: 33–42Google Scholar
  33. 33.
    Li S, Li Y, Liu Y et al (2007) A GA-based NN approach for makespan estimation. Appl Math Comput 185: 1003–1014zbMATHCrossRefGoogle Scholar
  34. 34.
    Li W-S, Clifton C, Liu S-Y (2000) Database integration using neural networks: implementation and experiences. Knowl Inf Syst 2: 73–96zbMATHCrossRefGoogle Scholar
  35. 35.
    Lin SC, Ting CJ (1996) Drill wear monitoring using neural networks. Int J Mach Tools Manuf 36: 465–475CrossRefGoogle Scholar
  36. 36.
    Lin S-W, Tseng T-Y, Chou S-Y et al (2008) A simulated annealing based approach for simultaneous parameter optimization and feature selection of back-propagation networks. Expert Syst Appl 34: 1491–1499CrossRefGoogle Scholar
  37. 37.
    Lin S-W, Lee Z-J, Chen S-C et al (2008) Parameter determination of support vector machines and feature selection using simulated annealing approach. Appl Soft Comput 8: 1505–1512CrossRefGoogle Scholar
  38. 38.
    Lin S-W, Ying K-C, Chen S-C et al (2008) Particle swarm optimization for parameter determination and feature selection of support vector machines. Expert Syst Appl 35: 1817–1824CrossRefGoogle Scholar
  39. 39.
    Lin S-W, Chou S-Y, Chen S-C (2007) Irregular shapes classification by back-propagation neural network. Int J Adv Manuf Technol 34: 1164–1172CrossRefGoogle Scholar
  40. 40.
    Liu H, Motoda H (1998) Feature selection for knowledge discovery and data mining. Kluwer Academic, BostonzbMATHGoogle Scholar
  41. 41.
    Liu L, Liu W, Cartes DA (2008) Particle swarm optimization-based parameter identification applied to permanent magnet synchronous motors. Eng Appl Artif Intell 21: 1092–1100CrossRefGoogle Scholar
  42. 42.
    Mahesh Parappagoudar B, Pratihar DK, Datta GL (2008) Forward and reverse mappings in green sand mould system using neural networks. Appl Soft Comput 8: 239–260CrossRefGoogle Scholar
  43. 43.
    Malhotra R, Malhotra DK (2003) Evaluating consumer loans using neural networks. Int J Manag Sci 31: 83–96Google Scholar
  44. 44.
    Mangasarian OL, Musicant DR (2001) Lagrangian support vector machine. J Mach Res 1: 161–177zbMATHCrossRefMathSciNetGoogle Scholar
  45. 45.
    Pudil P, Novovičová J, Kittler J (1994) Floating search methods in feature selection. Pattern Recognit Lett 15: 1119–1125CrossRefGoogle Scholar
  46. 46.
    Peng T, Zuo W, He F (2008) SVM based adaptive learning method for text classification from positive and unlabeled documents. Knowl Inf Syst 16: 281–301CrossRefGoogle Scholar
  47. 47.
    Qiao W, Gao Z, Harley RG et al (2008) Robust neuro-identification of nonlinear plants in electric power systems with missing sensor measurements. Eng Appl Artif Intell 21: 604–618CrossRefGoogle Scholar
  48. 48.
    Sasaki T, Tokoro M (2000) Comparison between Lamarckian and Darwinian evolution on a model using neural networks and genetic algorithms. Knowl Inf Syst 2: 201–222zbMATHCrossRefGoogle Scholar
  49. 49.
    Salzberg SL (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Min Knowl Discov 1: 317–327CrossRefGoogle Scholar
  50. 50.
    Schittenkopf C, Deco G, Brauer W (1997) Two strategies to avoid overfitting in feedforward networks. Neural Netw 10: 505–516CrossRefGoogle Scholar
  51. 51.
    Senthil Arumugam M, Rao MVC, Chandramohan A (2008) A new and improved version of particle swarm optimization algorithm with global-local best parameters. Knowl Inf Syst 16: 331–357CrossRefGoogle Scholar
  52. 52.
    Sedki A, Ouazar D, El Mazoudi E (2009) Evolving neural network using real coded genetic algorithm for daily rainfall-runoff forecasting. Expert Syst Appl 36: 4523–4527CrossRefGoogle Scholar
  53. 53.
    Sexton RS, Alidaee B, Dorsey RE et al (1998) Global optimization for artificial neural networks: a tabu search application. Eur J Oper Res 106: 570–584zbMATHCrossRefGoogle Scholar
  54. 54.
    Sexton RS, Dorsey RE, Johnson JD (1999) Optimization of neural networks: a comparative analysis of the genetic algorithm and simulated annealing. Eur J Oper Res 114: 589–601zbMATHCrossRefGoogle Scholar
  55. 55.
    Sexton RS, McMurtrey S, Cleavenger DJ (2006) Knowledge discovery using a neural network simultaneous optimization algorithm on a real world classification problem. Eur J Oper Res 168: 1009–1018zbMATHCrossRefGoogle Scholar
  56. 56.
    Shi Y, Eberhart R (1998) A modified particle swarm optimizer. In: Proceeding of the IEEE congress on Evolutionary Computation. pp 69–73Google Scholar
  57. 57.
    Shi Y, Eberhart R (1998) Parameter selection in particle swarm optimization. Lecture notes in computer science, vol 1447. pp 591–600Google Scholar
  58. 58.
    Sivagaminathan RK, Ramakrishnan S (2007) A hybrid approach for feature subset selection using neural networks and ant colony optimization. Expert Syst Appl 33: 49–60CrossRefGoogle Scholar
  59. 59.
    Smith M, Bull L (2005) GAP: constructing and selection features with evolutionary computing. In: Ghosh A, Jain LC (eds) Evolutionary computation in data mining. Springer, HeidelbergGoogle Scholar
  60. 60.
    Srinivas V, Ramanjaneyulu K (2007) An integrated approach for optimum design of bridge decks using genetic algorithms and artificial neural networks. Adv Eng Softw 38: 475–487CrossRefGoogle Scholar
  61. 61.
    Tsai C-Y, Chou S-Y, Lin S-W et al (2009) Location determination of mobile devices for an indoor WLAN application using a neural network. Knowl Inf Syst 20: 81–93CrossRefGoogle Scholar
  62. 62.
    Verikas A, Bacauskiene M (2002) Feature selection with neural networks. Pattern Recognit Lett 23: 1323–1335zbMATHCrossRefGoogle Scholar
  63. 63.
    Wang G, Zhang C, Huang L (2008) A study of classification algorithm for data mining based on hybrid intelligent systems. In: Proceeding of IEEE Ninth ACIS international conference on software engineering, artificial intelligence, networking, and parallel/distributed computing. pp 371–375Google Scholar
  64. 64.
    Wang T-Y, Huang C-Y (2007) Applying optimized BPN to a chaotic time series problem. Expert Syst Appl 32: 193–200CrossRefGoogle Scholar
  65. 65.
    Wang X, Wang W, Huang Y et al (2008) Design of neural network-based estimator for tool wear modeling in hard turning. J Intell Manuf 19: 383–396CrossRefGoogle Scholar
  66. 66.
    Yamazaki A, de Souto MCP, Ludermir TB (2002) Optimization of neural network weights and architectures for odor recognition using simulated annealing. Proc IEEE Neural Netw 1: 547–552Google Scholar
  67. 67.
    Yan H, Jiang Y, Zheng J et al (2006) A multilayer perceptorn-based medical decision support system for heart disease diagnosis. Expert Syst Appl 30: 272–281CrossRefGoogle Scholar
  68. 68.
    Yang J, Honavar V (1998) Feature subset selection using a genetic algorithm. IEEE Intell Syst 34: 44–49CrossRefGoogle Scholar
  69. 69.
    Yeung DS, Zeng X-Q (2002) Hidden neuron pruning for multilayer perceptrons using a sensitivity measure. Proc IEEE Mach Learn Cybern 4: 1751–1757CrossRefGoogle Scholar
  70. 70.
    Yu S, Zhu K, Diao F (2008) A dynamic all parameters adaptive BP neural networks model and its application on oil reservoir prediction. Appl Math Comput 195: 66–75zbMATHCrossRefMathSciNetGoogle Scholar
  71. 71.
    Yuen CWM, Wong WK, Qian SQ et al (2009) A hybrid model using genetic algorithm and neural network for classifying garment defects. Expert Syst Appl 36: 2037–2047CrossRefGoogle Scholar
  72. 72.
    Zhang L, Jack LB, Nandi AK (2005) Fault detection using genetic programming. Mech Syst Signal Process 19: 271–289CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2009

Authors and Affiliations

  • Shih-Wei Lin
    • 1
    Email author
  • Shih-Chieh Chen
    • 2
  • Wen-Jie Wu
    • 1
  • Chih-Hsien Chen
    • 3
  1. 1.Department of Information ManagementChang Gung UniversityTao-YuanTaiwan, ROC
  2. 2.Department of Industrial EngineeringNational Taiwan University of Science and TechnologyTaipeiTaiwan, ROC
  3. 3.Department of ManagementFo-Guang UniversityYilanTaiwan, ROC

Personalised recommendations