Review of input determination techniques for neural network models based on mutual information and genetic algorithms

Original Article

Abstract

The use of artificial neural networks (ANNs) models has grown considerably over the last decade. One of the difficulties in using ANNs is the fact that in most cases there are several numbers of input variables available. In the past, there was a tendency to use a large number of inputs in ANNs applications. This can have a number of detrimental effects on the network during training and it also requires a greater amount of data to efficiently estimate the connection weights. Additional inputs tend to increase the required time for training and the risk of the training algorithm becoming stuck in a local minimum. A large number of inputs also increases the risk of including spurious variables that merely increase the noise in the forecasts. Consequently, it is important to use an appropriate selection technique of the input variables in order to obtain the smallest number of independent inputs that are useful predictors for the system which is being researched. The aim of this paper is to review techniques that will allow the selection of appropriate model inputs based particularly on mutual information and genetic algorithms.

Keywords

Input determination techniques Artificial neural networks Mutual information Genetic algorithms 

References

  1. 1.
    Ahmed A, Dey L (2005) A feature selection technique for classificatory analysis. Pattern Recognit Lett 26(1):43–56. doi:10.1016/j.patrec.2004.08.015 CrossRefGoogle Scholar
  2. 2.
    Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–550CrossRefGoogle Scholar
  3. 3.
    Bowden GJ, Maier HR, And Dandy GC (2005) Input determination for neural network models in water resources applications, Part 1. Background and methodology. J Hydrol (Amst) 301:75–92. doi:10.1016/j.jhydrol.2004.06.021 CrossRefGoogle Scholar
  4. 4.
    Cai S, Toral H, Qiu J, Archer JS (1994) Neural network based objective flow regime identification in air-water two phase flow. Can J Chem Eng 72:440–445CrossRefGoogle Scholar
  5. 5.
    Davies L, Gather U (1993) The identification of multiple outliers. J Am Stat Assoc 88(423):782–792. doi:10.2307/2290763 MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Fano RM (1961) Transmission of information: a statistical theory of communications. Willey, New YorkGoogle Scholar
  7. 7.
    Fernando TMG, Maier HR, Dandy GC, May R (2005) Efficient selection of inputs for artificial neural network modelsGoogle Scholar
  8. 8.
    Fraser AM, Swinney HL (1986) Independent coordinates for strange attractors for mutual information. Phys Rev A 33(2):1134–1140. doi:10.1103/PhysRevA.33.1134 CrossRefMathSciNetGoogle Scholar
  9. 9.
    Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182. doi:10.1162/153244303322753616 MATHCrossRefGoogle Scholar
  10. 10.
    Islam S, Kothari R (2000) Artificial neural networks in remote sensing of hydrological processes. J Hydrol Eng 5(2):138–144. doi:10.1061/(ASCE)1084-0699(2000)5:2(138) CrossRefGoogle Scholar
  11. 11.
    Huang J, Cai Y, Xu X (2006a) A filter approach to feature selection based on mutual information. Proceedings of 5th IEEE international conference on cognitive informaticsGoogle Scholar
  12. 12.
    Huang J, Lv N, Li W (2006b) A novel feature selection approach by hybrid Genetic Algorithm. In: PRICAI 2006, LNAI, vol 4099, pp 721–729Google Scholar
  13. 13.
    Huang J, Cai Y, Xu X (2006c) A wrapper for feature selection based on mutual information. The 18th international conference on pattern recognition (ICPR’06)Google Scholar
  14. 14.
    Huang J, Cai Y, Xu X (2007) A hybrid genetic algorithm for feature selection wrapper based on mutual information. Pattern Recognit Lett 28:1825–1844. doi:10.1016/j.patrec.2007.05.011 CrossRefGoogle Scholar
  15. 15.
    Kohavi R, John G (1997) Wrappers for feature selection. Artif Intell 97(1–2):273–324. doi:10.1016/S0004-3702(97)00043-X MATHCrossRefGoogle Scholar
  16. 16.
    Kohonen T (1982) Self-organized formation of topologically correct feature maps. Biol Cybern 43:59–69. doi:10.1007/BF00337288 MATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Kohonen T (1990) The self-organizing map. Proc IEEE 78(9):1464–1480. doi:10.1109/5.58325 CrossRefGoogle Scholar
  18. 18.
    Kwak N, Choi C-H (2002) Input feature selection for classification problems. IEEE Trans Neural Netw 13(1):143–159. doi:10.1109/72.977291 CrossRefGoogle Scholar
  19. 19.
    Moon YI, Rajagopalan B, Lall U (1995) Estimation of mutual information using kernel density estimators. Phys Rev A 52(3):2318–2321Google Scholar
  20. 20.
    Moon YI, Rajagopalan B, Lall U (1995) Estimation of mutual information using kernel density estimators. Phys Rev R 52(3):2318–2321Google Scholar
  21. 21.
    Sarle WS (1997) Neural network FAQ, periodic posting to the Usenet newsgroup. www.comp.ai.neural-nets
  22. 22.
    Scott DW (1992) Multivariate density estimation: theory, practice and visualization. Probability and mathematical statistic. Wiley, New York, p 317Google Scholar
  23. 23.
    Sharma A (2000) Seasonal to inter-annual rainfall probabilistic forecast for improved water supply management: Part 1—a strategy for system predictor identification. J Hydrol (Amst) 239:232–239. doi:10.1016/S0022-1694(00)00346-2 CrossRefGoogle Scholar
  24. 24.
    Silverman BW (1986) Density estimation for statistics and data analysis, monographs on statistics and applied probability. Chapman and Hall, New York, p 175Google Scholar
  25. 25.
    Specht DF (1991) A general regression neural network. IEEE Trans Neural Netw 2(6):568–576. doi:10.1109/72.97934 CrossRefGoogle Scholar
  26. 26.
    Zhang CK, Hu H (2005a) An effective feature selection scheme via genetic algorithm using mutual information. In: Fuzzy systems and knowledge discovery, PT 2, Proceedings, vol 3614, pp 73–80 Part 2Google Scholar
  27. 27.
    Zhang CK, Hu H (2005b) Feature selection using the hybrid of ant colony optimization and mutual information for the forecaster. Proceedings of 2005 international conference on machine learning and cybernetics, vol 1–9, pp 1728–1732Google Scholar

Copyright information

© Springer-Verlag London Limited 2008

Authors and Affiliations

  1. 1.Departamento de Matemática, Faculdade de Ciências e Tecnologia da UNLQuinta da Torre, Monte de CaparicaMonte de CaparicaPortugal

Personalised recommendations