Abstract
Functional link neural network (FLNN) is a class of higher order neural networks (HONs) and have gained extensive popularity in recent years. FLNN have been successfully used in many applications such as system identification, channel equalization, short-term electric-load forecasting, and some of the tasks of data mining. The goals of this paper are to: (1) provide readers who are novice to this area with a basis of understanding FLNN and a comprehensive survey, while offering specialists an updated picture of the depth and breadth of the theory and applications; (2) present a new hybrid learning scheme for Chebyshev functional link neural network (CFLNN); and (3) suggest possible remedies and guidelines for practical applications in data mining. We then validate the proposed learning scheme for CFLNN in classification by an extensive simulation study. Comprehensive performance comparisons with a number of existing methods are presented.
Similar content being viewed by others
References
Haykin S (1999) Neural networks—a comprehensive foundation. Prentice Hall, Englewood Cliffs
Zhang GP (207) Avoiding pitfalls in neural network research. IEEE Trans Syst Man Cybern Part C Appl Rev 37(1):3–16
Anders U, Korn O (1999) Model selection in neural networks. Neural Netw 12:309–323
Benitez JM, Castro JL, Requena I (1997) Are artificial neural networks black boxes? IEEE Trans Neural Netw 8(5):1156–1164
Cheng B, Titterington D (1994) Neural networks: a review from a statistical perspective. Stat Sci 9(1):2–54
McCulloch W, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 7:115–133
Giles CL, Maxwell T (1987) Learning invariance, and generalization in a higher order neural networks. Appl Opt 26(23):4972–4978
Belli MR, Conti M, Crippa P, Turchetti C (1999) Artificial neural networks as approximators of stochastic processes. Neural Netw 12(4–5):647–658
Castro JL, Mantas CJ, Benitez JM (2000) Neural networks with a continuous squashing function in the output are universal approximators. Neural Netw 13(6):561–563
Funahashi K (1989) On the approximate realization of continuous mappings by neural networks. Neural Netw 2:183–192
Andrews R, Diederich J, Tickle AB (1995) Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl Based Syst 8(6):373–389
Castro JL, Requena I, Benitez JM (2002) Interpretation of artificial neural networks by means of fuzzy rules. IEEE Trans Neural Netw 13(1):101–116
Setiono R, Leow WK, Zurada J (2002) Extraction of rules from artificial neural networks for nonlinear regression. IEEE Trans Neural Network 13(3):564–577
Setiono R, Thong JYL (2004) An approach to generate rules from neural networks for regression problems. Eur J Oper Res 155:239–250
Gish H (1990) A probabilistic approach to the understanding and training of neural network classifiers. In: Proc IEEE international conference acoustic, speech signal process 3:1361–1364
Zhang GP (2000) Neural networks for classification: a survey. IEEE Trans Syst Man Cybern C 30(4):451–462
Michie D, Spiegelhalter DJ, Taylor CC (1994) Machine learning, neural and statistical classification. Ellis Horwood, New York
Adya M, Collopy F (1998) How effective are neural networks at forecasting and prediction? A review and evaluation. J Forecast 17:481–495
Callen JL, Kwan CCY, Yip PCY, Yuan Y (1996) Neural network forecasting of quarterly accounting earnings. Int J Forecast 12:475–482
Church KB, Curram SP (1996) Forecasting comsumers’ expenditure: a comparison between econometric and neural network models. Int J Forecast 12:255–267
Connor JT, Martin RD, Atlas LE (1994) Recurrent neural networks and robust time series prediction. IEEE Trans Neural Netw 51(2):240–254
Cottrell M, Girard B, Girard Y, Mangeas M, Muller C (1995) Neural modeling for time series: a statistical stepwise method for weight elimination. IEEE Trans Neural Netw 6(6):1355–1364
Faraway JJ, Chatfield C (1998) Time series forecasting with neural networks: a comparative study using the airline data. Appl Stat 47:231–250
Fletcher D, Goss E (1993) Forecasting with neural networks—an application using bankruptcy data. Inf Manag 24:159–167
Gorr WL (1994) Research prospective on neural network forecasting. Int J Forecast 10:1–4
Hippert HS, Pedreira CE, Souza RC (2001) Neural networks for short-term forecasting: a review and evaluation. IEEE Trans Power Syst 16(1):44–55
Hu MY, Zhang GP, Jiang CX, Patuwo BE (1999) A cross-validation analysis of neural network out-of-sample performance in exchange rate forecasting. Decis Sci 30:197–216
Kaastra I, Boyd M (1996) Designing a neural network for forecasting financial and economic time series. Neurocomputing 10:215–236
Maier HR, Dandy GC (2000) Neural networks for the prediction and forecasting of water resources variables: a review of modeling issues and applications. Environ Model Softw 15:101–124
Park YR, Murray TJ, Chen C (1996) Predicting sun spots using a layered perceptron neural network. IEEE Trans Neural Netw 7(2):501–505
Qi M, Zhang GP (2001) An investigation of model selection criteria for neural network time series forecasting. Eur J Oper Res 132:666–680
Kracha KA, Wagner U (1999) Applications of artificial neural networks in management science: a survey. J Retail Consum Serv 6:185–203
Wong BK, Bodnovich TA, Selvi Y (1997) Neural network applications in business: a review and analysis of the literature (1988–1995). Decis Support Syst 19:301–320
Flood I, Kartam N (1994) Neural network in civil engineering-I: principles and understanding. J Comput Civil Eng 8(2):131–148
Lu CN, Wu HT, Vemuri S (1993) Neural network based short term load forecasting. IEEE Trans Power Syst 8(1):336–342
Lisboa PJG (2002) A review of evidence of health benefit from artificial neural networks in medical intervention. Neural Netw 15:11–39
Protney LG, Watkins MP (2000) Foundations of clinical research: applications to practice. Prentice-Hall, Princeton
Hosseini-Nezhad SM, Yamashita TS, Bielefeld RA, Krug SE, Pao YH (1995) A neural network approach for the determination of interhospital transport mode. Comput Biomed Res 28(4):319–334
Tawfik H, Liatsis P (1997) Prediction of non-linear time series using higher order neural networks. In: Proceeding IWSSIP1997 conference, Poznan, Poland
Kaita T, Tomita S, Yamanaka J (2002) On a higher order neural network for distortion invariant pattern recognition. Pattern Recognit Lett 23:977–984
Ghosh J, Shin Y (1992) Efficient higher-order neural networks for classification and function approximation. Int J Neural Syst 3:323–350
Minsky M, Papert S (1969) Perceptrons. The MIT Press
Widrow B, Hoff ME (1960) Adaptive switching circuits. IRE WESCON Convention Record, pp 96–104
Widrow B, Lehr M (1990) 30 years of adaptive neural networks: perceptron, madaline, and back-propagation. Proc IEEE 78(9):1415–1442
Cover TM (1965) Geometrical and statistical properites of systems of linear inequalities with applications in pattern recognition. IEEE Trans Electron Comput 14:326–334
Hornik K et al (1989) Multi-layer feed-forward networks are universal approximators. Neural Netw 2:359–366
Giles CL, Maxwell T (1987) Learning, invariance and generalization in higher-order neural networks. Appl Opt 26(23):4972-4978
Pao YH (1989) Adaptive pattern recognition and neural network. Addison-Wesley, Reading, MA
Venkatesh SS, Baldi P (1991) Programmed interactions in higher order neural networks: maximal capacity. J Complex 7:316–337
Antyomov E, Pecht OY (2005) Modified higher order neural network for invariant pattern recognition. Pattern Recognit Lett 26:843–851
Misra BB, Dehuri S (2007) Functional link neural network for classification task in data mining. J Comput Sci 3(12):948–955
Mirea L, Marcu T (2002) System identification using functional link neural networks with dynamic structure. 15th Triennial World Congress, Barcelona, Spain
Cass R, Radl B (1996) Adaptive process optimization using functional link networks and evolutionary algorithms. Control Eng Pract 4(11):1579–1584
Pao Y-H, Philips SM (1995) The functional link net learning optimal control. Neurocomputing 9:149–164
Shin Y, Ghosh J (1995) Ridge polynomial networks. IEEE Trans Neural Netw 6(2):610–622
Shin Y, Ghosh J (1992) Approximation of multivariate functions using ridge polynomial networks. In: Proceedings of international joint conference on neural networks II, pp 380–385
Voutriaridis C, Boutalis YS, Mertzios G (2003) Ridge polynomial networks in pattern recognition. 4th EURASIP conference focused on video/image processing and multimedia communications, Croatia, pp 519–524
Shin Y, Ghosh J (1991) The pi-sigma networks: an efficient higher order neural network for pattern classification and function approximation. In: Proceedings of international joint conference on neural networks I, pp 13–18
Shin Y, Ghosh J (1992) Computationally efficient invariant pattern recognition with higher order pi-sigma networks. The University of Texas at Austin, Tech. Report
Shin Y, Ghosh J (1991) Realization of boolean functions using binary pi-sigma networks. In: Proceedings of conference on artificial neural networks in engineering, St. Louis
Hussain AJ, Liatsis P (2002) Recurrent pi-sigma networks for DPCM image coding. Neurocomputing 55:363–382
Xiong Y et al (2007) Training pi-sigma network by on-line gradient algorithm with penalty for small weight update. Neural Comput 19:3356–3368
Iyoda EM et al (2007) Image compression and reconstruction using pi t -sigma neural networks. Soft Comput 11:53–61
Hussain AJ et al (2008) Physical time series prediction using recurrent pi-sigma neural networks. Int J Artif Intell Soft Comput 1(1):130–145
Nie Y, Deng W (2008) A hybrid genetic learning algorithm for pi-sigma neural network and the analysis of its convergence. In: Proceedings of fourth international conference on natural computation, IEEE Press, pp 19–23
Zhu Q, Cai Y, Liu L (1999) A global learning algorithm for a RBF network. Neural Netw 12:527–540
Li M, Tian J, Chen F (2008) Imrpoving multiclass pattern recognition with a co-evolutionary RBFNN. Pattern Recognit Lett 29:392–406
Dybowski R (1998) Classification of incomplete feature vectors by radial basis function networks. Pattern Recognit Lett 19:1257–1264
Leonardis A, Bischof H (1998) An efficient MDL based construction of RBF networks. Neural Netw 11:963–973
Chen S, Wu Y, Luk BL (1999) Combined genetic algorithm optimization and regularized orthogonal least square learning for radial basis function networks. IEEE Tran Neural Netw 10(5):1239–1243
Lee YC, Doolen G, Chen HH, Sun GZ, Maxwell T, Lee HY, Giles CL (1986) Machine learning using a higher order correlation network. Physica 22D:276–306
Peretto P, Niez JJ (1986) Long-term memory storage capacity of multiconnected neural networks. Biol Cybern 54:5363
Psaltis D, Park CH (1986) Nonlinear discriminant functions and associative memories. In: Denker JS (ed) Neural networks for computing. Amererican Institute of Physics, New York, pp 370–375
Gardner E (1987) Multiconnected neural-network models. J Phys A Math Gen 20:3453–3464
Abbott LF, Arian Y (1987) Storage capacity of generalized networks. Phys Rev A 36:5091–5094
Kamp Y, Hasler M (1990) Recursive neural networks for associative memory. Wiley, New York
Horn D, Usher M (1988) Capacities of multiconnected memory models. J Phys France 49:389–395
Guillermo V (1998) A distributed approach to neural network simulation program. Master thesis, The University of Texas at E1 Paso, TX
Zurada JM (1992) Introduction to artificial neural system. West Publishing Company, St. Paul, MN
Beale R, Jackson T (1991) Neural computing: an introduction. Hilger, Philadelphia, PA
Haring B, Kok JN (1995) Finding functional links for neural networks by evolutionary computation. In: Van de Merckt T et al (eds) BENELEARN1995, proceedings of the fifth Belgian–Dutch conference on machine learning, Brussels, Belgium, pp 71–78
Panagiotopoulos DA et al (1999) Planning with a functional neural network architecture. IEEE Trans Neural Netw 10(1):115–127
Patra JC et al (1999) Identification of non -linear dynamic systems using functional link artificial neural networks. IEEE IEEE Trans Syst Man Cyber Part B Cybern 29(2):254–262
Sierra A, Macias JA, Corbacho F (2001) Evolution of Functional Link Networks. IEEE Tranas Evol Comput 5(1):54–65
Marcu T, Koppen-Seliger B (2004) Dynamic functional link neural networks genetically evolved applied to system identification. In: Proceedings of ESANN’2004, Bruges (Belgium), pp 115–120
Patra JC, Pal NR (1995) A functional link neural network for adaptive channel equalization. Signal Process 43:181–195
Zhao H, Zhang J (2008) Functional link neural network cascaded with Chebyshev orthogonal polynomial for non-linear channel equalization. signal Process 88:1946–1957
Haring et al (1997) Feature selection for neural networks through functional links found by evolutionary computation. In: Liu X et al (eds) Adavnces in intelligent data analysis (IDA-97). LNCS 1280:199–210
Patra JC et al (2000) Modelling of an intelligent pressure sensor using functional link artificial neural networks. ISA Trans 39:15–27
Dehuri S et al (2008) Genetic feature selection for optimal functional link neural network in classification. In: Fyfe C et al (eds) IDEAL 2008, LNCS 5326:156–163
Majhi B et al (2005) An improved scheme for digital watermarking using functional link artificial neural network. J Comput Sci 1(2):169–174
Patra JC et al (2008) Functional link neural networks-based intelligent sensors for Harsh Environments. Sens Transducers J 90:209–220
Dash PK et al (1999) A functional link neural network for short term electric load forecasting. J Intell Fuzzy Syst 7:209–221
Krishnaiah D et al (2008) Application of ultrasonic waves coupled with functional link neural network for estimation of carrageenan concentration. Int J Phys Sci 3(4):90–96
Sing SN, Srivastava KN (2002) Degree of insecurity estimation in a power system using functional link neural network. ETEP 12(5):353–359
Abu-Mahfouz I-A (2005) A comparative study of three artificial neural networks for the detection and classification of gear faults. Int J Gen Syst 34(3):261–277
Hu Y-C, Tseng F-M (2007) Functional-link net with fuzzy integral for bankruptcy prediction. Neurocomputing 70:2959–2968
Park GH, Pao YH (2000) Unconstrained word-based approach for off-line script recognition using density based random vector functional link net. Neurocomputing 31:45–65
Hu Y-C (2008) Functional link nets with genetic algorithm based learning for robust non-linear interval regression analysis. Neurocomputing. doi:10.1016/J.neucom.2008.07.002
Chen CLP et al (1998) An incremental adaptive implementation of functional link processing for function approximation, time series prediction, and system identification. Neurocomputing 18:11–31
Weng W-D, Yen CT (2004) Reduced decision feed-back FLANN non-linear channel equaliser for digital communication systems. IEE Proc Commun 151(4):305–311
Hussain A et al (1997) A new adaptive functional link neural network based DFE for overcoming co-channel interference. IEEE IEEE Trans Commun 45(11):1358–1362
Patra JC et al (1999) Non-linear channel equalization for QAM signal constellation using artificial neural networks. IEEE Tranasactions on Systems, Man, Cybernetics-Part B: Cybernetics 29(2):262–271
Purwar S et al (2007) On-line system identification of complex systems using Chebyshev neural networks. Appl Soft Comput 7:364–372
Weng W-D et al (2007) A channel equalizer usi ng reduced decision feedback Chebyshev function link artificial neural networks. Inf Sci 177:2642–2654
Patra JC et al (2002) Non-linear dynamic system identification using Chebyshev functional link artificial neural networks. IEEE Trans Syst Man Cybern Part B Cybern 32(4):505–511
Fogel DB (2000) Evolutionary computation: towards a new philosophy of machine intelligence. IEEE Press, New York
Pearson DW et al (eds) (1995) Artificial neural networks and genetic algorithms. Springer Verlag
Suzuki J (1995) A Markov chain analysis on simple genetic algorithms. IEEE Trans Syst Man Cybern 25(4):6–659
Kennedy J, Eberhart RC (1995) Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks. Pisacataway, NJ, pp 1942–9148
Schaffer JD, Whitley D, Eshelman LJ (1992) Combinations of genetic algorithms and neural networks: a survey of the state of the art. In: Proceedings of international workshop on combinations of genetic algorithms and neural networks pp 1–37
Davidor Y (1990) Epistasis variance: suitability of a representation to genetic algorithms. Complex Syst 4:368–383
Eshelman LJ, Schaffer JD (1993) Real coded genetic algorithms and interval schemata. In: Whitley LD (ed) Foundation of genetic algorithms. Morgan Kaufmann, San Mateo, pp 187–202
Muhlenbein H, Schlierkamp-Voosen D (1993) Predictive models for the breeder genetic algorithm I. Continuous parameters optimization. Evol Comput 1(1):24–49
Schutte JF, Groenwold AA (2005) A study of global optimization using particle swarms. J Glob Optim 31(1):93–108
Ali MM, Kaelo P (2008) Improved particle swarm algorithms for global optimization. Appl Math Comput 196:578–593
Yu J, Wang S, Xi L (2008) Evolving artificial neural networks using an improved PSO and DPSO. Neurocomputing 71:1054–1060
Da Y, Ge XR (2005) An improved PSO-based ANN with simulated annealing technique. Neurocomput Lett 63:527–533
Blake CL, Merz CJ (1998) UCI repository of machine learning databases. http://www.ics.uci.edu/mlearn/MLRepository.html
Pao Y-H, Phillips SM, Sobajic DJ (1992) Neural-net computing and intelligent control systems. Int J Control 56(2):263–289
Hornik K (1991) Approximation capabilities of multilayer feed-forward networks. Neural Netw 4:251–257
Smith KA, Gupta JND (2002) Neural networks in business: techniques and applications. Idea Group, Hershey, PA
Lee TT, Jeng JT (1998) The Chebyshev polynomial based unified model neural networks for function approximations. IEEE Trans Syst Man Cybern Part B 28:925–935
Namatame A, Veda N (1992) Pattern classification with Chebyshev neural network. Int J Neural Netw 3:23–31
Klasser MS, Pao YH (1988) Characteristics of the functional link net: a higher order delta rule net. IEEE proceedings of 2nd annual international conference on neural networks, San Diago, CA
Pao YH, Takefuji Y (1992) Functional link net computing: theory, system, architecture and functionalities. IEEE Comput, pp 76–79
Goldberg DE (1989) Genetic algorithms in search, optimization and machine learning. Morgan Kaufmann, San Mateo
Kennedy J, Eberhart RC (1999) The particle swarm: social adaptation in information processing systems. In: Corne D, Dorigo M, Glover F (eds) New ideas in optimization. McGraw–Hill, Cambridge, UK, pp 379–387
Shi Y, Eberhart RC (1998) A modified particle swarm optimizer. In: Proceedings of the IEEE international conference on evolutionary computation. IEEE Press, Pisacataway, NJ, pp 69–73
Shi Y, Eberhart RC (1998) Parameter selection in particle swarm optimization. Evolutionary Programming VII, LNCS, Springer, Berlin 1447:591–600
Forie PC, Groenwold AA (2002) The particle swarm optimization algorithm in size and shape optimization. Struct Multidiscipl Optim 23(4):259–267
Clerc M, Kennedy J (2002) The particle swarm explosion, stability and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73
Zhang JR et al (2007) A hybrid particle swarm optimization-back-propagation algorithm for feed-forward neural network training. Appl Math Comput 185:1026–1037
Ratnaweera A, Halgamuge SK, Watson HC (2004) Self-organizing hierarchical particle swarm optimizer with time varying acceleration coefficients. IEEE Trans Evol Comput 8(3):240–255
Lippmann R (1987) An introduction to computing with neural networks. IEEE ASSP Mag 4:4–22
Preshelt L (1994) Proben1-a set of neural network benchmark problems and benchmarking rules. Technical Report 21/94, Universitat Karlsruhe, Germany
Ghosh A, Dehuri S, Ghosh S (2008) Multi-objective evolutionary algorithms for knowledge discovery from databases. Springer
Kriegel H-P et al (2007) Future trends in data mining. Data Mining Knowl Discov 15(1):87–97
Vellido A, Lisboa PJG, Vaughan J (1999) Neural networks in business: a survey of applications (1992–1998). Expert Syst Appl 17:51–70
Liatsis P, Hussain AJ (1999) Non-linear one dimensional DPCM image prediction using polynomial neural network. In: Proceedings of SPIE applications of artificial neural networks in image processing IV, San Jose, CA 3647:58–68
Acknowledgments
Authors would like to thank BK21 research program on Next Generation Mobile Software at Yonsei University, South Korea for their financial support. The authors greatly appreciate all the reviewers’ constructive comments that motivated them to think more and improve the presentation of this paper.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Dehuri, S., Cho, SB. A comprehensive survey on functional link neural networks and an adaptive PSO–BP learning for CFLNN. Neural Comput & Applic 19, 187–205 (2010). https://doi.org/10.1007/s00521-009-0288-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-009-0288-5