Abstract
The selection of the frequencies of the new hidden units for sequential Feed-forward Neural Networks (FNNs) usually involves a non-linear optimization problem that cannot be solved analytically. Most models found in the literature choose the new frequency so that it matches the previous residue as best as possible. Several exceptions to the idea of matching the residue perform an (implicit or explicit) orthogonalization of the output vectors of the hidden units. An experimental study of the aforementioned approaches to select the frequencies in sequential FNNs is presented. Our experimental results indicate that the orthogonalization of the hidden vectors outperforms the strategy of matching the residue, both for approximation and generalization purposes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Geman, S., Bienenstock, E., Doursat, R.: Neural Networks and the Bias/Variance Dilemma. Neural Computation 4, 1–58 (1992)
Barron, A.R.: Approximation and Estimation Bounds for Artificial Neural Networks. Machine Learning 14, 115–133 (1994)
Kwok, T.Y., Yeung, D.Y.: Constructive Algorithms for Structure Learning in Feedforward Neural Networks for Regression Problems. IEEE Transactions on Neural Networks 8, 630–645 (1997)
Krková, V.: Incremental Approximation by Neural Networks. In: Karny, M., Warwick, K., Krková, V. (eds.) Dealing With Complexity: A Neural Network Approach, pp. 177–188. Springer, London (1998)
Kwok, T.Y., Yeung, D.Y.: Objective Functions for Training New Hidden Units in Constructive Neural Networks. IEEE Transactions on Neural Networks 8, 1131–1148 (1997)
Achieser, N.I.: Theory of Approximation. Frederick Ungar Pub. Co, New York (1956)
Friedman, J.H., Stuetzle, W.: Projection Pursuit Regression. Journal of the American Statistical Association 76, 817–823 (1981)
Mallat, S.G., Zhang, Z.: Matching Pursuits with Time-Frequency Dictionaries. IEEE Transactions on Signal Processing 41, 3397–3415 (1993)
Hwang, J.N., Ray, S.R., Maechler, M., Martin, D., Schimert, J.: Regression Modelling in Back-Propagation and Projection Pursuit Learning. IEEE Transactions on Neural Networks 5, 342–353 (1994)
Krková, V., Beliczyński, B.: Incremental Approximation by One-Hidden-Layer Neural Networks. In: International Conference on Artificial Neural Networks, vol. 1, pp. 505–510 (1995)
Chen, S., Cowan, C.F.N., Grant, P.M.: Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks. IEEE Transactions on Neural Networks 2, 302–309 (1991)
Vincent, P., Bengio, Y.: Kernel Matching Pursuit. Machine Learning 48, 165–187 (2002); Special Issue on New Methods for Model Combination and Model Selection
Romero, E., Alquézar, R.: A New Incremental Method for Function Approximation using Feed-forward Neural Networks. In: International Joint Conference on Neural Networks, vol. 2, pp. 1968–1973 (2002)
Jones, L.K.: On a Conjecture of Huber Concerning the Convergence of Projection Pursuit Regression. The Annals of Statistics 15, 880–882 (1987)
Romero, E.: Learning with Feed-forward Neural Networks: Three Schemes to Deal with the Bias/Variance Trade-off. PhD thesis, Departament de Llenguatges i Sistemes Informàtics, Universitat Politècnica de Catalunya, Spain (2004)
Treadgold, N.K., Gedeon, T.D.: Exploring Constructive Cascade Networks. IEEE Transactions on Neural Networks 10, 1335–1350 (1999)
Ma, L., Khorasani, K.: New Training Strategies for Constructive Neural Networks with Application to Regression Problems. Neural Networks 17, 589–609 (2004)
Mühlenbein, H., Schlierkamp-Voosen, D.: Predictive Models for the Breeder Genetic Algorithm I. Continuous Parameter Optimization. Evolutionary Computation 1, 25–49 (1993)
Liu, H., Motoda, H.: Feature Selection for Knowledge Discovery and Data Mining. Kluwer Academic Publishers, Dordrecht (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Romero, E. (2005). Selection of Weights for Sequential Feed-Forward Neural Networks: An Experimental Study. In: Cabestany, J., Prieto, A., Sandoval, F. (eds) Computational Intelligence and Bioinspired Systems. IWANN 2005. Lecture Notes in Computer Science, vol 3512. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11494669_28
Download citation
DOI: https://doi.org/10.1007/11494669_28
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26208-4
Online ISBN: 978-3-540-32106-4
eBook Packages: Computer ScienceComputer Science (R0)