Abstract
The generalization ability of different sizes architectures with one and two hidden layers trained with backpropagation combined with early stopping have been analyzed. The dependence of the generalization process on the complexity of the function being implemented is studied using a recently introduced measure for the complexity of Boolean functions. For a whole set of Boolean symmetric functions it is found that large neural networks have a better generalization ability on a large complexity range of the functions in comparison to smaller ones and also that the introduction of a small second hidden layer of neurons further improves the generalization ability for very complex functions. Quasi-random generated Boolean functions were also analyzed and we found that in this case the generalization ability shows small variability across different network sizes both with one and two hidden layer network architectures.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Haykin, S.: Neural Networks: A Comprehensive Foundation. Macmillan/IEEE Press (1994)
Baum, E.B., Haussler, D.: What size net gives valid generalization? Neural Computation 1, 151–160 (1989)
Lawrence, S., Giles, C.L., Tsoi, A.C.: What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617, Institute for Advanced Computer Studies, Univ. of Maryland (1996)
Caruana, R., Lawrence, S., Giles, C.L.: Overfitting in Neural Networks: Backpropagation, Conjugate Gradient, and Early Stopping. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 402–408. MIT Press, Cambridge (2001)
Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, vol. 4, pp. 950–957. Morgan Kaufmann, San Francisco (1992)
Prechelt, L.: Automatic Early Stopping Using Cross Validation: Quantifying the Criteria. Neural Networks 11, 761–767 (1998)
Setiono, R.: Feedforward neural network construction using cross-validation. Neural Computation 13, 2865–2877 (2001)
Bartlett, P.L.: For valid generalization the size of the weights is more important than the size of the network. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 134–140. MIT Press, Cambridge (1997)
Franco, L., Anthony, M.: On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, pp. 973–978. IEEE Press, Los Alamitos (2004)
Wegener, I.: The complexity of Boolean functions. Wiley and Sons Inc, Chichester (1987)
Siu, K.Y., Roychowdhury, V.P., Kailath, T.: Depth-Size Tradeoffs for Neural Computation. IEEE Transactions on Computers 40, 1402–1412 (1991)
Franco, L., Cannas, S.A.: Non glassy ground-state in a long-range antiferromagnetic frustrated model in the hypercubic cell. Physica A 332, 337–348 (2004)
Franco, L., Cannas, S.A.: Generalization and Selection of Examples in Feedforward Neural Networks. Neural Computation 12(10), 2405–2426 (2000)
Franco, L., Cannas, S.A.: Generalization Properties of Modular Networks: Implementing the Parity Function. IEEE Transactions on Neural Networks 12, 1306–1313 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Franco, L., Jerez, J.M., Bravo, J.M. (2005). Role of Function Complexity and Network Size in the Generalization Ability of Feedforward Networks. In: Cabestany, J., Prieto, A., Sandoval, F. (eds) Computational Intelligence and Bioinspired Systems. IWANN 2005. Lecture Notes in Computer Science, vol 3512. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11494669_1
Download citation
DOI: https://doi.org/10.1007/11494669_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26208-4
Online ISBN: 978-3-540-32106-4
eBook Packages: Computer ScienceComputer Science (R0)