Role of Function Complexity and Network Size in the Generalization Ability of Feedforward Networks
- 1.3k Downloads
The generalization ability of different sizes architectures with one and two hidden layers trained with backpropagation combined with early stopping have been analyzed. The dependence of the generalization process on the complexity of the function being implemented is studied using a recently introduced measure for the complexity of Boolean functions. For a whole set of Boolean symmetric functions it is found that large neural networks have a better generalization ability on a large complexity range of the functions in comparison to smaller ones and also that the introduction of a small second hidden layer of neurons further improves the generalization ability for very complex functions. Quasi-random generated Boolean functions were also analyzed and we found that in this case the generalization ability shows small variability across different network sizes both with one and two hidden layer network architectures.
KeywordsHide Layer Boolean Function Network Size Symmetric Function Hide Neuron
Unable to display preview. Download preview PDF.
- 1.Haykin, S.: Neural Networks: A Comprehensive Foundation. Macmillan/IEEE Press (1994)Google Scholar
- 3.Lawrence, S., Giles, C.L., Tsoi, A.C.: What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617, Institute for Advanced Computer Studies, Univ. of Maryland (1996)Google Scholar
- 4.Caruana, R., Lawrence, S., Giles, C.L.: Overfitting in Neural Networks: Backpropagation, Conjugate Gradient, and Early Stopping. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 402–408. MIT Press, Cambridge (2001)Google Scholar
- 5.Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, vol. 4, pp. 950–957. Morgan Kaufmann, San Francisco (1992)Google Scholar
- 8.Bartlett, P.L.: For valid generalization the size of the weights is more important than the size of the network. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 134–140. MIT Press, Cambridge (1997)Google Scholar
- 9.Franco, L., Anthony, M.: On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, pp. 973–978. IEEE Press, Los Alamitos (2004)Google Scholar