Neural Network Architecture Selection: Size Depends on Function Complexity

  • Iván Gómez
  • Leonardo Franco
  • José L. Subirats
  • José M. Jerez
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4131)


The relationship between generalization ability, neural network size and function complexity have been analyzed in this work. The dependence of the generalization process on the complexity of the function implemented by neural architecture is studied using a recently introduced measure for the complexity of the Boolean functions. Furthermore an association rule discovery (ARD) technique was used to find associations among subsets of items in the whole set of simulations results. The main result of the paper is that for a set of quasi-random generated Boolean functions it is found that large neural networks generalize better on high complexity functions in comparison to smaller ones, which performs better in low and medium complexity functions.


Boolean Function Association Rule Network Size Network Architecture Hide Neuron 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Haykin, S.: Neural Networks: A Comprehensive Foundation. Macmillan/IEEE Press (1994)Google Scholar
  2. 2.
    Baum, E.B., Haussler, D.: What size net gives valid generalization? Neural Computation 1, 151–160 (1989)CrossRefGoogle Scholar
  3. 3.
    Lawrence, S., Giles, C.L., Tsoi, A.C.: What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation. In: Technical Report UMIACS-TR-96-22 and CS-TR-3617, Institute for Advanced Computer Studies, Univ. of Maryland (1996)Google Scholar
  4. 4.
    Caruana, R., Lawrence, S., Giles, C.L.: Overfitting in Neural Networks: Backpropagation, Conjugate Gradient, and Early Stopping. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 402–408. MIT Press, Cambridge (2001)Google Scholar
  5. 5.
    Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, vol. 4, pp. 950–957. Morgan Kaufmann, San Mateo (1992)Google Scholar
  6. 6.
    Prechelt, L.: Automatic Early Stopping Using Cross Validation: Quantifying the Criteria. Neural Networks 11, 761–767 (1998)CrossRefGoogle Scholar
  7. 7.
    Setiono, R.: Feedforward neural network construction using cross-validation. Neural Computation 13, 2865–2877 (2001)MATHCrossRefGoogle Scholar
  8. 8.
    Bartlett, P.L.: For valid generalization the size of the weights is more important than the size of the network. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 134–140. MIT Press, Cambridge (1997)Google Scholar
  9. 9.
    Franco, L., Anthony, M.: On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, pp. 973–978. IEEE Press, Los Alamitos (2004)Google Scholar
  10. 10.
    Franco, L.: Generalization ability of Boolean functions implemented in feedforward neural networks. Neurocomputing (2006) (In Press)Google Scholar
  11. 11.
    Franco, L., Anthony, M.: The influence of oppositely classified examples on the generalization complexity of Boolean functions. IEEE Transactions on Neural Networks (2006) (In Press)Google Scholar
  12. 12.
    Wegener, I.: The complexity of Boolean functions. Wiley and Sons Inc., Chichester (1987)MATHGoogle Scholar
  13. 13.
    Siu, K.Y., Roychowdhury, V.P., Kailath, T.: Depth-Size Tradeoffs for Neural Computation. IEEE Transactions on Computers 40, 1402–1412 (1991)CrossRefMathSciNetGoogle Scholar
  14. 14.
    Franco, L., Cannas, S.A.: Non glassy ground-state in a long-range antiferromagnetic frustrated model in the hypercubic cell. Physica A 332, 337–348 (2004)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Franco, L., Cannas, S.A.: Generalization and Selection of Examples in Feedforward Neural Networks. Neural Computation 12(10), 2405–2426 (2000)CrossRefGoogle Scholar
  16. 16.
    Franco, L., Cannas, S.A.: Generalization Properties of Modular Networks: Implementing the Parity Function. IEEE Transactions on Neural Networks 12, 1306–1313 (2001)CrossRefGoogle Scholar
  17. 17.
    Becquet, C., Blachon, S., Jeudy, B., Boulicaut, J.F., Gandrillon, O.: Strong association rules mining for large-scale gene-expression data analysis: A case study on human SAGE data. Genome Biology 3, 1–16 (2002)CrossRefGoogle Scholar
  18. 18.
    Creighton, C., Hanash, S.: Mining gene expressions databases for association rules. Bioinformatics 19, 79–86 (2003)CrossRefGoogle Scholar
  19. 19.
    Agrawal, R., Imielinski, T., Swami, A.: Mining associations rules between sets of items in large databases. In: Proceedings of the ACM SIGMOD international conference on Management of data, Washignton D.C, pp. 207–216 (1993)Google Scholar
  20. 20.
    Brian, S., Motwani, R., Silverstein, C.: Beyond Market baskets: Generalizing associations rules to correlations. In: Proceedings of the ACM SIGMOD conference, Tucson, pp. 265–276 (1997)Google Scholar
  21. 21.
    Franco, L., Jerez, J.M., Bravo, J.M.: Role of function complexity and network size in the generalization ability of feedforward networks. In: Cabestany, J., Prieto, A.G., Sandoval, F. (eds.) IWANN 2005. LNCS, vol. 3512, pp. 1–8. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Iván Gómez
    • 1
  • Leonardo Franco
    • 1
  • José L. Subirats
    • 1
  • José M. Jerez
    • 1
  1. 1.Departamento de Lenguajes y Ciencias de la ComputaciónUniversidad de MálagaMálagaSpain

Personalised recommendations