Tight bounds on the size of neural networks for classification problems

  • Valeriu Beiu
  • Thierry de Pauw
Neural Nets Simulation, Emulation and Implementation
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1240)


This paper relies on the entropy of a data-set (i.e., number-of-bits) to prove tight bounds on the size of neural networks solving a classification problem. First, based on a sequence of geometrical steps, we constructively compute an upper bound of O(mn) on the number-of-bits for a given data-set—here m is the number of examples and n is the number of dimensions (i.e., IRn). This result is used further in a nonconstructive way to bound the size of neural networks which correctly classify that data-set.


neural networks size complexity entropy classification problems Boolean functions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abu-Mostafa, Y.S.: Connectivity versus entropy, in D.Z. Anderson (ed): Neural information processing systems (NIPS*87, Denver, CO), American Institute of Physics, NY (1988) 1–8.Google Scholar
  2. 2.
    Arai, M.: Bounds on the number of hidden units in binary-valued three-layer neural networks, Neural Networks 6(6) (1993) 855–860.Google Scholar
  3. 3.
    Arbib, M.A. (ed.): The handbook of brain theory and neural networks, MIT Press Cambridge, MA (1995).Google Scholar
  4. 4.
    Baum, E.B.: On the Capabilities of Multilayer Perceptrons, J. of Complexity 4 (1988) 193–215.Google Scholar
  5. 5.
    Baum, E.B.: When are k-nearest neighbour and back propagation accurate for feasible size sets of examples ?, in E. Amaldi and E. Mayoraz: Mathematical foundations of artificial neural networks (summer school lecture notes), Swiss Federal Institute of Technology (Lausanne) and Kurt Bösch Academic Institute (Sion), 1992. Also in S.J. Hanson, G.A. Drastal and R.L. Rivest (eds.): Computational learning theory and natural learning systems—Vol. I: Constraints and prospects, Bradford/MIT Press (1994).Google Scholar
  6. 6.
    Beiu, V.: Entropy bounds for classification algorithms, Neural Network World 6(4) (1996) 497–505.Google Scholar
  7. 7.
    Beiu, V.: Digital integrated circuit implementations, Chapter E1.4 in [13].Google Scholar
  8. 8.
    Beiu, V.: VLSI complexity of discrete neural networks, Gordon and Breach (1997).Google Scholar
  9. 9.
    Beiu, V., Taylor, J.G.: VLSI-optimal neural network learning algorithm, in D.W. Pearson, N.C. Steele and R.F. Albrecht (eds.): Artificial neural nets and genetic algorithms (ICANNGA'95, Alès, France), Springer-Verlag, Vienna (1995) 61–64.Google Scholar
  10. 10.
    Beiu, V., Taylor, J.G.: Direct synthesis of neural networks, Proc. MicroNeuro'96 (Lausanne, Switzerland), IEEE CS Press, Los Alamitos, CA (1996) 257–264.Google Scholar
  11. 11.
    Bishop, C.M.: Neural networks for pattern recognition, Oxford University Press, Oxford, UK (1995).Google Scholar
  12. 12.
    Bruck, J., Smolensky, R.: Polynomial threshold functions, AC 0 functions and spectral norms, Res. Rep. RJ 7410 (67387), 11/15/89, IBM Yorktown Heights, NY (1989). Also in SIAM J. Computing 21(1) (1992) 33–42.Google Scholar
  13. 13.
    Fiesler, E., Beale, R. (eds.): Handbook of neural computation. Oxford University Press and the Institute of Physics Publishing, NY (1996).Google Scholar
  14. 14.
    Hassoun, M.H.: Fundamentals of artificial neural networks, MIT Press, Cambridge, MA (1995).Google Scholar
  15. 15.
    Hecht-Nielsen, R., Neurocomputing, Addison-Wesley, Reading, MA (1989).Google Scholar
  16. 16.
    Huang, S.-C., Huang, Y.-F.: Bounds on the number of hidden neurons of multilayer perceptrons in classification and recognition, IEEE Trans. on Neural Networks 2(1) (1991) 47–55.Google Scholar
  17. 17.
    Kohonen, T.: Self-organizing and associative memory, Springer-Verlag, Berlin (1988).Google Scholar
  18. 18.
    Krishnamoorthy, A.V., Paturi, R., Blume, M., Linden, G.D., Linden L.H., Esener, S.C.: Hardware tradeoffs for Boolean concept learning, Proc. World Conf. on Neural Networks '94 (WCNN'94, San Diego), Lawrence Erlbaum & INNS Press, Hillsdale (1994) vol. 1, 551–559.Google Scholar
  19. 19.
    Lupanov, O.B.: The synthesis of circuits from threshold elements, Problemy Kibernetiki 20 (1973) 109–140.Google Scholar
  20. 20.
    Neciporuk, E.I.: The synthesis of networks from threshold elements, Problemy Kibernetiki 11 (1964) 49–62. English translation in Automation Express 7(1) 35–39 and 7(2) 27–32.Google Scholar
  21. 21.
    Parberry, I.: Circuit complexity and neural networks, MIT Press, Cambridge, MA (1994).Google Scholar
  22. 22.
    Paugam-Moisy, H.: Optimisation des réseaux des neurones artificiels. Ph.D. thesis, Laboratoire de l'Informatique du Parallélisme LIP-IMAG, École Normale Supérieure de Lyon, 46 Allée d'Italie, 69364 Lyon, France (1992).Google Scholar
  23. 23.
    Raghavan, P.: Learning in threshold networks: A computational model and applications. Tech. Rep. RC 13859, IBM Res. (1988). Also in Proc. 1st Workshop on Computational Learning Theory (Cambridge, MA), ACM Press, NY (1988) 19–27.Google Scholar
  24. 24.
    Siu K-Y Roychowdhury, V., Kailath, T.: Depth-size tradeoffs for neural computations, IEEE Trans. on Comp. 40(12) (1991) 1402–1412.Google Scholar
  25. 25.
    Williamson, R.C.: ɛ-entropy and the complexity of feedforward neural networks, in R.P. Lippmann, J.E. Moody and D.S. Touretzky (eds.): Neural information processing systems (NIPS*90, Denver, CO), Morgan Kaufmann, San Mateo, CA (1991) 946–952.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1997

Authors and Affiliations

  • Valeriu Beiu
    • 1
  • Thierry de Pauw
    • 2
  1. 1.Division NIS-1Los Alamos National LaboratoryLos AlamosUSA
  2. 2.Départment de MathématiqueUniversité Catholique de LouvainLouvain-la-NeuveBelgium

Personalised recommendations