Estimating the Number of Hidden Neurons in a Feedforward Network Using the Singular Value Decomposition

  • Eu Jin Teoh
  • Cheng Xiang
  • Kay Chen Tan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3971)


We attempt to quantify the significance of increasing the number of neurons in the hidden layer of a feedforward neural network architecture using the singular value decomposition (SVD). Through this, we extend some well-known properties of the SVD in evaluating the generalizability of single hidden layer feedforward networks (SLFNs) with respect to the number of hidden neurons. The generalization capability of the SLFN is measured by the degree of linear independency of the patterns in hidden layer space, which can be indirectly quantified from the singular values obtained from the SVD, in a post-learning step.


Hide Layer Singular Value Decomposition Hide Neuron Feedforward Neural Network Feedforward Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Huang, S., Huang, Y.: Bounds on Number of Hidden Neurons of Multilayer Perceptrons in Classification and Recognition. IEEE International Symposium on Circuits and Systems 4, 2500–2503 (1990)CrossRefGoogle Scholar
  2. 2.
    Sartori, M., Antsaklis, P.: A Simple Method to Derive Bounds on the Size and to Train Multi-Layer Neural Networks. IEEE Trans. on Neural Networks 2(4), 467–471 (1991)CrossRefGoogle Scholar
  3. 3.
    Tamura, S.: Capabilities of a Three-Layer Feedforward Neural Network. In: Proc. Int. Joint Conf. on Neural Networks (1991)Google Scholar
  4. 4.
    Tamura, S., Tateishi, M.: Capabilities of a Four-Layered Feedforward Neural Network: Four Layers Versus Three. IEEE Trans. on Neural Networks 8(2), 251–255 (1997)CrossRefGoogle Scholar
  5. 5.
    Huang, G., Babri, H.: Upper Bounds on the Number of Hidden Neurons in Feedforward Networks with Arbitrary Bounded Nonlinear Activation Functions. IEEE Trans. Neural Networks 9(1) (1998)Google Scholar
  6. 6.
    Huang, G.: Learning Capability and Storage Capacity of Two-Hidden-Layer Feedforward Networks. IEEE Trans. on Neural Networks 14(2), 274–281 (2003)CrossRefGoogle Scholar
  7. 7.
    Cover, T.: Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition. IEEE Trans. Electronic. Comput. 14, 326–334 (1965)MATHCrossRefGoogle Scholar
  8. 8.
    Hayashi, M.: A Fast Algorithm for the Hidden Units in a Multilayer Perceptron. Proc. Int. Joint Conf. on Neural Networks 1, 339–342 (1993)Google Scholar
  9. 9.
    Tamura, S., Tateishi, M., Matsumoto, M., Akita, S.: Determination of the Number of Redundant Hidden Units in a Three-Layered Feedforward Neural Network. Proc. Int. Joint Conf. on Neural Networks 1, 335–338 (1993)Google Scholar
  10. 10.
    Psichogios, D., Ungar, L.: SVD-NET: An Algorithm that Automatically Selects Network Structure. IEEE Trans. on Neural Networks 5(3), 513–515 (1994)CrossRefGoogle Scholar
  11. 11.
    Xiang, C., Ding, S., Lee, T.: Geometrical Interpretation and Architecture Selection of MLP. IEEE Trans. Neural Networks 16(1), 84–96 (2005)CrossRefGoogle Scholar
  12. 12.
    Stewart, G.: Determining Rank in the Presence of Error. Technical Report (TR-92-108) Institute for Advanced Computer Studies (TR-2972) Department of Computer Science, University of Maryland, College Park (October 1992)Google Scholar
  13. 13.
    Golub, G.H., Van Loan, C.: Matrix Computations, 3rd edn. John Hopkins University Press, Baltimore (1999)Google Scholar
  14. 14.
    Hansen, P.: Rank-Deficient and Discrete Ill-Posed Problems: Numerical Aspects of Linear Inversion. SIAM, Philadelphia (1998)CrossRefGoogle Scholar
  15. 15.
    Golub: G., Klema, V., Stewart, G.: Rank Degeneracy and the Least Squares Problem. Technical Report (STAN-CS-76-559), Computer Science Department, School of Humanities and Sciences, Stanford University (1976)Google Scholar
  16. 16.
    Ljung, L.: System Identification: Theory for the User. Prentice-Hall, Englewood Cliffs (1987)MATHGoogle Scholar
  17. 17.
    Konstantinides, K., Yao, K.: Statistical Analysis of Effective Singular Values in Matrix Rank Determination. IEEE Trans. on Acoustics, Speech and Signal Processing 36(5), 757–763 (1988)MATHCrossRefGoogle Scholar
  18. 18.
    Moller, M.: A Scaled Conjugate Gradient Algorithm for Fast Supervised Learning. Neural Networks 6, 525–533 (1993)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Eu Jin Teoh
    • 1
  • Cheng Xiang
    • 1
  • Kay Chen Tan
    • 1
  1. 1.Department of Electrical and Computer EngineeringNational University of SingaporeSingapore

Personalised recommendations