Abstract
We consider the computational complexity of learning by neural nets. We are interested in how hard it is to design appropriate neural net architectures and to train neural nets for general and specialized learning tasks. Our main result shows that the training problem for 2-cascade neural nets (which have only two non-input nodes, one of which is hidden) is \(\Re P\)-complete, which implies that finding an optimal net (in terms of the number of non-input units) that is consistent with a set of examples is also \(\Re P\)-complete. This result also demonstrates a surprising gap between the computational complexities of one-node (perceptron) and two-node neural net training problems, since the perceptron training problem can be solved in polynomial time by linear programming techniques. We conjecture that training a k-cascade neural net, which is a classical threshold network training problem, is also \(\Re P\)-complete, for each fixed k ≥ 2. We also show that the problem of finding an optimal perceptron (in terms of the number of non-zero weights) consistent with a set of training examples is \(\Re P\)-hard.
Our neural net learning model encapsulates the idea of modular neural nets, which is a popular approach to overcoming the scaling problem in training neural nets. We investigate how much easier the training problem becomes if the class of concepts to be learned is known a priori and the net architecture is allowed to be sufficiently non-optimal. Finally, we classify several neural net optimization problems within the polynomial-time hierarchy.
Article PDF
Similar content being viewed by others
References
Baum, E., & Haussler, D.(1989). What size net gives valid generalization? Neural Computation, 1,151–160.
Blum, A., & Rivest, R. L.(1988). Training a 3-node neural network is 316 ℜ℘-complete. Proceedings of the First ACM Workshop on the Computational Learning Theory (pp.9–18). Cambridge, MA.
Blumer, A., Ehrenfeucht, A., Haussler, D., & Warmuth, M. K.(1989). Learnability and the Vapnik-Chervonenkis dimension. Journal of the Association for Computing Machinery, 36, 929–965.
Dertouzos, M. L.(1965).Threshold logic:A synthesis approach. Cambridge, MA: MIT Press.
Garey, M. R., & Johnson, D. S.(1979).Computers and intractability:A guide to the theory of3l9-completeness. San Francisco, CA: W. H. Freeman and Co.
Haussler, D.(1988). Quantifying inductive bias:AI learning algorithms and Valiant's learning framework. Artificial Intelligence, 36, 177–221.
Hinton, G. E.(1989). Connectionist learning procedures.Artificial Intelligence, 40, 185–234.
Judd, J. S.(1987). Complexity of connectionist learning with various node functions.(COINS Technical Report No.87–60). University of Massachusetts.
Judd, J. S.(1988). On the complexity of loading shallow neural networks. Journal of Complexity, 4, 177–192.
Masek, W. J.(1978). Some SiKP-complete set cover problems. MIT Laboratory for Computer Science, unpublished manuscript.
Rumelhart, D. E., Hinton, G. E., & Williams, R. J.(1986). Learning internal representations by error propagation. In D. E. Rumelhart & I. E. McClelland (Eds.),Parallel distributed processing. Cambridge, MA: MIT Press.
Stockmeyer, L. J.(1977). The polynomial-time hierarchy. Theoretical Computer Science, 3, 1–22.
Stockmeyer, L. J., & Meyer, A. R.(1973). Word problems requiring exponential time:Preliminary report. Proceed-ings of the fifth Annual Symposium on the Theory of Computing (pp.1–9).
Valiant, L. G.(1984). A theory of the learnable.Communications of the ACM, 27, 1134–1142.
Weibel, A.(1989). Modular construction of time-delay neural networks for speech recognition. Neural Computa tion, 1, 39–46.
Weibel, A., & Hampshire, J.(1989). Building blocks for speech. Byte, August, 235–242.
Wrathall, C.(1977). Complete sets and the polynomial-time hierarchy. Theoretical Computer Science, 3, 23–33.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Lin, JH., Vitter, J.S. Complexity Results on Learning by Neural Nets. Machine Learning 6, 211–230 (1991). https://doi.org/10.1023/A:1022657626762
Issue Date:
DOI: https://doi.org/10.1023/A:1022657626762