Abstract
A novel approach to estimate generalisation errors of the simple perceptron of the worst case is introduced. It is well known that the generalisation error of the simple perceptron is of the form d/t with an unknown constant d which depends only on the dimension of inputs, where t is the number of learned examples. Based upon extreme value theory in statistics we obtain an exact form of the generalisation error of the simple perceptron. The method introduced in this paper opens up new possibilities to consider generalisation errors of a class of neural networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Albeverio, S., Feng, J., and Qian, M. (1995), The role of noises in neural networks, Phys. Rev. E., 52, 6593–6606.
Amari, S., Murata, N., and Ikeda, K. (1995), Statistical theory of learning curves, in: Oh, J., Kwon, Ch., and Chao, S. (eds), Neural Networks: The Statistical Mechanics Perspective, 3–17.
Baum, E.B. (1990), The perceptron algorithm is fast for nonmalicious distribution, Neural computation, 2, 248.
Baum, E.B., and Haussler, D. (1989), What size net gives valid generalization, Neural computation, 4, 151–160.
Cohn, D., and Tesauro, G. (1992), How tight are the Vapnik-Chervonenkis bounds, Neural Computation, 4, 249–269.
Engel, A., and den Broeck, C.V. (1993), Statistical mechanics calculation of Vapnik Chervonenkis bounds for perceptrons, J. Phys, 26 6893–6914.
Feng, J. (1997), Behaviours of spike output jitter in the integrate-and-fire model. Phys. Rev. Letters (in press).
Feng, J. (1997), Lyapunov functions for neural nets with nondifferentiable inputoutput characterstics, Neural Computation, 9, 45–51.
Feng, J. (1997), Generalisation error of the simple perceptron, (preprint).
Feng, J., and Hadeler, K. P. (1996), Qualitative behaviors of some simple neural networks, J. Phys. A, 29, 5019–5033.
Feng, J., Pan, H., and Roychowdhury, V. P. (1996), On neurodynamics with limiter function and Linsker’s developmental model, Neural Computation, 8, 1003–1019.
Feng, J., and Tirozzi, B. (1995), The SLLN for the free-energy of the Hopfield and spin glass model, Helvetica Physica Acta, 68, 365–379.
Galambos, J. (1984), Introductory Probability Theory, Marcek Dekker, INC., New York, 164–168.
Gray, M.S., Lawrence, D.T., Golomb, B.A., and Sejnowski, T.J. (1995), A perceptron reveals the face of sex, Neural Computation 7, 1160–1164.
Haussler, D., Kearns, M., and Shapire, R. (1991), Bounds on the sample complexity of Bayesian learning using information theory and the VC dimension, Proc. 4th Ann. Workshop on computational Learning Theory, Morgan Kaufmann, San Mateo, CA, 61–74.
Haussler, D., Littlestone, N., and Warmuth, K. (1988), Predicting {0, 1} functions on randomly drawn points, Proc. COLT’88, Morgan Kaufmann, San mateo, CA, 280–295.
Leadbetter, M.R., Lindgren, G., and Rootzén, H. (1983), Extremes and Related Properties of Random Sequences and Processes, Springer-Verlag, New York, Heidelberg, Berlin.
Levin, E., Tishby, N., and Solla, S.A. (1990), A statistical approach to learning and generalization in layered neural networks, Proceeding of the IEEE, 78(10), 1568–1574.
Murata, N., Yoshizawa, S., and Amari, S. (1994), Network information criteriondeterminate the number of hidden units for an artificial neural network model, IEEE Trans. NN, 6, 865–872.
Newman, C., and Stein, D.L. (1996), Non-mean-field behavior of realistic spin glass, Physical Review Letter 76(3), 515–518.
Opper, M., and Haussler, D. (1991), Calculation of the learning curve of Bayes optimal classification algorithm for learning perceptron with noise, Proceedings of the Fourth Annual Workshop on Computer Learning Theory, 75–87.
Opper, M., and Haussler, D. (1995), Bounds for predictive errors in the statistical mechanics of supervised learning, Physical Review Letter 75, 3772–3775.
Seung, H.S., Sompolinsky, H., and Tishbby, N. (1992), Statistical mechanics of learning from examples, Physical Review A, 45, 6056–6091.
Vapnik, V.N., and Chervonenkis, A.Y. (1971), On the uniform convergence of relative frequencies of events to their probabilities, Theory of Probab. and its Appl. 16(2), 264–280.
Vapnik, E., Levin, E., and LeCun, Y. (1994), Measuring the VC dimension of a learning machine, Neural Computation, 5, 851–876.
Watkin, T.L.H., Rau, A., and Biehl, M. (1993), The statistical mechanics of learning a rule, Rev. Mod. Phys., 65, 499–556.
Yamanishi, K. (1991), A loss bound model for on-line stochastic prediction strategies, Proceeding of the Fourth Annual Workshop on Computer Learning Theory, 290–302.
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 1999 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Feng, J. (1999). Estimating exact form of generalisation errors. In: Mira, J., Sánchez-Andrés, J.V. (eds) Foundations and Tools for Neural Modeling. IWANN 1999. Lecture Notes in Computer Science, vol 1606. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0098198
Download citation
DOI: https://doi.org/10.1007/BFb0098198
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-66069-9
Online ISBN: 978-3-540-48771-5
eBook Packages: Springer Book Archive