Advances in Computational Mathematics

, Volume 5, Issue 1, pp 269–280 | Cite as

Linear unlearning for cross-validation

  • Lars Kai Hansen
  • Jan Larsen


The leave-one-out cross-validation scheme for generalization assessment of neural network models is computationally expensive due to replicated training sessions. In this paper we suggest linear unlearning of examples as an approach to approximative cross-validation. Further, we discuss the possibility of exploiting the ensemble of networks offered by leave-one-out for performing ensemble predictions. We show that the generalization performance of the equally weighted ensemble predictor is identical to that of the network trained on the whole training set.

Numerical experiments on the sunspot time series prediction benchmark demonstrate the potential of the linear unlearning technique.


Neural Network Time Series Numerical Experiment Network Model Training Session 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    H. Akaike, Fitting autoregressive models for prediction, Annals of the Institute of Statistical Mathematics 21, 1969, 243–247.Google Scholar
  2. [2]
    W.L. Buntine and A.S. Weigend, Bayesian back-propagation, Complex Systems 5, 1991, 603–643.Google Scholar
  3. [3]
    T. Fox, D. Hinkley and K. Larntz, Jackknifing in nonlinear regression, Technometrics 22, 1980, 29–33.Google Scholar
  4. [4]
    A. Krogh and J. Vedelsby, Neural network ensembles, cross validation, and active learning, in:Advances in Neural Information Processing Systems 7, eds. G. Tesauro et al., MIT Press, Cambridge, Massachusetts, 1995.Google Scholar
  5. [5]
    J. Larsen and L.K. Hansen, Generalization performance of regularized neural network models, in:Proceedings of the IEEE Workshop on Neural Networks for Signal Processing IV, eds. J. Vlontzos, J.-N. Hwang and E. Wilson, IEEE, Piscataway, NJ, 1994, pp. 42–51.Google Scholar
  6. [6]
    J. Larsen and L.K. Hansen, Empirical generalization assessment of neural network models, in:Proceedings of the IEEE Workshop on Neural Networks for Signal Processing V, eds. F. Girosi, J. Makhoul, E. Manolakos and E. Wilson, IEEE, Piscataway, NJ, 1995, pp. 30–39.Google Scholar
  7. [7]
    L. Ljung,System Identification: Theory for the User, Prentice-Hall, Englewood Cliffs, NJ, 1987.Google Scholar
  8. [8]
    J. Moody, Note on generalization, regularization, and architecture selection in nonlinear learning systems, in:Proceedings of the 1st IEEE Workshop on Neural Networks for Signal Processing, eds. B.H. Juang, S.Y. Kung and C.A. Kamm, IEEE, Piscataway, NJ, 1991, pp. 1–10.Google Scholar
  9. [9]
    J. Moody, Prediction risk and architecture selection for neural networks, in:From Statistics to Neural Networks: Theory and Pattern Recognition Applications, eds. V. Cherkassky, J.H. Friedman and H. Wechsler, Series F, vol. 136, Springer, Berlin, 1994.Google Scholar
  10. [10]
    N. Murata, S. Yoshizawaand and S. Amari, Network information criterion — determining the number of hidden units for an artificial neural network model, IEEE Transactions on Neural Networks, 5, 1994, 865–872.CrossRefGoogle Scholar
  11. [11]
    G.A.F. Seber and C.J. Wild,Nonlinear Regression, Wiley, New York, 1989.Google Scholar
  12. [12]
    M. Stone, Cross-validatory choice and assessment of statistical predictors, Journal of the Royal Statistical Society B 36(2), 1974, 111–147.Google Scholar
  13. [13]
    G.T. Toussaint, Bibliography on estimation of misclassification, IEEE Transactions on Information Theory 20(4), 1974, 472–479.CrossRefGoogle Scholar
  14. [14]
    G. Wahba, Spline models for observational data,CBMS-NSF Regional Conference Series in Applied Mathematics, vol. 59, SIAM, 1990.Google Scholar
  15. [15]
    A.S. Weigend, B.A. Hubermann and D.E. Rumelhart, Predicting the future: a connectionist approach, International Journal of Neural Systems 1(3), 1990, 193–209.CrossRefGoogle Scholar

Copyright information

© J.C. Baltzer AG, Science Publishers 1996

Authors and Affiliations

  • Lars Kai Hansen
    • 1
  • Jan Larsen
    • 1
  1. 1.connect, Electronics Institute B349Technical University of DenmarkLyngbyDenmark

Personalised recommendations