Optimizing neural network architectures using generalization error estimators

Abstract

This paper addresses the optimization of neural network architectures. It is suggested to optimize the architecture by selecting the model with minimal estimated averaged generalization error. We consider a least-squares (LS) criterion for estimating neural network models, i.e., the associated model weights are estimated by minimizing the LS criterion. The quality of a particular estimated model is measured by the average generalization error. This is defined as the expected squared prediction error on a novel input-output sample averaged over all possible training sets. An essential part of the suggested architecture optimization scheme is to calculate an estimate of the average generalization error. We suggest using the GEN-estimator [9, 10] which allows for dealing with nonlinear, incomplete models, i.e., models which are not capable of modeling the underlying nonlinear relationship perfectly. In most neural network applications, it is impossible to suggest a perfect model, and consequently the ability to handle incomplete models is urgent. A concise derivation of the GEN-estimator is provided, and its qualities are demonstrated by comparative numerical studies.