Hypothesising Neural Nets

Abstract

This chapter introduces a technique for empirically testing feed-forward Neural Network architectures. The technique, Artificial Network Generation (ANG), makes possible a controlled series of experiments that statistically validates Occam’s Razor as a design methodology for network architectures in the context ofgradient descent learning algorithms. This chapter introduces a new method for network architecture pruning, Network Regression Pruning, (NRP). NRP differs radically from existing pruning algorithms in that it attempts to hold a trained network’s mapping fixed as the pruning procedure is carried out. ANG is used to analyse the pruning technique’s ability to deliver Minimally Descriptive Networks. A method is introduced that uses NRP to infer a small set of candidate architectures for a given learning problem. Finally, it is shown how NRP can be used in conjunction with a Genetic Algorithm for full Neural Network parameterisation.