Data Dependent Hyperparameter Assignment

Abstract

We show that in supervised learning from a particular data set Bayesian model selection, based on the evidence, does not optimise generalization performance even for a learnable linear problem. This is achieved by examining the finite size effects in hyperparameter assignment from the evidence procedure and its effect on generalisation. Using simulations we corroborate our analytic results and examine an alternative model selection criterion, namely cross-validation. This numerical study shows that in the learnable linear case for finite sized systems leave one out cross-validation estimates correlate more strongly with optimal performance than do those of the evidence.