The Bayesian Evidence Scheme for Regularisation

Abstract

This chapter generalises the Bayesian evidence scheme, introduced to the neural network community by David MacKay for the regularisation of networks under the assumption of Gaussian homoscedastic noise, to the prediction of arbitrary conditional probability densities. The idea is to optimise parameters and hyperparameters seperately, and to find the mode with respect to the hyperparameters only after the parameters have been integrated out. This integration is carried out by Gaussian approximation, which requires the calculation of the Hessian of the error function at the mode. The derivation of this matrix can be accomplished with a generalised version of the EM algorithm, as exposed in detail in the appendix.