Accelerating the Convergence of EM-Based Training Algorithms for RBF Networks

Abstract

In this paper, we propose a new Expectation-Maximization (EM) algorithm which speeds up the training of feedforward networks with local activation functions such as the Radial Basis Function (RBF) network. The core of the conventional EM algorithm for supervised learning of feedforward networks consists of decomposing the observations into their individual output units and then estimating the parameters of each unit separately. In previously proposed approaches, at eac h E-step the residual is decomposed equally among the units or proportionally to the weights of the output layer. However, this approach tends to slow down the training of networks with local activation units. To overcome this drawback in this paper we use a new E-step which applies a soft decomposition of the residual among the units. Inparticular, the residual is decomposed according to the probability of each RBF unit given each input-output pattern. It is shown that this variant not only speeds up the training in comparison with other EM-type algorithms, but also provides better results than a global gradient-descent technique since it has the capability of avoiding some unwanted minima of the cost function.

This work has been supported by the European Community and the Spanish Government through FEDER project 1FD97-1863-C02-01.