Constructing higher order neurons of increasing complexity in cascade networks
A problem faced by many constructive neural networks using a cascade architecture is the large network depth. This results in large fan-in and propagation delays, problems especially relevant for VLSI implementation of these networks. This work explores the effect of limiting the depth of the cascades created by CasPer, a constructive cascade algorithm. Instead of a single cascade of hidden neurons, a series of cascade towers are built. Each cascade tower can be viewed as a single Higher Order Neuron (HON). The optimal complexity of the HON required for a given problem is difficult to estimate, and is a form of the bias-variance dilemma. This problem is overcome via the construction of HONs with increasing complexity. It is shown that by constructing HONs in this manner the chance of overfitting is reduced, especially with noisy data.
Unable to display preview. Download preview PDF.
- 1.Treadgold, N.K. and Gedeon, T.D. “A Cascade Network Employing Progressive RPROP,” Int. Work Conf. On Artificial and Natural Neural Networks, 1997, pp. 733–742.Google Scholar
- 2.Treadgold, N.K. and Gedeon, T.D. “Extending CasPer: A Regression Survey,” Int. Conf. On Neural Information Processing, 1997, pp. 310–313.Google Scholar
- 3.Fahlman, S.E. and Lebiere, C. “The cascade-correlation learning architecture,” Advances in Neural Information Processing, vol. 2, D.S. Touretzky, (Ed.) San Mateo, CA: Morgan Kauffman, 1990, pp. 524–532.Google Scholar
- 4.Treadgold, N.K. and Gedeon, T.D. “Exploring Architecture Variations in Constructive Cascade Networks, “ Int. Joint Conference on Neural Networks, 1998, to appear.Google Scholar
- 5.Riedmiller, M. and Braun, H. “A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm,” Proc IEEE Int. Conf on Neural Networks, 1993, pp. 586–591.Google Scholar
- 6.Treadgold, N.K. and Gedeon, T.D. “A Simulated Annealing Enhancement to Resilient Backpropagation,” Proc. Int. Panel Conf. Soft and Intelligent Computing, Budapest, 1996, pp. 293–298.Google Scholar
- 7.Hwang, J., Lay, S., Maechler, R. and Martin, D. “Regression Modeling in BackPropagation and Projection Pursuit Learning,” IEEE Trans. Neural Networks 5(3), 1994, pp. 342–353.Google Scholar