Experiments on regularizing MLP models with background knowledge
In this contribution we present results of using possibly inaccurate knowledge of model derivatives as part of the training data for a multilayer perceptron network (MLP). Even simple constraints offer significant improvements and the resulting models give better prediction performance than traditional data driven MLP models.
Unable to display preview. Download preview PDF.
- 1.Y. S. Abu-Mostafa, Hints and the VC dimension, Neural Computation, Vol. 5, No. 2, 1993, pp. 278–288.Google Scholar
- 2.C. Bishop, Curvature-Driven Smoothing: A Learning Algorithm for Feedforward Networks, IEEE Tr. on Neural Networks, Vol. 4, No. 5, Sep 1993, pp. 882–884.Google Scholar
- 3.C. Bishop, Regularization and Complexity Control in Feed-forward Networks, Proc. ICANN'95, Vol. 1., 1995, pp. 141–148.Google Scholar
- 4.C. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.Google Scholar
- 5.S. Geman, E.Bienenstock and R.Doursat, Neural Networks and the Bias/Variance Dilemma, Neural Networks 4, 1992, pp. 1-58.Google Scholar
- 6.S. Haykin, Neural Networks, A Comprehensive Foundation, Macmillan, New York, NY, 1994.Google Scholar
- 7.G. Hinton, Connectionist Learning Procedures, Artificial Intelligence, 40, 1989, pp. 185–234.Google Scholar
- 8.J. Lampinen and O.Taipale, Optimization and Simulation of Quality Properties in Paper Machine with Neural Networks, Proc. of IEEE International Conference on Neural Networks, Orlando, FL, June 28–July 2, 1994, pp. 3812–3815.Google Scholar
- 9.J. Lampinen and A. Selonen, Multilayer Perceptron Training with Inaccurate Derivative Information, Proc. 1995 IEEE International Conference on Neural Networks ICNN'95, Perth, WA, Vol. 5, pp. 2811–2815, 1995.Google Scholar
- 10.Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel Backpropagation applied to hand written zip code recognition, Neural Computation, Vol. 1, 1989, pp. 541–551.Google Scholar
- 11.M. Riedmiller and H. Braun, A direct adaptive method for faster backpropagation learning: The RPROP algorithm, in H. Ruspini, (Ed), Proc. IEEE Internation Conf. on Neural Networks, San Francisco, 1993, pp. 586–591.Google Scholar
- 12.A. Selonen, J. Lampinen, L. Ikonen, Using External Knowledge in Neural Network Models, in Proc. SPIE on Intelligent Robots and Computer Vision XV: Algorithms, Techniques, Active Vision, and Materials Handling, Vol. 2904, Boston, MA, 1996, pp. 239–249.Google Scholar
- 13.P. Y. Simard, B. Victorri, Y. LeCun and J. Denker, Tangent prop — a formalism for specifying selected invariances in an adaptive network, In Neural Information Processing Systems, Vol. 4, San Mateo, CA, 1992.Google Scholar