Experiments on regularizing MLP models with background knowledge

  • Arto Selonen
  • Jouko Lampinen
Part III: Learning: Theory and Algorithms
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1327)


In this contribution we present results of using possibly inaccurate knowledge of model derivatives as part of the training data for a multilayer perceptron network (MLP). Even simple constraints offer significant improvements and the resulting models give better prediction performance than traditional data driven MLP models.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Y. S. Abu-Mostafa, Hints and the VC dimension, Neural Computation, Vol. 5, No. 2, 1993, pp. 278–288.Google Scholar
  2. 2.
    C. Bishop, Curvature-Driven Smoothing: A Learning Algorithm for Feedforward Networks, IEEE Tr. on Neural Networks, Vol. 4, No. 5, Sep 1993, pp. 882–884.Google Scholar
  3. 3.
    C. Bishop, Regularization and Complexity Control in Feed-forward Networks, Proc. ICANN'95, Vol. 1., 1995, pp. 141–148.Google Scholar
  4. 4.
    C. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 1995.Google Scholar
  5. 5.
    S. Geman, E.Bienenstock and R.Doursat, Neural Networks and the Bias/Variance Dilemma, Neural Networks 4, 1992, pp. 1-58.Google Scholar
  6. 6.
    S. Haykin, Neural Networks, A Comprehensive Foundation, Macmillan, New York, NY, 1994.Google Scholar
  7. 7.
    G. Hinton, Connectionist Learning Procedures, Artificial Intelligence, 40, 1989, pp. 185–234.Google Scholar
  8. 8.
    J. Lampinen and O.Taipale, Optimization and Simulation of Quality Properties in Paper Machine with Neural Networks, Proc. of IEEE International Conference on Neural Networks, Orlando, FL, June 28–July 2, 1994, pp. 3812–3815.Google Scholar
  9. 9.
    J. Lampinen and A. Selonen, Multilayer Perceptron Training with Inaccurate Derivative Information, Proc. 1995 IEEE International Conference on Neural Networks ICNN'95, Perth, WA, Vol. 5, pp. 2811–2815, 1995.Google Scholar
  10. 10.
    Y. Le Cun, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jackel Backpropagation applied to hand written zip code recognition, Neural Computation, Vol. 1, 1989, pp. 541–551.Google Scholar
  11. 11.
    M. Riedmiller and H. Braun, A direct adaptive method for faster backpropagation learning: The RPROP algorithm, in H. Ruspini, (Ed), Proc. IEEE Internation Conf. on Neural Networks, San Francisco, 1993, pp. 586–591.Google Scholar
  12. 12.
    A. Selonen, J. Lampinen, L. Ikonen, Using External Knowledge in Neural Network Models, in Proc. SPIE on Intelligent Robots and Computer Vision XV: Algorithms, Techniques, Active Vision, and Materials Handling, Vol. 2904, Boston, MA, 1996, pp. 239–249.Google Scholar
  13. 13.
    P. Y. Simard, B. Victorri, Y. LeCun and J. Denker, Tangent prop — a formalism for specifying selected invariances in an adaptive network, In Neural Information Processing Systems, Vol. 4, San Mateo, CA, 1992.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1997

Authors and Affiliations

  • Arto Selonen
    • 1
  • Jouko Lampinen
    • 1
  1. 1.Laboratory of Computational EngineeringHelsinki University of TechnologyFinland

Personalised recommendations