An Efficient Generalization of Battiti-Shanno’s Quasi-Newton Algorithm for Learning in MLP-Networks

  • Carmine Di Fiore
  • Stefano Fanelli
  • Paolo Zellini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3316)

Abstract

This paper presents a novel Quasi-Newton method for the minimization of the error function of a feed-forward neural network. The method is a generalization of Battiti’s well known OSS algorithm. The aim of the proposed approach is to achieve a significant improvement both in terms of computational effort and in the capability of evaluating the global minimum of the error function. The technique described in this work is founded on the innovative concept of “convex algorithm” in order to avoid possible entrapments into local minima. Convergence results as well numerical experiences are presented.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Al Baali, M.: Improved Hessian approximations for the limited memory BFGS method. Numer. Algorithms 22, 99–112 (1999)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Battiti, R.: First- and second-order methods for learning: between steepest descent and Newton’s method. Neural Computation 4, 141–166 (1992)CrossRefGoogle Scholar
  3. 3.
    Bianchini, M., Fanelli, S., Gori, M., Protasi, M.: Non-suspiciousness: a generalization of convexity in the frame of foundations of Numerical Analysis and Learning. In: IJCNN 1998, Anchorage, vol. II, pp. 1619–1623 (1998)Google Scholar
  4. 4.
    Bianchini, M., Fanelli, S., Gori, M.: Optimal algorithms for well-conditioned nonlinear systems of equations. IEEE Transactions on Computers 50, 689–698 (2001)CrossRefMathSciNetGoogle Scholar
  5. 5.
    Bortoletti, A., Di Fiore, C., Fanelli, S., Zellini, P.: A new class of quasi-newtonian methods for optimal learning in MLP-networks. IEEE Transactions on Neural Networks 14, 263–273 (2003)CrossRefGoogle Scholar
  6. 6.
    Di Fiore, C., Fanelli, S., Zellini, P.: Matrix algebras in quasi-newtonian algorithms for optimal learning in multi-layer perceptrons. In: ICONIP Workshop and Expo, Dunedin, pp. 27–32 (1999)Google Scholar
  7. 7.
    Di Fiore, C., Fanelli, S., Zellini, P.: Optimisation strategies for nonconvex functions and applications to neural networks. In: ICONIP 2001, Shanghai, vol. 1, pp. 453–458 (2001)Google Scholar
  8. 8.
    Di Fiore, C., Fanelli, S., Zellini, P.: Computational experiences of a novel algorithm for optimal learning in MLP-networks. In: ICONIP 2002, Singapore, vol. 1, pp. 317–321 (2002)Google Scholar
  9. 9.
    Di Fiore, C., Fanelli, S., Lepore, F., Zellini, P.: Matrix algebras in Quasi-Newton methods for unconstrained optimization. Numerische Mathematik 94, 479–500 (2003)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Di Fiore, C., Fanelli, S., Zellini, P.: Convex algorithms for optimal learning in MLPnetworks. In: PreparationGoogle Scholar
  11. 11.
    Duda, R.O., Hart, P.E.: Pattern Classification and Scene Analysis. Wiley, Chichester (1973)MATHGoogle Scholar
  12. 12.
    Frasconi, P., Fanelli, S., Gori, M., Protasi, M.: Suspiciousness of loading problems. IEEE Int. Conf. on Neural Networks 2, 1240–1245 (1997)Google Scholar
  13. 13.
    Liu, D.C., Nocedal, J.: On the limited memory BFGS method for large scale optimization. Math. Programming 45, 503–528 (1989)MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Nocedal, J., Wright, S.J.: Numerical Optimization. Springer, New York (1999)MATHCrossRefGoogle Scholar
  15. 15.

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Carmine Di Fiore
    • 1
  • Stefano Fanelli
    • 1
  • Paolo Zellini
    • 1
  1. 1.Department of MathematicsUniversity of Rome “Tor Vergata”RomeItaly

Personalised recommendations