Multi-Objective Machine Learning pp 151-171

Part of the Studies in Computational Intelligence book series (SCI, volume 16) | Cite as

Multi-Objective Algorithms for Neural Networks Learning

  • Antônio Pádua Braga
  • Ricardo H. C. Takahashi
  • Marcelo Azevedo Costa
  • Roselito de Albuquerque Teixeira


Most supervised learning algorithms for Artificial Neural Networks (ANN)aim at minimizing the sum of the squared error of the training data [12, 11, 5, 10]. It is well known that learning algorithms that are based only on error minimization do not guarantee good generalization performance models. In addition to the training set error, some other network-related parameters should be adapted in the learning phase in order to control generalization performance. The need for more than a single objective function paves the way for treating the supervised learning problem with multi-objective optimization techniques. Although the learning problem is multi-objective by nature, only recently it has been given a formal multi-objective optimization treatment [16]. The problem has been treated from different points of view along the last two decades.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    B. Boser, I. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. Fifth Annual Workshop on Computational Learning Theory, pages 144–152, 1992.Google Scholar
  2. [2]
    V. Chankong and Y. Y. Haimes. Multiobjective Decision Making: Theory and Methodology, volume 8. North-Holland (Elsevier), New York, 1983.Google Scholar
  3. [3]
    C. Cortes and V. Vapnik. Support vector networks. Machine Learning, 20:273–279, 1995.MATHGoogle Scholar
  4. [4]
    Yann Le Cun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems 2, pages 598–605, 1990.Google Scholar
  5. [5]
    S. E. Fahlman. Faster-learning variations on back-propagation: an empirical study. In D. Touretzky, G. Hinton, and T. Sejnowski, editors, Proceedings of the 1988 Connectionist Models Summer School, Pittsburg, pages 38–51, San Mateo, CA, 1988. Morgan Kaufmann.Google Scholar
  6. [6]
    S. R. Gunn. Support vector machines for classification and regression. Technical report, Image Speech and Intelligent Systems Research Group, University of Southampton, 1997.Google Scholar
  7. [7]
    S. Haykin. Neural Networks: A Comprehensive Foundation. Prentice Hall, 1999.Google Scholar
  8. [8]
    Ehud D. Karnin. A simple procedure for pruning back-propagation trained neural networks. IEEE Transactions on Neural Networks, 1(2):239–242, 1990.CrossRefGoogle Scholar
  9. [9]
    M. C. Mozer and P. Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. Advances in Neural Information Processing, vol. 1, pages 107–115, 1989.Google Scholar
  10. [10]
    Gustavo G. Parma, Antonio P. Braga, and Benjamim R. Menezes. Sliding mode algorithm for training multi-layer neural networks. IEE Electronics Letters, 38(1):97–98, January 1998.CrossRefGoogle Scholar
  11. [11]
    Martin Riedmiller and Heinrich Braun. A direct adaptive method for faster back propagation learning: The RPROP algorithm. In Proc. of the IEEE Intl. Conf. on Neural Networks, pages 586–591, San Francisco, CA, April 1993.Google Scholar
  12. [12]
    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back-propagating errors. Nature, 323:533–536, 1986.CrossRefGoogle Scholar
  13. [13]
    N. Z. Shor. Cut-off method with space extension in convex programming problems. Cybernetics, 12:94–96, 1977.Google Scholar
  14. [14]
    R. H. C. Takahashi, P. L. D. Peres, and P. A. V. Ferreira. H2/h-infinity multiobjective pid design. IEEE Control Systems Magazine, 17(5):37–47, June 1997.CrossRefGoogle Scholar
  15. [15]
    V. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, 1995.Google Scholar
  16. [16]
    R. A. Teixeira, A. P. Braga, R. H. C. Takahashi, and R. R. Saldanha. Improving generalization of mlps with multi-objective optimization. Neurocomputing, 35(1–4):189–194, 2000.MATHCrossRefGoogle Scholar
  17. [17]
    G. A. Hinton. Connectionist learning procedures. Artificial Intelligence, 40:185–234, 1989.CrossRefGoogle Scholar
  18. [18]
    V. Pareto. Cours D'Economie Politique. Rouse, Lausanne, 1896. vols. I and II.Google Scholar
  19. [19]
    R. A. Teixeira, A. P. Braga, R. H. C. Takahashi, and R. R. Saldanha. Utilização de seção áurea no cálculo de soluçõ efficientes para treinamento de redes neurais artificiais através de otimização multi-objetivo. 8th Brazilian Symposium on Neural Networks, November 2004.Google Scholar
  20. [20]
    U. Itkis. Control Systems of Variable Structure. Keter Publishing House Jerusalem LTD, 1976.Google Scholar
  21. [21]
    M. A. Costa, A. P. Braga, B. R. de Menezes, G. G. Parma, and R. A. Teixeira. Training neural networks with a multi-objective sliding mode control algorithm. Neurocomputing, 51:467–473, 2003.CrossRefGoogle Scholar
  22. [22]
    M. A. Costa, A. P. Braga and B. R. de Menezes. Improving neural networks generalization with new constructive and pruning methods. Journal of Intelligent & Fuzzy Systems, 10:1–9, 2003.Google Scholar
  23. [23]
    C.L. Blake and C.J. Merz. {UCI} Repository of machine learning databases. University of California, Irvine, Dept. of Information and Computer Sciences, html, 1998.Google Scholar
  24. [24]
    S. E. Fahlman and C. Lebiere, The cascade-correlation learning architecture, Morgan Kaufmann, In Advances in Neural Information Processing Systems 2 (D. S. Touretzky, Editor), 1990Google Scholar
  25. [25]
    Jean-Pierre Nadal, Study of a growth algorithm for a feedforward network, International Journal of Neural Systems, 1(1):55–59, 1989.CrossRefMathSciNetGoogle Scholar
  26. [26]
    Ron Kohavi, A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection,, 1995.Google Scholar
  27. [27]
    S. Geman and E. Bienenstock and R. Doursat. Neural Networks and the Bias/Variance Dilemma, Neural Computation, 4(1):1–58, 1992.Google Scholar

Copyright information

© Springer 2006

Authors and Affiliations

  • Antônio Pádua Braga
    • 1
  • Ricardo H. C. Takahashi
    • 1
  • Marcelo Azevedo Costa
    • 1
  • Roselito de Albuquerque Teixeira
    • 2
  1. 1.Federal University of Minas GeraisBrazil
  2. 2.Eastern University Centre of Minas GeraisBrazil

Personalised recommendations