Evolutionary Multiobjective Neural Network Models Identification: Evolving Task-Optimised Models

  • Pedro M. Ferreira
  • António E. Ruano

Abstract

In the system identification context, neural networks are black-box models, meaning that both their parameters and structure need to be determined from data. Their identification is often done iteratively in an ad-hoc fashion focusing the first aspect. Frequently the selection of inputs, model structure, and model order are underlooked subjects by practitioners, because the number of possibilities is commonly huge, thus leaving the designer at the hands of the curse of dimensionality. Moreover, the design criteria may include multiple conflicting objectives, which gives to the model identification problem a multiobjective combinatorial optimisation character. Evolutionary multiobjective optimisation algorithms are particularly well suited to address this problem because they can evolve optimised model structures that meet pre-specified design criteria in acceptable computing time. In this article the subject is reviewed, the authors present their approach to the problem in the context of identifying neural network models for time-series prediction and for classification purposes, and two application case studies are described, one in each of these fields.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amari, S., Murata, N., Müller, K.R., Finke, M., Yang, H.: Asymptotic statistical theory of overtraining and cross-validation. IEEE Transactions on Neural Networks 8(5), 985–996 (1997), doi:10.1109/72.623200CrossRefGoogle Scholar
  2. 2.
    Bauer, M., Buchtala, O., Horeis, T., Kern, R., Sick, B., Wagner, R.: Technical data mining with evolutionary radial basis function classifiers. Applied Soft Computing 9, 765–774 (2009), doi:10.1016/j.asoc.2008.07.007CrossRefGoogle Scholar
  3. 3.
    Billings, S.A., Zheng, G.L.: Radial basis function network configuration using genetic algorithms. Neural Networks 8(6), 877–890 (1995)CrossRefGoogle Scholar
  4. 4.
    Branke, J.: Evolutionary algorithms for neural network design and training. Tech. Rep. 322, University of Karlsruhe, Institute AIFB, Karlsruhe, Germany (1995)Google Scholar
  5. 5.
    Carse, B., Pipe, A.G., Fogarty, T.C., Hill, T.: Evolving radial basis function neural networks using a genetic algorithm. In: IEEE International Conference on Evolutionary Computation, vol. 1, pp. 300–305 (1995), doi:10.1109/ICEC.1995.489163Google Scholar
  6. 6.
    Chen, S., Wu, Y., Luk, B.: Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks. IEEE Transactions on Neural Networks 10(5), 1239–1243 (1999)CrossRefGoogle Scholar
  7. 7.
    Coello Coello, C.: Recent trends in evolutionary multiobjective optimization. In: Jain, L., Wu, X., Abraham, A., Jain, L., Goldberg, R. (eds.) Evolutionary Multiobjective Optimization, Advanced Information and Knowledge Processing, pp. 7–32. Springer, Heidelberg (2005), http://dx.doi.org/10.1007/1-84628-137-7_2, doi:10.1007/1-84628-137-7-2CrossRefGoogle Scholar
  8. 8.
    Coello Coello, C.: Evolutionary multi-objective optimization: a historical view of the field. IEEE Computational Intelligence Magazine 1(1), 28–36 (2006), http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1597059&tag=1, doi:10.1109/MCI.2006.1597059CrossRefMathSciNetGoogle Scholar
  9. 9.
    Crispim, E.M., Ferreira, P.M., Ruano, A.E.: Prediction of the solar radiation evolution using computational intelligence techniques and cloudiness indices. International Journal of Innovative Computing, Information and Control 4(5), 1121–1133 (2008)Google Scholar
  10. 10.
    Deb, K.: Multi-Objective Optimization using Evolutionary Algorithms. John Wiley & Sons., Ltd, Chichester (2001)MATHGoogle Scholar
  11. 11.
    Ferreira, P.M., Faria, E., Ruano, A.E.: Neural network models in greenhouse air temperature prediction. Neurocomputing 43(1-4), 51–75 (2002)CrossRefMATHGoogle Scholar
  12. 12.
    Ferreira, P.M., Martins, I.A., Ruano, A.E.: Cloud and clear sky pixel classification in ground-based all-sky hemispherical digital images. In: Ferreira, P.M. (ed.) Proceedings of CMTEE 2010, the IFAC Conference on Control Methodologies and Technology for Energy Efficiency. International Federation of Automatic Control, Vilamoura, Portugal (2010)Google Scholar
  13. 13.
    Ferreira, P.M., Ruano, A.E.: Exploiting the separability of linear and non-linear parameters in radial basis function neural networks. In: IEEE Symposium 2000: Adaptive Systems for Signal Processing, Communications, and Control, Canada, pp. 321–326 (2000), http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=882493, doi:10.1109/ASSPCC.2000.882493
  14. 14.
    Ferreira, P.M., Ruano, A.E.: Application of computational intelligence methods to greenhouse environmental modelling. In: (WCCI 2008) IJCNN 2008 (IEEE World Congress on Computational Intelligence). 2008 IEEE International Joint Conference on Neural Networks, pp. 3582–3589 (2008), doi:10.1109/IJCNN.2008.4634310Google Scholar
  15. 15.
    Ferreira, P.M., Ruano, A.E.: On-line sliding-window methods for process model adaptation. IEEE Transactions on Instrumentation and Measurement 58(9), 3012–3020 (2009), doi:10.1109/tim.2009.2016818CrossRefGoogle Scholar
  16. 16.
    Ferreira, P.M., Ruano, A.E., Fonseca, C.: Genetic assisted selection of rbf model structures for greenhouse inside air temperature prediction. In: IEEE Conference on Control Applications, Turkey, pp. 576–581 (2003)Google Scholar
  17. 17.
    Ferreira, P.M., Ruano, A.E., Fonseca, C.: Evolutionary multi-objective design of radial basis function networks for greenhouse environmental control. In: IFAC World Congress on Automatic Control 16th, Czech Republic (2005)Google Scholar
  18. 18.
    Ferreira, P.M., Ruano, A.E., Pestana, R.: Improving the identification of rbf predictive models to forecast the portuguese electricity consumption. In: Ferreira, P.M. (ed.) Proceedings of CMTEE 2010, the IFAC Conference on Control Methodologies and Technology for Energy Efficiency. International Federation of Automatic Control, Vilamoura, Portugal (2010)Google Scholar
  19. 19.
    Ferreira, P.M., Ruano, A.E., Pestana, R., Kóczy, L.T.: Evolving rbf predictive models to forecast the portuguese electricity consumption. In: ICONS 2009: The 2nd IFAC Int. Conference on Intelligent Control Systems and Signal Processing, Istanbul, Turkey (2009)Google Scholar
  20. 20.
    Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley Interscience, Hoboken (2000)CrossRefGoogle Scholar
  21. 21.
    Fonseca, C., Fleming, P.: Non-linear system identification with multiobjective genetic algorithms. In: Proceedings of the 13 IFAC World Congress, vol. C, pp. 187–192 (1996)Google Scholar
  22. 22.
    Fonseca, C., Fleming, P.: Multiobjective optimization and multiple constraint handling with evolutionary algorithms i: A unified formulation. IEEE Transactions on Systems, Man and Cybernetics - Part A: Systems and Humans 28(1), 26–37 (1998), doi:10.1109/3468.650319CrossRefGoogle Scholar
  23. 23.
    Gill, P.E., Murray, W., Wright, M.H.: Practical Optimization. Academic Press, Inc., London (1981)MATHGoogle Scholar
  24. 24.
    Griffin, I., Fleming, P.: An overview of non-linear identification and control with neural networks. In: Ruano, A.E. (ed.) Intelligent Control Using Soft-Computing Methodologies. Control Series, pp. 89–118. IEE Publishing (2005)Google Scholar
  25. 25.
    Guillén, A., Pomares, H., González, J., Rojas, I., Valenzuela, O., Prieto, B.: Parallel multiobjective memetic rbfnns design and feature selection for function approximation problems. Neurocomputing 72(16-18), 3541–3555 (2009), doi:10.1016/j.neucom.2008.12.037CrossRefGoogle Scholar
  26. 26.
    Haykin, S.: Neural Networks: a Comprehensive Foundation, 2nd edn. Prentice Hall, Inc., Englewood Cliffs (1999)MATHGoogle Scholar
  27. 27.
    Jung, J., Reggia, J.: Evolutionary design of neural network architectures using a descriptive encoding language. IEEE Transactions on Evolutionary Computation 10(6), 676–688 (2006), doi:10.1109/TEVC.2006.872346CrossRefGoogle Scholar
  28. 28.
    Kaylani, A., Georgiopoulos, M., Mollaghasemi, M., Anagnostopoulos, G.C., Sentelle, C., Zhong, M.: An adaptive multiobjective approach to evolving art architectures. IEEE Transactions on Neural Networks 21(4), 529–550 (2010)CrossRefGoogle Scholar
  29. 29.
    Lee, C.W., Shin, Y.C.: Growing radial basis function networks using genetic algorithm and orthogonalization. International Journal of Innovative Computing, Information and Control 5(11(A)), 3933–3948 (2009)Google Scholar
  30. 30.
    Leung, C., Lam, F.: Performance analysis for a class of iterative image thresholding algorithms. Pattern Recognition 29(9), 1523–1530 (1996)CrossRefGoogle Scholar
  31. 31.
    Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. Journal of the Society for Industrial and Applied Mathematics 11(2), 431–441 (1963)CrossRefMathSciNetMATHGoogle Scholar
  32. 32.
    McDonnell, J., Waagen, D.: Determining neural network hidden layer size using evolutionary programming. In: Proceedings of the 1993 World Congress on Neural Networks, vol. III, pp. 564–657 (1993)Google Scholar
  33. 33.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man and Cybernetics SMC-9(1), 62–66 (1979)MathSciNetGoogle Scholar
  34. 34.
    Peck, C., Dhawan, A., Meyer, C.: Genetic algorithm based input selection for a neural network function approximator with applications to ssme health monitoring. In: IEEE International Conference on Neural Networks, vol. 2, pp. 1115–1122 (1993), doi:10.1109/ICNN.1993.298714Google Scholar
  35. 35.
    Ridler, T., Calvard, S.: Picture thresholding using an iterative selection method. IEEE Transactions on Systems, Man and Cybernetics SMC-8(8), 630–632 (1978)Google Scholar
  36. 36.
    Rodríguez-Vázquez, K., Fonseca, C., Fleming, P.: Identifying the structure of nonlinear dynamic systems using multiobjective genetic programming. IEEE Transactions on Systems, Man and Cybernetics - Part A: Systems and Humans 34(4), 531–545 (2004), doi:10.1109/TSMCA.2004.826299CrossRefGoogle Scholar
  37. 37.
    Ruano, A., Crispim, E., Frazão, P.: Moga design of neural network predictors of inside temperature in public buildings. In: Balas, V., Fodor, J., Várkonyi-Kóczy, A. (eds.) Soft Computing Based Modeling in Intelligent Systems. SCI, vol. 196, pp. 35–61. Springer, Heidelberg (2009), doi:10.1007/978-3-642-00448-3-3CrossRefGoogle Scholar
  38. 38.
    Ruano, A., Fleming, P., Jones, D.: Connectionist approach to pid autotuning. IEE Proceedings (part D), 139(3), 279–285 (1992), http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=141517CrossRefGoogle Scholar
  39. 39.
    Ruano, A., Jones, D., Fleming, P.: A new formulation of the learning problem of a neural network controller. In: Proceedings of the 30th IEEE Conference on Decision and Control, vol. 1, pp. 865–866 (1991), http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=261439, doi:10.1109/CDC.1991.261439
  40. 40.
    Ruano, A.E., Ferreira, P.M., Cabrita, C., Matos, S.: Training neural networks and neuro-fuzzy systems: A unified view. In: Proceedings of the 15th IFAC World Congress, vol. 15 (2002)Google Scholar
  41. 41.
    Ruano, A.E., Ferreira, P.M., Fonseca, C.: An overview of non-linear identification and control with neural networks. In: Ruano, A.E. (ed.) Intelligent Control Using Soft-Computing Methodologies. Control Series, pp. 37–87. IEE Publishing (2005)Google Scholar
  42. 42.
    Ruano, A.E., Ferreira, P.M., Mendes, H.: Moga design of temperature and relative humidity models for predictive thermal comfort. In: Ferreira, P.M. (ed.) Proceedings of CMTEE 2010, the IFAC Conference on Control Methodologies and Technology for Energy Efficiency. International Federation of Automatic Control, Vilamoura, Portugal (2010)Google Scholar
  43. 43.
    Ruano, A.E., Fleming, P., Teixeira, C., Rodríguez-Vázquez, K., Fonseca, C.: Nonlinear identification of aircraft gas-turbine dynamics. Neurocomputing 55(3-4), 551–579 (2003), doi:10.1016/S0925-2312(03)00393-XCrossRefGoogle Scholar
  44. 44.
    Sezgin, M., Sankur, B.: Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging 13(1), 146–165 (2004)CrossRefGoogle Scholar
  45. 45.
    Sjöberg, J., Ljung, L.: Overtraining, regularization, and searching for minimum with application to neural networks. In: Preprint IFAC Symposium on Adaptive Systems in Control and Signal Processing, pp. 669–674 (1994)Google Scholar
  46. 46.
    Trussel, H.: Comments on picture thresholding using an iterative selection method. IEEE Transactions on Systems, Man and Cybernetics SMC-9(5), 311 (1979)CrossRefGoogle Scholar
  47. 47.
    Yen, G.: Multi-objective evolutionary algorithm for radial basis function neural network design. In: Jin, Y. (ed.) Multi-objective machine learning. SCI, vol. 16, pp. 221–239. Springer, Heidelberg (2006), doi:10.1007/3-540-33019-4-10CrossRefGoogle Scholar
  48. 48.
    Zitzler, E., Laumanns, M., Bleuler, S.: A tutorial on evolutionary multiobjective optimization. In: Gandibleux, X., et al. (eds.) Metaheuristics for Multiobjective Optimisation. Lecture Notes in Economics and Mathematical Systems, vol. 535, pp. 3–37. Springer, Heidelberg (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Pedro M. Ferreira
    • 1
    • 2
  • António E. Ruano
    • 2
  1. 1.Algarve Science & Technology ParkUniversity of AlgarveFaroPortugal
  2. 2.Centre for Intelligent SystemsUniversity of Algarve - FCTFaroPortugal

Personalised recommendations