Applied Intelligence

, Volume 40, Issue 4, pp 710–723 | Cite as

An analysis of accuracy-diversity trade-off for hybrid combined system with multiobjective predictor selection

  • Athanasios Tsakonas


This study examines the contribution of diversity under a multi-objective context for the promotion of learners in an evolutionary system that generates combinations of partially trained learners. The examined system uses a grammar-driven genetic programming to evolve hierarchical, multi-component combinations of multilayer perceptrons and support vector machines for regression. Two advances are studied. First, a ranking formula is developed for the selection probability of the base learners. This formula incorporates both a diversity measure and the performance of learners, and it is tried over a series of artificial and real-world problems. Results show that when the diversity of a learner is incorporated with equal weights to the learner performance in the evolutionary selection process, the system is able to provide statistically significantly better generalization. The second advance examined is a substitution phase for learners that are over-dominated, under a multi-objective Pareto domination assessment scheme. Results here show that the substitution does not improve significantly the system performance, thus the exclusion of very weak learners, is not a compelling task for the examined framework.


Ensemble systems Function approximation Genetic programming Multi-objective optimization 



The author wishes to gratefully acknowledge discussions with Professor Bogdan Gabrys throughout this research. The research leading to these results has received funding from the European Commission within the Marie Curie Industry and Academia Partnerships and Pathways (IAPP) programme under grant agreement n. 251617.


  1. 1.
    Abbass H (2003) Pareto neuro-evolution: constructing ensemble of neural networks using multi-objective optimization. In: The IEEE Congress on evolutionary computation (IEEE-CEC), vol 3. IEEE Press, Los Alamitos, pp 2074–2080 Google Scholar
  2. 2.
    Alba E, Cotta C, Troya J (1996) Evolutionary design of fuzzy logic controllers using strongly-typed gp. In: Proc 1996 IEEE int’l symposium on intell control, pp 127–132 Google Scholar
  3. 3.
    Andre D, Koza J (1996) Parallel genetic programming: a scalable implementation using the transputer network. In: Advances in genetic programming, pp 317–337 Google Scholar
  4. 4.
    Banfield R, Hall L, Bowyer K, Kegelmeyer W (2005) Ensemble diversity measures and their application to thinning. Inf Fusion 6(1):49–62 CrossRefGoogle Scholar
  5. 5.
    Barrero D, R-Moreno M, Castano B, Camacho D (2011) An empirical study on the accuracy of computational effort in genetic programming. In: 2011 IEEE Congress on evolutionary computation (CEC), pp 1164–1171 CrossRefGoogle Scholar
  6. 6.
    Basseur M, Zitzler E (2006) Handling uncertainty in indicator-based multiobjective optimization. Int J Comput Intell Res 2(3):255–272 CrossRefMathSciNetGoogle Scholar
  7. 7.
    Bentley PJ, Wakefield JP (1997) Finding acceptable solutions in the pareto-optimal range using multiobjective genetic algorithms. In: Soft computing in engineering design and manufacturing, vol 5, pp 231–240 Google Scholar
  8. 8.
    Bishop C (1995) Neural networks for pattern recognition. Oxford University Press, London Google Scholar
  9. 9.
    Brown G, Wyatt J, Harris R, Yao X (2005) Diversity creation methods: a survey and categorisation. J Inf Fusion 6:5–20 CrossRefGoogle Scholar
  10. 10.
    Budka M, Gabrys B (2010) Ridge regression ensemble for toxicity prediction. Proc Comput Sci 1(1):193–201 CrossRefGoogle Scholar
  11. 11.
    Chandra A, Yao X (2006) Ensemble learning using multi-objective evolutionary algorithms. J Math Model Algorithms 5(4):417–445 zbMATHMathSciNetGoogle Scholar
  12. 12.
    Christensen S, Oppacher F (2002) An analysis of Koza’s computational effort statistic for genetic programming. In: Proc EuroGP ’02, 5th European conference on genetic programming. Springer, Berlin, pp 182–191 Google Scholar
  13. 13.
    Clemen R (1989) Combining forecasts: a review and annotated bibliography. Int J Forecast 5(4):559–583 CrossRefGoogle Scholar
  14. 14.
    Coelho A, Fernandes E, Faceli K (2011) Multi-objective design of hierarchical consensus functions for clustering ensembles via genetic programming. Decis Support Syst 51(4):794–809 CrossRefGoogle Scholar
  15. 15.
    Colobert R, Bengio S (2001) Svmtorch: support vector machines for large- scale regression problems. J Mach Learn Res 1:143–160 MathSciNetGoogle Scholar
  16. 16.
    Detrano R, Janosi A, Steinbrunn W, Pfisterer M, Schmid J, Sandhu S, Guppy K, Lee S, Froelicher V (1989) International application of a new probability algorithm for the diagnosis of coronary artery disease. Am J Cardiol 64:304–310 CrossRefGoogle Scholar
  17. 17.
    Dietterich T (2000) Ensemble methods in machine learning. Multiple Classif Syst 1557:1–15 CrossRefGoogle Scholar
  18. 18.
    Drechsler N, Drechsler R, Becker B (2001) Multi-objective optimisation based on relation favour. In: 1st int’l conf on evol multi-criterion optimization. Springer, Berlin, pp 105–145 Google Scholar
  19. 19.
    Falco I, Tarantino E, Cioppa A, Gagliardi F (2005) A novel grammar-based genetic programming approach to clustering. In: Proc 2005 ACM symp on appl comp, Santa Fe, New Mexico, pp 928–932 CrossRefGoogle Scholar
  20. 20.
    Fernandez F, Tommassini M, Vanneschi L (2003) An empirical study of multipopulation genetic programming. Genet Program Evol Mach 4(1) Google Scholar
  21. 21.
    Folino G, Pizzuti C, Spezzano G Ensemble techniques for parallel genetic programming based classifiers Google Scholar
  22. 22.
    Folino G, Pizzuti C, Spezzano G, Vanneschi L, Tommassini M (2003) Diversity analysis in cellular and multipopulation genetic programming. In: Proc of the 2003 Congress on evolutionary computation, CEC2003 Google Scholar
  23. 23.
    Fonseca C, Fleming P (1993) Genetic algorithms for multiobjective optimization: formulation, discussion and generalization. In: 5th international conference genetic algorithms, pp 416–423 Google Scholar
  24. 24.
    Friedman J (1991) Multivariate adaptive regression splines. Ann Stat 19:1–141 CrossRefzbMATHGoogle Scholar
  25. 25.
    Gabrys B (2004) Learning hybrid neuro-fuzzy classifier models from data: to combine or not to combine? Fuzzy Sets Syst 147:39–56 CrossRefzbMATHMathSciNetGoogle Scholar
  26. 26.
    Gacto MJ, Alcala R, Herrera F (2012) A multi-objective evolutionary algorithm for an effective tuning of fuzzy logic controllers in heating, ventilating and air conditioning systems. Appl Intell 36(2):330–347 CrossRefGoogle Scholar
  27. 27.
    Hong JH, Cho SB (2006) The classification of cancer based on DNA microarray data that uses diverse ensemble genetic programming. Artif Intell Med 36(1):43–58 CrossRefGoogle Scholar
  28. 28.
    Jacobs R (1997) Bias-variance analyses of mixture-of-experts architectures. Neural Comput 9:369–383 CrossRefzbMATHGoogle Scholar
  29. 29.
    Jang JS (1993) Anfis: Adaptive-network-based fuzzy inference system. IEEE Trans Syst Man Cyber 23(3) Google Scholar
  30. 30.
    Jordan MI, Jacobs RA (1993) Hierarchical mixtures of experts and the em algorithm. In: IJCNN ’93—proc of 1993 int’l joint conf on neural networks, vol 2, pp 1339–1344 Google Scholar
  31. 31.
    Kibler D, Aha D, Albert MK (1989) Instance-based prediction of real-valued attributes. Comput Intell 5(2):51–57 CrossRefGoogle Scholar
  32. 32.
    Kohavi R, Wolpert D (1996) Bias plus variance decomposition for zero-one loss functions. In: Machine learning: proc of the 13th int’l conf Google Scholar
  33. 33.
    Koza J (1992) Genetic programming—on the programming of computers by means of natural selection. MIT Press, Cambridge zbMATHGoogle Scholar
  34. 34.
    Kumar R, Rockett P (2002) Improved sampling of the Pareto-front in multiobjective genetic optimizations by steady-state evolution: a Pareto converging genetic algorithm. Evol Comput 10(3):283–314 CrossRefGoogle Scholar
  35. 35.
    Kuncheva L, Whitaker C (2003) Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach Learn 51:181–207 CrossRefzbMATHGoogle Scholar
  36. 36.
    Laumanns M, Laumanns N (2005) Evolutionary multiobjective design in automotive development. Appl Intell 23(1):55–70 CrossRefGoogle Scholar
  37. 37.
    Lee H, Kim E, Pedrycz W (2012) A new selective neural network ensemble with negative correlation. Appl Intell 37(4):488–498 CrossRefGoogle Scholar
  38. 38.
    Liu Y, Yao X (1999) Ensemble learning via negative correlation. Neural Netw 12(10):82–102 CrossRefGoogle Scholar
  39. 39.
    Manderick B, Spiessens P (1989) Fine-grained parallel genetic algorithms. In: Proc of the 3rd int’l conf on genetic algorithms Google Scholar
  40. 40.
    Montana D (1995) Strongly typed genetic programming. Evol Comput 3(2) Google Scholar
  41. 41.
    Niehaus J, Banzhaf W (2003) More on computational effort statistics for genetic programming. In: Genetic programming. Lecture notes in computer science, vol 2610, pp 164–172 CrossRefGoogle Scholar
  42. 42.
    Opitz D, Shavlik J (1996) Generating acuurate and diverse members of a neural-network ensemble. In: Adv in Neural Information Processing Systems, vol 8. MIT Press, Cambridge Google Scholar
  43. 43.
    Perrone MP, Cooper LN (1993) When networks disagree: ensemble methods for hybrid neural networks. In: Neural networks for speech and image processing. Chapman-Hall, London, pp 126–142 Google Scholar
  44. 44.
    Pierro F, Khu ST, Savic D (2007) An investigation on preference order ranking scheme for multiobjective evolutionary optimization. IEEE Trans Evol Comput 11(1):17–45 CrossRefGoogle Scholar
  45. 45.
    Prechelt L (1994) PROBEN1—a set of neural network benchmark problems and benchmarking rules. Technical report, University of Freiburg Google Scholar
  46. 46.
    Ruta D, Gabrys B (2002) New measure of classifier dependency in multiple classifier systems. In: Proc of the MCS’2002 conf Google Scholar
  47. 47.
    Ruta D, Gabrys B (2005) Classifier selection for majority voting. Inf Fusion 6(1):63–81 CrossRefGoogle Scholar
  48. 48.
    Srinivas N, Deb K (1994) Multiobjective optimization using nondominated sorting in genetic algorithms. Evol Comput 2(3):221–248 CrossRefGoogle Scholar
  49. 49.
    Stronger D, Stone P (2008) Polynomial regression with automated degree: A function approximator for autonomous agents. Art Intell Tools 17(1) Google Scholar
  50. 50.
    Tanese R (1989) Distributed genetic algorithms. In: Proc of the 3rd int’l conf on genetic algorithms. Morgan Kaufmann, San Mateo, pp 434–439 Google Scholar
  51. 51.
    Tsakonas A, Gabrys B (2012) Gradient: grammar-driven genetic programming framework for building multi-component, hierarchical predictive systems. Expert Syst Appl 39:13,253–13,266 CrossRefGoogle Scholar
  52. 52.
    Tumer K, Ghosh J (1996) Error correlation and error reduction in ensemble classifiers. Connect Sci 8(3):385–404 CrossRefGoogle Scholar
  53. 53.
    Yeh I-C (2008) Modeling slump of concrete with fly ash and superplasticizer. Comput Concr 5(6):559–572 CrossRefGoogle Scholar
  54. 54.
    Zajaczkowski J, Verma B (2012) Selection and impact of different topologies in multi-layered hierarchical fuzzy systems. Appl Intell 36(3):564–584 CrossRefGoogle Scholar
  55. 55.
    Zhang Y, Bhattacharyya S (2004) Genetic programming in classifying large-scale data: an ensemble method. Inf Sci 163(1–3):85–101 CrossRefGoogle Scholar
  56. 56.
    Zhou A, Qu BY, Li H, Zhao SZ, Suganthan PN, Zhang Q (2011) Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol Comput 1:32–49 CrossRefGoogle Scholar
  57. 57.
    Zhou Z, Wu J, Jiang Y, Chen R (2001) Genetic algorithm based selective neural network ensemble. In: Proc of 17th int’l joint conf artif intell, vol 2. Morgan Kaufmann, San Mateo, pp 797–802 Google Scholar
  58. 58.
    Zitzler E, Künzli S (2004) Indicator-based selection in multiobjective search. In: LNCS: parallel problem solving from nature, PPSN VIII, vol 3242 Google Scholar
  59. 59.
    Zitzler E, Laumanns M, Thiele L (1999) Spea2: improving the strength pareto evolutionary algorithm for multiobjective optimization. In: Evolutionary methods for design optimisation and control, pp 95–100 Google Scholar
  60. 60.
    Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans Evol Comput 3(4):257–271 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Smart Technology Research CentreBournemouth UniversityBournemouthUK

Personalised recommendations