Evolutionary Intelligence

, Volume 4, Issue 1, pp 3–16 | Cite as

Novel evolutionary algorithms for supervised classification problems: an experimental study

Special Issue

Abstract

Evolutionary Algorithms (EAs) are population-based, stochastic search algorithms that mimic natural evolution. Over the years, EAs have been successfully applied to many classification problems. In this paper, we present three novel evolutionary approaches and analyze their performances for synthesizing classifiers with EAs in supervised data mining scenarios. The first approach is based on encoding rule sets with bit string genomes, while the second one utilizes Genetic Programming (GP) to create decision trees with arbitrary expressions attached to the nodes. The novelty of these two approaches lies in the use of solutions on the Pareto front as an ensemble. The third approach, EDDIE-101, is also based on GP but uses a new, advanced fitness measure and some novel genetic operators. We compare these approaches to a number of well-known data mining methods, including C4.5 and Random-Forest, and show that the performances of our evolved classifiers can be very competitive as far as the solution quality is concerned. In addition, the proposed approaches work well across a wide range of configurations, and EDDIE-101 particularly has been highly efficient. To further evaluate the flexibility of EDDIE-101 across different problem domains, we also test it on some real financial datasets for finding investment opportunities and compare the results with those obtained using other classifiers. Numerical experiments confirm that EDDIE-101 can be successfully extended to financial forecasting.

Keywords

Data mining Evolutionary algorithms Rule-based classifiers Decision trees EDDIE-101 

References

  1. 1.
    Alba Torres E, Tomassini M (2002) Parallelism and evolutionary algorithms. IEEE Trans Evol Comput 6(5):443–462CrossRefGoogle Scholar
  2. 2.
    Anderson E (1935) The irises of the Gaspé Peninsula. Bull Am Iris Soc 59:2–5Google Scholar
  3. 3.
    Au WH, Chan KCC, Yao X (2003) A novel evolutionary data mining algorithm with applications to churn prediction. IEEE Trans Evol Comput 7(6):532–545CrossRefGoogle Scholar
  4. 4.
    Avnimelech R, Intrator N (1999) Boosting regression estimators. Neural Comput Appl 11(2):499–520CrossRefGoogle Scholar
  5. 5.
    Bacardit J, Butz MV (2007) Data mining in learning classifier systems: comparing XCS with GAssist. In: Revised selected papers of the international workshops on learning classifier systems, Springer, Lecture notes in artificial intelligence, vol 4399, pp 282–290Google Scholar
  6. 6.
    Bäck T (1996) Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford University Press, OxfordMATHGoogle Scholar
  7. 7.
    Bako L (2010) Real-time classification of datasets with hardware embedded neuromorphic neural networks. Briefings Bioinf 11(3):348–363CrossRefGoogle Scholar
  8. 8.
    Balian R (2004) Entropy, a protean concept. In: Poincaré seminar 2003, Birkhäuser Verlag, Progress in mathematical physics, vol 38, pp 119–144Google Scholar
  9. 9.
    Bernadó E, Llorà X, Garrell i Guiu JM (2001) XCS and GALE: a comparative study of two learning classifier systems with six other learning algorithms on classification tasks. In: Advances in learning classifier systems, Revised Papers of IWLCS’01, Springer, Lecture notes in artificial intelligence, vol 2321, pp 115–132Google Scholar
  10. 10.
    Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140MathSciNetMATHGoogle Scholar
  11. 11.
    Breiman L (2001) Random forests. Mach Learn 45(1):5–32CrossRefMATHGoogle Scholar
  12. 12.
    Bull, L, Bernadó-Mansilla, E, Holmes, J (eds) (2008) Learning classifier systems in data mining. In: Studies in computational intelligence, vol 125. Springer, BerlinGoogle Scholar
  13. 13.
    Cantú-Paz E, Kamath C (2000) Using evolutionary algorithms to induce oblique decision trees. In: Proceedings of the genetic and evolutionary computation conference, Morgan Kaufmann Publishers, pp 1053–1060Google Scholar
  14. 14.
    Chiong, R (eds) (2009) Nature-inspired algorithms for optimisation. In: Studies in computational intelligence, vol 193. Springer, BelinGoogle Scholar
  15. 15.
    Chiong R, Neri F, McKay RI (2009) Nature that breeds solutions. In: Nature-inspired informatics for intelligent applications and knowledge discovery: implications in business, science and engineering, information science reference, chap 1, pp 1–24Google Scholar
  16. 16.
    Corcoran AL, Sen S (1994) Using real-valued genetic algorithms to evolve rule sets for classification. In: Proceedings of the first IEEE conference on evolutionary computation, IEEE computer society, vol 1, pp 120–124Google Scholar
  17. 17.
    De Jong KA, Spears WM (1991) Learning concept classification rules using genetic algorithms. In: Mylopoulos J, Reiter R (eds) Proceedings of the 12th international joint conference on artificial intelligence, Morgan Kaufmann Publishers, vol 2, pp 651–656Google Scholar
  18. 18.
    Fernández A, García S, Luengo J, Bernadó-Mansilla E, Herrera F (2010) Genetics-based machine learning for rule induction: state of the art, taxonomy, and comparative study. IEEE Trans Evol Comput. doi:10.1109/TEVC.2009.2039140
  19. 19.
    Fidelis M, Lopes HS, Freitas AA (2000) Discovering comprehensible classification rules with a genetic algorithm. In: Proceedings of the IEEE congress on evolutionary computation, IEEE computer society, vol 1, pp 805–810Google Scholar
  20. 20.
    Forina M, Lanteri S, Armanino C et al (1988) PARVUS—an extendible package for data exploration, classification and correlation. Institute of Pharmaceutical and Food Analysis and Technologies, GenoaGoogle Scholar
  21. 21.
    Forsyth R (1981) BEAGLE—a Darwinian approach to pattern recognition. Kybernetes 10(3):159–166CrossRefMathSciNetGoogle Scholar
  22. 22.
    Frank E, Hall MA, Holmes G, Kirkby R, Pfahringer B, Witten IH, Trigg L (2005) WEKA—a machine learning workbench for data mining. In: The data mining and knowledge discovery Handbook, Springer, chap 62, pp 1305–1314Google Scholar
  23. 23.
    Frawley WJ, Piatetsky-Shapiro G, Matheus CJ (1992) Knowledge discovery in databases: an overview. AI Mag 13(3):213–228Google Scholar
  24. 24.
    Freitas AA (1997) A genetic programming framework for two data mining tasks: classification and generalized rule induction. In: Proceedings of the second annual conference on genetic programming, Morgan Kaufmann Publishers, pp 96–101Google Scholar
  25. 25.
    Freitas AA (2002) Data mining and knowledge discovery with evolutionary algorithms. Natural computing series. Springer, BerlinGoogle Scholar
  26. 26.
    García-Almanza AL, Tsang EPK (2006) The repository method for chance discovery in financial forecasting. In: Proceedings of the 10th international conference on knowledge-based intelligent information and engineering systems, Part III, Springer, Lecture notes in artificial intelligence, vol 4253, pp 30–37Google Scholar
  27. 27.
    García-Almanza AL, Tsang EPK, Galván-López E (2008) Evolving decision rules to discover patterns in financial data sets. In: Computational methods in financial engineering—essays in honour of Manfred Gilli, Springer, chap II-5, pp 239–255Google Scholar
  28. 28.
    Gehrke J, Ramakrishnan R, Ganti V (1998) RainForest—a framework for fast decision tree construction of large datasets. In: Proceedings of 24rd international conference on very large data bases, Morgan Kaufmann Publishers, pp 416–427Google Scholar
  29. 29.
    Ghosh, A, Jain, LC (eds) (2005) Evolutionary computation in data mining. In: Studies in fuzziness and soft computing, vol 163. Springer, BerlinGoogle Scholar
  30. 30.
    Gong G, Cestnik B (1988) Hepatitis data set. UCI Machine Learning Repository, University of CaliforniaGoogle Scholar
  31. 31.
    Grzymala-Busse JW (1997) A new version of the rule induction system LERS. Fundamenta Informaticae – Annales Societatis Mathematicae Polonae, Series IV 31(1):27–39MATHGoogle Scholar
  32. 32.
    Harding JA, Shahbaz M, Srinivas, Kusiak A (2006) Data mining in manufacturing: a review. J Manuf Sci Eng 128(4):969–977CrossRefGoogle Scholar
  33. 33.
    Holland JH (1986) Escaping brittleness: the possibilities of general-purpose learning algorithms applied to parallel rule-based systems. In: Machine learning: an artificial intelligence approach, vol II, William Kaufmann, pp 593–623Google Scholar
  34. 34.
    Holmes G, Donkin A, Witten IH (1994) WEKA: a machine learning workbench. In: Proceedings of the second Australia and New Zealand conference on intelligent information systems, IEEE Computer Society Press, pp 357–361Google Scholar
  35. 35.
    Hsu PL, Lai R, Chiu CC (2003) The hybrid of association rule algorithms and genetic algorithms for tree induction: an example of predicting the student course performance. Expert Syst Appl Int J 25(1):51–62CrossRefGoogle Scholar
  36. 36.
    Jabeen H, Baig AR (2010) Review of classification using genetic programming. Int J Eng Sci Technol 2(2):94–103Google Scholar
  37. 37.
    Kharbat F, Bull L, Odeh M (2007) Mining breast cancer data with XCS. In: Proceedings of 9th genetic and evolutionary computation conference, ACM Press, pp 2066–2073Google Scholar
  38. 38.
    Koza JR (1990) Concept formation and decision tree induction using the genetic programming paradigm. In: Proceedings of the 1st workshop on parallel problem solving from nature, Springer, Lecture notes in computer science, vol 496, pp 124–128Google Scholar
  39. 39.
    Koza JR (1992) Genetic programming: on the programming of computers by means of natural selection. Bradford Books, MIT Press, CambridgeMATHGoogle Scholar
  40. 40.
    Li J (2001) FGP: a genetic programming based tool for financial forecasting. PhD thesis, University of EssexGoogle Scholar
  41. 41.
    Li J, Li X, Yao X (2005) Cost-sensitive classification with genetic programming. In: Proceedings of the IEEE congress on evolutionary computation, IEEE Computer Society, pp 2114–2121Google Scholar
  42. 42.
    Liu TY, Yang Y, Wan H, Zeng HJ, Chen Z, Ma WY (2005) Support vector machines classification with a very large-scale taxonomy. ACM SIGKDD Explor Newsl 7(1):36–43CrossRefGoogle Scholar
  43. 43.
    Martin WN, Lienig J, Cohoon JP (1997) Island (Migration) models: evolutionary algorithms based on punctuated equilibria. In: Handbook of evolutionary computation, computational intelligence library, Oxford University Press, chap C6.3, pp 448–463Google Scholar
  44. 44.
    Mehta M, Agrawal R, Rissanen J (1996) SLIQ: a fast scalable classifier for data mining. In: Advances in database technology—5th international conference on extending database technology, Springer, Lecture notes in computer science, vol 1057, pp 18–32Google Scholar
  45. 45.
    Muni DP, Pal NR, Das J (2004) A novel approach to design classifiers using genetic programming. IEEE Trans Evol Comput 8(2):183–196CrossRefGoogle Scholar
  46. 46.
    Orriols-Puig A, Bernadó-Mansilla E (2009) Evolutionary rule-based systems for imbalanced data sets. Soft Comput 13(3):213–225CrossRefGoogle Scholar
  47. 47.
    Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2008) Genetic-based machine learning systems are competitive for pattern recognition. Evol Intell 1(3):209–232CrossRefGoogle Scholar
  48. 48.
    Orriols-Puig A, Casillas J, Bernadó-Mansilla E (2009) Fuzzy-UCS: a Michigan-style learning fuzzy-classifier system for supervised learning. IEEE Trans Evol Comput 13(2):260–283CrossRefGoogle Scholar
  49. 49.
    Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann Publishers, MassachusettsGoogle Scholar
  50. 50.
    Rastogi R, Shim K (1998) PUBLIC: a decision tree classifier that integrates building and pruning. In: Proceedings of 24th international conference on very large data bases. Morgan Kaufmann Publishers, Massachusetts, pp 404–415Google Scholar
  51. 51.
    Schapire RE (1990) The strength of weak learnability. Mach Learn 5:197–227Google Scholar
  52. 52.
    Shafer JC, Agrawal R, Mehta M (1996) SPRINT: a scalable parallel classifier for data mining. In: Proceedings of 22nd international conference on very large data bases. Morgan Kaufmann Publishers, Massachusetts, pp 544–555Google Scholar
  53. 53.
    Siegel S, Castellan NJ Jr (1956) Nonparametric statistics for the behavioral sciences. Humanities/social sciences/languages. McGraw-Hill, New YorkGoogle Scholar
  54. 54.
    Sir Fisher RA (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7:179–188Google Scholar
  55. 55.
    Smith SF (1980) A learning system based on genetic adaptive algorithms. PhD thesis, University of PittsburghGoogle Scholar
  56. 56.
    Spears WM, De Jong KA (1990) Using genetic algorithms for supervised concept learning. In: Proceedings of the 2nd international IEEE conference on tools for artificial intelligence, IEEE Computer Society Press, pp 335–341Google Scholar
  57. 57.
    Stefanowski J, Slowinski K (1996) Rough sets as a tool for studying attribute dependencies in the urinary stones treatment data set. In: Rough sets and data mining: analysis of imprecise data. Kluwer Academic Publishers, pp 177–196Google Scholar
  58. 58.
    Tanwani AK, Farooq M (2009) Performance evaluation of evolutionary algorithms in classification of biomedical datasets. In: Proceedings of the 11th annual conference—companion on genetic and evolutionary computation conference, ACM, pp 2617–2624Google Scholar
  59. 59.
    Tapia JJ, Morett E, Vallejo EE (2009) A clustering genetic algorithm for genomic data mining. In: Foundations of computational intelligence—vol 4: bio-inspired data mining. Studies in computational intelligence, vol 204, Springer, pp 249–275Google Scholar
  60. 60.
    Tsang EPK, Butler JM, Li J (1998) EDDIE beats the bookies. Int J Softw Pract Exper 28(10):1033–1043CrossRefGoogle Scholar
  61. 61.
    Tsang EPK, Li J, Markose SM, Hakan ER, Salhi A, Iori G (2000) EDDIE in financial decision making. J Manage Econ 4(4)Google Scholar
  62. 62.
    Tsang EPK, Yung P, Li J (2004) EDDIE-automation—a decision support tool for financial forecasting. Decis Support Syst 37(4):559–565CrossRefGoogle Scholar
  63. 63.
    van Veldhuizen DA, Merkle LD (1999) Multiobjective evolutionary algorithms: classifications, analyses, and new innovations. PhD thesis, Air University, Air Force Institute of Technology: Wright-Patterson Air Force Base, OH, USAGoogle Scholar
  64. 64.
    Weise T (2009a) Evolving distributed algorithms with genetic programming. PhD thesis, University of Kassel, Distributed Systems Group: Kassel, GermanyGoogle Scholar
  65. 65.
    Weise T (2009b) Global optimization algorithms—theory and application. http://www.it-weise.de/
  66. 66.
    Weise T, Geihs K (2006) DGPF—an adaptable framework for distributed multi-objective search algorithms applied to the genetic programming of sensor networks. In: Proceedings of the second international conference on bioinspired optimization methods and their applications, Jožef Stefan Institute, Informacijska Družba, pp 157–166Google Scholar
  67. 67.
    Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics Bull 1(6):80–83CrossRefGoogle Scholar
  68. 68.
    Wolberg WH, Mangasarian O (1989) Breast cancer Wisconsin (Original) data set. UCI Machine Learning Repository, University of CaliforniaGoogle Scholar
  69. 69.
    Zhang GP (2000) Neural networks for classification: a survey. IEEE Trans Syst Man Cybern Part C Appl Rev 30(4):451–462CrossRefGoogle Scholar
  70. 70.
    Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: empirical results. Evol Comput 8(2):173–195CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Nature Inspired Computation & Applications LaboratoryUniversity of Science and Technology of ChinaHefeiChina
  2. 2.Faculty of Information & Communication TechnologiesSwinburne University of TechnologyMelbourneAustralia

Personalised recommendations