Genetic Programming and Evolvable Machines

, Volume 10, Issue 2, pp 111–140 | Cite as

Using enhanced genetic programming techniques for evolving classifiers in the context of medical diagnosis

  • Stephan M. Winkler
  • Michael Affenzeller
  • Stefan Wagner
Original Paper

Abstract

There are several data based methods in the field of artificial intelligence which are nowadays frequently used for analyzing classification problems in the context of medical applications. As we show in this paper, the application of enhanced evolutionary computation techniques to classification problems has the potential to evolve classifiers of even higher quality than those trained by standard machine learning methods. On the basis of five medical benchmark classification problems taken from the UCI repository as well as the Melanoma data set (prepared by members of the Department of Dermatology of the Medical University Vienna) we document that the enhanced genetic programming approach presented here is able to produce comparable or even better results than linear modeling methods, artificial neural networks, kNN classification, support vector machines and also various genetic programming approaches.

Keywords

Adaptation/self-adaptation Data mining Classifier systems Genetic programming Empirical study Medicine 

JEL Classification

C02 C61 C65 C67 C80 I29 

References

  1. 1.
    M. Affenzeller, Segregative genetic algorithms (SEGA): a hybrid superstructure upwards compatible to genetic algorithms for retarding premature convergence. IJCSS 2(1), 18–32 (2001)Google Scholar
  2. 2.
    M. Affenzeller, Population Genetics and Evolutionary Computation: Theoretical and Practical Aspects. Schriften der Johannes Kepler Universität Linz. Universitätsverlag Rudolf Trauner (2005)Google Scholar
  3. 3.
    M. Affenzeller, S. Wagner, SASEGASA: a new generic parallel evolutionary algorithm for achieving highest quality results. J. Heuristics - Special Issue on New Advances on Parallel Meta-Heuristics for Complex Problems 10, 239–263 (2004)Google Scholar
  4. 4.
    M. Affenzeller, S. Wagner, Offspring selection: a new self-adaptive selection scheme for genetic algorithms, in Adaptive and Natural Computing Algorithms, ed. by B. Ribeiro, R.F. Albrecht, A. Dobnikar, D.W. Pearson, N.C. Steele (Springer Computer Science, Springer, 2005), pp. 218–221CrossRefGoogle Scholar
  5. 5.
    D. Alberer, L. del Re, S. Winkler, P. Langthaler, Virtual sensor design of particulate and nitric oxide emissions in a di diesel engine. in Proceedings of the 7th International Conference on Engines for Automobile ICE 2005, 2005-24-063, Capri, Italy, 2005Google Scholar
  6. 6.
    W. Banzhaf, C. Lasarczyk, Genetic programming of an algorithmic chemistry. in Genetic Programming Theory and Practice II. ed. by U. O’Reilly, T. Yu, R. Riolo, B. Worzel (University of Michigan, Ann Arbor, 2004), pp. 175–190Google Scholar
  7. 7.
    H. Beyer, The Theory of Evolution Strategies. Springer, New York (2001)Google Scholar
  8. 8.
    C. Bojarczuk, H. Lopes, A. Freitas, Discovering comprehensible classification rules using genetic programming: a case study in a medical domain. in Proceedings of the Genetic and Evolutionary Computation Conference, vol. 2, Orlando, Florida, USA, 1999, pp. 953–958Google Scholar
  9. 9.
    C. Bojarczuk, H. Lopes, A. Freitas, E. Michalkiewicz, A constrained-syntax genetic programming system for discovering classification rules: application to medical data sets. Artif. Intell. Med. 30(1), 27–48 (2004)CrossRefGoogle Scholar
  10. 10.
    A. Bradley, The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit. 30, 1145–1159 (1997)CrossRefGoogle Scholar
  11. 11.
    M. Brameier, W. Banzhaf, A comparison of linear genetic programming and neural networks inmedical data mining. Evolutionary Computation, IEEE Transactions on 5(1), 17–26 (2001)CrossRefGoogle Scholar
  12. 12.
    C. Brodley, P. Utgoff, Multivariate decision trees. Mach. Learn. 19(1), 45–77 (1995)MATHGoogle Scholar
  13. 13.
    G. Brown, Diversity in neural network ensembles. Ph.D. thesis, School of Computer Science, University of Birmingham, 2003Google Scholar
  14. 14.
    I. De Falco, A. Della Cioppa, E. Tarantino, Discovering interesting classification rules with genetic programming. Appl. Soft Comput. J. 1(4), 257–269 (2002)CrossRefGoogle Scholar
  15. 15.
    S. Dreiseitl, L. Ohno-Machado, H. Kittler, S. Vinterbo, H. Billhardt, M. Binder, A comparison of machine learning methods for the diagnosis of pigmented skin lesions. J. Biomed. Inform. 34, 28–36 (2001)CrossRefGoogle Scholar
  16. 16.
    W. Duch, R. Adamczak, K. Grabczewski, A new methodology of extraction, optimization and application of crisp and fuzzy logical rules. IEEE Trans. Neural Netw. 12, 277–306 (2001)CrossRefGoogle Scholar
  17. 17.
    R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, 2nd edn. (Wiley Interscience, 2000)Google Scholar
  18. 18.
    J. Eggermont, J.N. Kok, W.A. Kosters, Genetic programming for data classification: partitioning the search space. in Proceedings of the 2004 Symposium on applied computing ACM SAC’04, Nicosia, Cyprus, ACM, 2004, pp. 1001–1005Google Scholar
  19. 19.
    C. Gathercole, P. Ross, Dynamic training subset selection for supervised learning in genetic programming. in Parallel Problem Solving from Nature III, LNCS, vol. 866 ed. by Y. Davidor, H.P. Schwefel, R. Männer, (Springer-Verlag, 1994), pp. 312–321Google Scholar
  20. 20.
    P. Gill, W. Murray, M. Wright, Practical Optimization. (Academic Press, 1982)Google Scholar
  21. 21.
    H. Hamilton, N. Shan, N. Cercone, Riac: a rule induction algorithm based on approximate classification. Tech. Rep. CS 96-06, (Regina University, 1996)Google Scholar
  22. 22.
    P.L. Hammer, A. Kogan, B. Simeone, S. Szedmak, Pareto-optimal patterns in logical analysis of data. Discrete Appl. Math. 144, 102 (2004)MathSciNetGoogle Scholar
  23. 23.
    I. Jonyer, L.B. Holder, D.J. Cook, Attribute-value selection based on minimum description length. in Proceedings of the International Conference on Artificial Intelligence, Las Vegas, Nevada, USA, 2004, pp. 1154–1159Google Scholar
  24. 24.
    S. Keerthi, S. Shevade, C. Bhattacharyya, K. Murthy, Improvements to platt’s SMO algorithm for SVM classifier design. Neural Comput. 13(3), 637–649 (2001)CrossRefMATHGoogle Scholar
  25. 25.
    J.R. Koza, Genetic Programming: On the Programming of Computers by Means of Natural Selection. The MIT Press (1992)Google Scholar
  26. 26.
    K. Levenberg, A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2, 164–168 (1944)MathSciNetMATHGoogle Scholar
  27. 27.
    D.P.X. Li, V. Ciesielski, Multi-objective techniques in genetic programming for evolving classifiers. in Proceedings of the 2005 Congress on Evolutionary Computation (CEC ’05), Munich, Germany, 2005, pp. 183–190Google Scholar
  28. 28.
    P. Lichodzijewski, M. Heywood, Pareto-coevolutionary genetic programming for problem decomposition in multi-class classification. in Proceedings of the Genetic and Evolutionary Computation Conference GECCO’07, London, England, 2006, pp. 464–471Google Scholar
  29. 29.
    L. Ljung, System Identification – Theory For the User, 2nd edn. (PTR Prentice Hall, Upper Saddle River, NJ, 1999)Google Scholar
  30. 30.
    T. Loveard, V. Ciesielski, Representing classification problems in genetic programming. in Proceedings of the Congress on Evolutionary Computation, vol. 2 (IEEE Press, COEX, World Trade Center, 159 Samseong-dong, Gangnam-gu, Seoul, Korea, 2001), pp. 1070–1077Google Scholar
  31. 31.
    D.W. Marquardt, An algorithm for least-squares estimation of nonlinear parameters. SIAM J. Appl. Math. 11, 431–441 (1963)CrossRefMathSciNetMATHGoogle Scholar
  32. 32.
    T.M. Mitchell, Machine Learning. (McGraw-Hill, New York, 2000)Google Scholar
  33. 33.
    B. Moghaddam, G. Shakhnarovich, Boosted dyadic kernel discriminants. in Advances in Neural Information Processing Systems (NIPS), vol. 15 ed. by S. Becker, S. Thrun, K. Obermayer (2002)Google Scholar
  34. 34.
    K. Morik, M. Imhoff, P. Brockhausen, T. Joachims, U. Gather, Knowledge discovery and knowledge validation in intensive care. Artif. Intell. Med. 19, 225–249 (2000)CrossRefGoogle Scholar
  35. 35.
    O. Nelles, Nonlinear System Identification. (Springer Verlag, Berlin Heidelberg, New York, 2001)MATHGoogle Scholar
  36. 36.
    P. Ngan, M. Wong, K. Leung, J. Cheng, Using grammar based genetic programming for data mining of medical knowledge. (Genetic Programming, 1998), pp. 254–259Google Scholar
  37. 37.
    M. Nørgaard, Neural network based system identification toolbox. Tech. Rep. 00-E-891, Technical University of Denmark (2000)Google Scholar
  38. 38.
    J. Platt, Fast training of support vector machines using sequential minimal optimization. in Advances in Kernel Methods-Support Vector Learning, ed. by B. Schoelkopf, C. Burges, A. Smola, (MIT Press, 1999). pp. 185–208Google Scholar
  39. 39.
    L. Prechelt, Proben1 - a set of neural network benchmark problems and benchmarking rules. Tech. rep., Fakultät für Informatik, Universität Karlsruhe (1994)Google Scholar
  40. 40.
    M.L. Raymer, L.A. Kuhn, W.F. Punch, Knowledge discovery in biological datasets using a hybrid bayes classifier/evolutionary algorithm. in BIBE ’01: Proceedings of the 2nd IEEE International Symposium on Bioinformatics and Bioengineering. (IEEE Computer Society, Washington, DC, USA, 2001), pp. 236–244Google Scholar
  41. 41.
    S.J. Russell, P. Norvig, Artificial Intelligence: A Modern Approach, 2nd edn. (Prentice Hall, 2003)Google Scholar
  42. 42.
    W. Schiffmann, M. Joost, R. Werner, Optimization of the backpropagation algorithm for training multilayer perceptrons. Tech. Rep. 15, University of Koblenz, Institute of Physics (1992)Google Scholar
  43. 43.
    W. Schiffmann, M. Joost, R. Werner, Comparison of optimized backpropagation algorithms. in Proceedings of the European Symposium on Artificial Neural Networks ESANN ’93, Brussels, Belgium, 1993, pp. 97–104Google Scholar
  44. 44.
    I. Taha, J. Ghosh, Evaluation and ordering of rules extracted from feedforward networks. in Proceedings of the IEEE International Conference on Neural Networks, Houston, Texas, USA, 1997, pp. 221–226Google Scholar
  45. 45.
    M.K. Titsias, A.C. Likas, Shared kernel models for class conditional density estimation. IEEE-NN 12, 987–997 (2001)Google Scholar
  46. 46.
    V. Vapnik, Statistical Learning Theory. (Wiley, New York, 1998)MATHGoogle Scholar
  47. 47.
    S. Wagner, M. Affenzeller, Heuristiclab: A generic and extensible optimization environment. in Adaptive and Natural Computing Algorithms, ed. by B. Ribeiro, R.F. Albrecht, A. Dobnikar, D.W. Pearson, N.C. Steele, (Springer Computer Science, Springer, 2005a), pp. 538–541Google Scholar
  48. 48.
    S. Wagner, M. Affenzeller, Sexual GA: gender-specific selection for genetic algorithms. in Proceedings of the 9th World Multi-Conference on Systemics, Cybernetics and Informatics (WMSCI) 2005, vol. 4, Orlando, Florida, USA, ed. by N. Callaos, W. Lesso, E. Hansen. (International Institute of Informatics and Systemics, 2005b), pp. 76–81Google Scholar
  49. 49.
    S. Weiss, I. Kapouleas, An empirical comparison of pattern recognition, neural nets, and machine learning classification methods. in Readings in Machine Learning. ed. by Shavlik J.W., Dietterich T.G. (Kaufmann, San Mateo, CA), pp. 177–183 (1990)Google Scholar
  50. 50.
    J. Wen-Hua, D. Madigan, S.L. Scott, On bayesian learning of sparse classifiers. Tech. Rep. 2003-08, Avaya Labs Research (2003)Google Scholar
  51. 51.
    S. Winkler, Evolutionary system identification—modern concepts and practical applications. Ph.D. thesis, Institute for Formal Models and Verification, Johannes Kepler University Linz, 2008Google Scholar
  52. 52.
    S. Winkler, M. Affenzeller, S. Wagner, Automatic data based patient classification using genetic programming. in Cybernetics and Systems 2006, vol. 1, ed. by R. Trappl, R. Brachman, R. Brooks, H. Kitano, D. Lenat, O. Stock, W. Wahlster, M. Wooldridge. (Austrian Society for Cybernetic Studies, 2006a), pp. 251–256Google Scholar
  53. 53.
    S. Winkler, M. Affenzeller, S. Wagner, Using enhanced genetic programming techniques for evolving classifiers in the context of medical diagnosis—an empirical study. in Proceedings of the GECCO 2006 Workshop on Medical Applications of Genetic and Evolutionary Computation (MedGEC 2006), Seattle, Washington, USA. Association for Computing Machinery (ACM), 2006bGoogle Scholar
  54. 54.
    S. Winkler, M. Affenzeller, S. Wagner, Advanced genetic programming based machine learning. J. Math. Model. Algorithms 6(3), 455–480 (2007a)CrossRefMathSciNetMATHGoogle Scholar
  55. 55.
    S. Winkler, M. Affenzeller, S. Wagner, Selection pressure driven sliding window genetic programming. Lecture Notes in Computer Science 4739: Computer Aided Systems Theory - EuroCAST 2007, pp. 789–795 (2007b)Google Scholar
  56. 56.
    S. Winkler, M. Affenzeller, S. Wagner, Offspring selection and its effects on genetic propagation in genetic programming based system identification. in Cybernetics and Systems 2008, vol. 2, ed. by R. Trappl. (Austrian Society for Cybernetic Studies, 2008), pp. 549–554Google Scholar
  57. 57.
    I. Witten, E. Frank, Data Mining: Practical Machine Learning Tools and Techniques. (Morgan Kaufmann, San Francisco, 2005)MATHGoogle Scholar
  58. 58.
    M.L. Wong, K.S. Leung, Inducing logic programs with genetic algorithms: the genetic logicprogramming system genetic logic programming and applications. IEEE Expert 10(5), 68–76 (1995)CrossRefGoogle Scholar
  59. 59.
    M.L. Wong, K.S. Leung, Evolutionary program induction directed by logic grammars. Evol. Comput. 5(2), 143–180 (1997)CrossRefMathSciNetGoogle Scholar
  60. 60.
    Z.H. Zhou, Y. Jiang, Nec4.5: neural ensemble based c4.5. IEEE Trans. Knowl. Data Eng. 16(6), 770–773 (2004)CrossRefMathSciNetGoogle Scholar
  61. 61.
    M.H. Zweig, Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39, 561–577 (1993)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Stephan M. Winkler
    • 1
  • Michael Affenzeller
    • 2
  • Stefan Wagner
    • 2
  1. 1.Research Center Hagenberg, Upper Austria University of Applied SciencesHagenbergAustria
  2. 2.Department of Software EngineeringUpper Austria University of Applied SciencesHagenbergAustria

Personalised recommendations