Genetic Programming for Kernel-Based Learning with Co-evolving Subsets Selection

  • Christian Gagné
  • Marc Schoenauer
  • Michèle Sebag
  • Marco Tomassini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4193)


Support Vector Machines (SVMs) are well-established Machine Learning (ML) algorithms. They rely on the fact that i) linear learning can be formalized as a well-posed optimization problem; ii) non-linear learning can be brought into linear learning thanks to the kernel trick and the mapping of the initial search space onto a high dimensional feature space. The kernel is designed by the ML expert and it governs the efficiency of the SVM approach. In this paper, a new approach for the automatic design of kernels by Genetic Programming, called the Evolutionary Kernel Machine (EKM), is presented. EKM combines a well-founded fitness function inspired from the margin criterion, and a co-evolution framework ensuring the computational scalability of the approach. Empirical validation on standard ML benchmark demonstrates that EKM is competitive using state-of-the-art SVMs with tuned hyper-parameters.


Support Vector Machine Subset Selection Kernel Trick Test Error Rate Linear Learning 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)Google Scholar
  2. 2.
    Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)MATHGoogle Scholar
  3. 3.
    Gilad-Bachrach, R., Navot, A., Tishby, N.: Margin based feature selection - theory and algorithms. In: Proc. of the 21st Int. Conf. on Machine Learning, pp. 43–50 (2004)Google Scholar
  4. 4.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. John Wiley & Sons, Inc., New York (2001)MATHGoogle Scholar
  5. 5.
    Yu, K., Ji, L., Zhang, X.: Kernel nearest neighbor algorithm. Neural Processing Letters 15(2), 147–156 (2002)MATHCrossRefGoogle Scholar
  6. 6.
    Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines and Other Kernel-based Learning Methods. Cambridge University Press, Cambridge (2000)Google Scholar
  7. 7.
    Potter, M.A., De Jong, K.A.: Cooperative coevolution: An architecture for evolving coadapted subcomponents. Evolutionary Computation 8(1), 1–29 (2000)CrossRefGoogle Scholar
  8. 8.
    Hillis, W.D.: Co-evolving parasites improve simulated evolution as an optimization procedure. Physica D 42, 228–234 (1990)CrossRefGoogle Scholar
  9. 9.
    Freund, Y., Shapire, R.: Experiments with a new boosting algorithm. In: Proc. of the 13th. Int. Conf. on Machine Learning, pp. 148–156 (1996)Google Scholar
  10. 10.
    Newman, D., Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998),
  11. 11.
    Gagné, C., Parizeau, M.: Genericity in evolutionary computation software tools: Principles and case-study. Int. J. on Artif. Intell. Tools 15(2), 173–194 (2006)CrossRefGoogle Scholar
  12. 12.
    Collobert, R., Bengio, S., Mariéthoz, J.: Torch: a modular machine learning software library. Technical Report IDIAP-RR 02-46, IDIAP (2002)Google Scholar
  13. 13.
    Howley, T., Madden, M.G.: The genetic kernel support vector machine: Description and evaluation. Artificial Intelligence Review 24(3-4), 379–395 (2005)CrossRefGoogle Scholar
  14. 14.
    Eads, D., Hill, D., Davis, S., Perkins, S., Ma, J., Porter, R., Theiler, J.: Genetic algorithms and support vector machines for time series classification. In: Applications and Science of Neural Networks, Fuzzy Systems, and Evolutionary Computations V, pp. 74–85 (2002)Google Scholar
  15. 15.
    Friedrichs, F., Igel, C.: Evolutionary tuning of multiple SVM parameters. Neurocomputing 64, 107–117 (2005)CrossRefGoogle Scholar
  16. 16.
    Weinberger, K., John, B., Lawrence, S.: Distance metric learning for large margin nearest neighbor classification. In: Neural Information Processing Systems, pp. 1473–1480 (2005)Google Scholar
  17. 17.
    Gathercole, C., Ross, P.: Dynamic training subset selection for supervised learning in genetic programming. In: Parallel Problem Solving From Nature, pp. 312–321 (1994)Google Scholar
  18. 18.
    Song, D., Heywood, M.I., Zincir-Heywood, A.N.: Training genetic programming on half a million patterns: an example from anomaly detection. IEEE Transactions on Evolutionary Computation 9(3), 225–239 (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Christian Gagné
    • 1
    • 2
  • Marc Schoenauer
    • 2
  • Michèle Sebag
    • 2
  • Marco Tomassini
    • 1
  1. 1.Information Systems InstituteUniversité de LausanneDorignySwitzerland
  2. 2.Équipe TAO – INRIA Futurs / CNRS UMR 8623, LRIUniversité Paris SudOrsayFrance

Personalised recommendations