Applied Intelligence

, Volume 15, Issue 3, pp 181–197 | Cite as

Scaling Up Inductive Logic Programming: An Evolutionary Wrapper Approach

  • Philip G.K. Reiser
  • Patricia J. Riddle


Inductive logic programming (ILP) algorithms are classification algorithms that construct classifiers represented as logic programs. ILP algorithms have a number of attractive features, notably the ability to make use of declarative background (user-supplied) knowledge. However, ILP algorithms deal poorly with large data sets (>104 examples) and their widespread use of the greedy set-covering algorithm renders them susceptible to local maxima in the space of logic programs.

This paper presents a novel approach to address these problems based on combining the local search properties of an inductive logic programming algorithm with the global search properties of an evolutionary algorithm. The proposed algorithm may be viewed as an evolutionary wrapper around a population of ILP algorithms.

The evolutionary wrapper approach is evaluated on two domains. The chess-endgame (KRK) problem is an artificial domain that is a widely used benchmark in inductive logic programming, and Part-of-Speech Tagging is a real-world problem from the field of Natural Language Processing. In the latter domain, data originates from excerpts of the Wall Street Journal. Results indicate that significant improvements in predictive accuracy can be achieved over a conventional ILP approach when data is plentiful and noisy.

evolutionary algorithms inductive logic programming sampling machine learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    T.G. Dietterich, “Machine learning research: Four current directions,” AI Magazine, vol. 18, No. 4, pp. 97-136, 1997.Google Scholar
  2. 2.
    T. Bäck. Evolutionary Algorithms in Theory and Practice. Oxford University Press, 1996.Google Scholar
  3. 3.
    K.A. DeJong and W.M. Spears, “Using genetic algorithms to solve NP-complete problems,” in International Conference on Genetic Algorithms, 1989, pp. 124-132.Google Scholar
  4. 4.
    D.E. Goldberg, “Genetic and evolutionary algorithms come of age,” Communications of the ACM, vol. 37, pp. 113-119, 1994.Google Scholar
  5. 5.
    K.A. DeJong, W.M. Spears, and D.F. Gordon, “Using genetic algorithms for concept learning,” Machine Learning, vol. 13, pp. 161-188, 1993.Google Scholar
  6. 6.
    A. Giordana and L. Saitta, “Regal: An integrated system for learning relations using genetic algorithms,” in Proceedings of 2nd InternationalWorkshop on Multistrategy Learning, Morgan Kaufmann, pp. 234-249, 1993.Google Scholar
  7. 7.
    A.C. Schultz and J.J. Grefenstette, “Improving tactical plans with genetic algorithms,” in Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence, number IEEE Cat. No. 90CH2915-7, Herndon,VA, 6-9Nov. 1990. IEEE Computer Society Press: Los Alamitos, CA, pp. 328-334.Google Scholar
  8. 8.
    M.L. Wong and K.S. Leung, “Inductive logic programming using genetic algorithms,” in Advances in Artificial Intelligence-Theory and Application II, edited by J.W. Brahan and G.E. Lasker, pp. 119-124, 1994.Google Scholar
  9. 9.
    J.R. Koza, Genetic Programming: On the Programming of Computers by Natural Selection, MIT Press: Cambridge, MA, 1992.Google Scholar
  10. 10.
    N.J. Radcliffe and P.D. Surry, “Cooperation through hierarchical competition in genetic data mining,” Technical Report TR9409, University of Edinburgh, Parallel Computing Centre, Edinburgh, 1994.Google Scholar
  11. 11.
    D.J. Montana, “Strongly typed genetic programming,” BBN Technical Report #7866, Bolt Beranek and Newman, Inc., 10 Moulton Street, Cambridge, MA 02138, 1994.Google Scholar
  12. 12.
    T.D. Haynes, D.A. Schoenefeld, and R.L. Wainwright, “Type inheritance in strongly typed genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear, Jr., MIT Press: Cambridge, MA, Chap. 18, pp. 359-376, 1996.Google Scholar
  13. 13.
    M.L. Wong and K.S. Leung, “An induction system that learns programs in different programming languages using genetic programming and logic grammars,” in Seventh International Conference on Tools with Artificial Intelligence, Herndon, Virginia, Nov. 1995, The Chinese University of Hong Kong, IEEE Computer Society Press, pp. 380-387.Google Scholar
  14. 14.
    M.L. Wong and K.S. Leung, “Combining genetic programming and inductive logic programming using logic grammars,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 733-736.Google Scholar
  15. 15.
    M.L. Wong and K.S. Leung, “Applying logic grammars to induce sub-functions in genetic programming,” in Proceedings of the IEEE International Conference on Evolutionary Computation, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 737-740.Google Scholar
  16. 16.
    M.L. Wong and K.S. Leung, “Evolving recursive functions for the even-parity problem using genetic programming,” in Advances in Genetic Programming 2, edited by P.J. Angeline and K.E. Kinnear Jr., MIT Press, Cambridge, MA, 1996.Google Scholar
  17. 17.
    P.A. Whigham, “Grammatically-based genetic programming,” in Proc. Workshop on Genetic Programming: From Theory to Real-World Applications, edited by J. Rosca, Morgan Kaufmann, pp. 33-41, 1995.Google Scholar
  18. 18.
    P.A. Whigham, “Search bias, language bias, and genetic programming,” in Genetic Programming 1996: Proceedings of the First Conference, Stanford University, CA, 28-31 July 1996, edited by John R. Koza, David E. Goldberg, David B. Fogel, and Rick L. Riolo, MIT Press.Google Scholar
  19. 19.
    F. Gruau, “On using syntactic constraints with genetic programming,” in Advances in Genetic Programming III, edited by P.J. Angeline and K.E. Kinnear, MIT Press: Cambridge, MA, Chap. 19, pp. 377-394, 1996.Google Scholar
  20. 20.
    P.S. Ngan, M.L. Wong, W. Lam, K.S. Leung, and J.C.Y. Cheng, “Medical data mining using evolutionary computation,” Artificial Intelligence in Medicine, vol. 16, pp. 73-96, 1999.Google Scholar
  21. 21.
    P.A. Whigham and P.F. Crapper, “Time series modelling using genetic programming: An application to rainfall-runoff models,” in Advances in Genetic Programming, edited by L. Spector, W.B. Langdon, U.-M. O'Reilly, and P.J. Angeline, MIT Press: Cambridge, MA, Chap. 5, pp. 89-104. 1999.Google Scholar
  22. 22.
    K.A. DeJong and W.M. Spears, “Learning concept classification rules using genetic algorithms,” in Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, 1991, pp. 651-656.Google Scholar
  23. 23.
    C.Z. Janikow, “A knowledge-intensive genetic algorithm for supervised learning,” Machine Learning, vol. 13, pp. 189-228, 1993.Google Scholar
  24. 24.
    J.J. Grefenstette, “A system for learning control strategies with genetic algorithms,” George Mason University, pp. 183-190. Naval Research Laboratory, 1989Google Scholar
  25. 25.
    J.J. Grefenstette, “Learning decision strategies with genetic algorithms,” in Proceedings of the International Workshop on Analogical and Inductive Inference, Lecture Notes in Artificial Intelligence 642, pp. 35-50, 1992.Google Scholar
  26. 26.
    J.J. Grefenstette, “The evolution of strategies for multiagent environments,” Adaptive Behaviour, vol. 1, no. 1, pp. 65-90, 1992.Google Scholar
  27. 27.
    J.J. Grefenstette, “Lamarckian learning in multiagent environments,” in Proceedings of 4th International Conference of Scaling Up Inductive Logic Programming 197 Genetic Algorithms, Morgan Kaufmann, pp. 303-310, 1992.Google Scholar
  28. 28.
    S. Augier, G. Venturini, and Y. Kodratoff, “Learning first order logic rules with a genetic algorithm,” in Proceedings of the First International Conference on Knowledge Discovery and Data Mining, 1995, pp. 21-26.Google Scholar
  29. 29.
    A. Giordana, L. Saitta, and F. Zini, “Learning disjunctive concepts by means of genetic algorithms,” in Proceedings of the 11th International Conference on Machine Learning, 1994, pp. 96-104.Google Scholar
  30. 30.
    F. Neri, “First order logic concept learning by means of a distributed genetic algorithm,” Ph.D. dissertation, Università of Torino, Dipartimento di Informatica, 1997.Google Scholar
  31. 31.
    L. Davis, Handbook of Genetic Algorithms, Van Nostrand Reinhold: New York, 1991.Google Scholar
  32. 32.
    W.E. Hart and R.K. Belew, “Optimization with genetic algorithm hybrids that use local search,” in Adaptive Individuals in Evolving Populations: Models and Algorithms, edited by R.K. Belew and M. Mitchell, SFI Studies in the Sciences of Complexity, vol. 26, Chap. 27, pp. 483-496, 1996.Google Scholar
  33. 33.
    D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley: Reading, MA, 1989.Google Scholar
  34. 34.
    S. Muggleton, “Inductive logic programming,” New Generation Computing, vol. 8, no. 4, pp. 295-318, 1991.Google Scholar
  35. 35.
    S. Muggleton, “Inductive logic programming: Derivations, successes and shortcomings,” SIGART Bulletin, vol. 5, no. 1, pp. 5-11, 1994.Google Scholar
  36. 36.
    S. Muggleton, “Inverse entailment and progol,” New Generation Computing, vol. 13, 1995.Google Scholar
  37. 37.
    U. Pompe, M. Kovačič, and I. Kononenko, “SFOIL: Stochastic approach to inductive logic programming,” in Proceedings of the 2nd Electrotechnical and Computer Science Conference, Portoroz, Slovenija, p. 31, 1993.Google Scholar
  38. 38.
    M. Kovačič, “Stochastic inductive logic programming,” Ph.D. Thesis, University of Ljubljana, Slovenia, 1994.Google Scholar
  39. 39.
    L. Breiman, “Bagging predictors,” Technical Report No. 421, University of California, Berkeley, Department of Statistics, 1994.Google Scholar
  40. 40.
    S.H. Muggleton, M. Bain, J. Hayes-Michie, and D. Michie, “An experimental comparison of human and machine learning formalisms,” in Proc. Sixth International Workshop on Machine Learning, Morgan Kaufmann, San Mateo, CA, 1989. pp. 113-118.Google Scholar
  41. 41.
    J. Cussens, “Part-of-speech tagging using progol,” in Inductive Logic Programming: Proceedings of the 7th InternationalWorkshop (ILP-97), 1997, volume 1297 of Lecture Notes in Artificial Intelligence, Springer pp. 93-108.Google Scholar
  42. 42.
    M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz, “Building a large annotated corpus of English: the Penn treebank,” Computational Linguistics, vol. 19, 1993.Google Scholar
  43. 43.
    J. Cussens, “Part-of-speech disambiguation using ILP,” Technical Report PRG-TR-25-96, Oxford University Computing laboratory, 1996.Google Scholar
  44. 44.
    J.R. Quinlan, “Discovering rules from large collections of examples: A case study,” in Expert Systems in the Micro-electronic Age, edited by D. Michie, Edinburgh University Press, Edinburgh, pp. 168-201, 1979.Google Scholar
  45. 45.
    J.W. Lloyd, Foundations of Logic Programming, Springer-Verlag: Berlin, 1984.Google Scholar
  46. 46.
    T.G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895-1924, 1998.Google Scholar
  47. 47.
    D. Fogel and C. deSilva, eds. Proceedings of the IEEE International Conference on Evolutionary Computation, Perth,Western Australia, Nov. 1995, IEEE Computer Society Press.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Philip G.K. Reiser
    • 1
  • Patricia J. Riddle
    • 1
  1. 1.Department of Computer ScienceUniversity of AucklandAucklandNew Zealand

Personalised recommendations