# Scaling Up Inductive Logic Programming: An Evolutionary Wrapper Approach

- 60 Downloads
- 3 Citations

## Abstract

Inductive logic programming (ILP) algorithms are classification algorithms that construct classifiers represented as logic programs. ILP algorithms have a number of attractive features, notably the ability to make use of declarative background (user-supplied) knowledge. However, ILP algorithms deal poorly with large data sets (>10^{4} examples) and their widespread use of the greedy set-covering algorithm renders them susceptible to local maxima in the space of logic programs.

This paper presents a novel approach to address these problems based on combining the local search properties of an inductive logic programming algorithm with the global search properties of an evolutionary algorithm. The proposed algorithm may be viewed as an evolutionary wrapper around a population of ILP algorithms.

The evolutionary wrapper approach is evaluated on two domains. The chess-endgame (KRK) problem is an artificial domain that is a widely used benchmark in inductive logic programming, and Part-of-Speech Tagging is a real-world problem from the field of Natural Language Processing. In the latter domain, data originates from excerpts of the Wall Street Journal. Results indicate that significant improvements in predictive accuracy can be achieved over a conventional ILP approach when data is plentiful and noisy.

## Preview

Unable to display preview. Download preview PDF.

## References

- 1.T.G. Dietterich, “Machine learning research: Four current directions,”
*AI Magazine*, vol. 18, No. 4, pp. 97-136, 1997.Google Scholar - 2.T. Bäck.
*Evolutionary Algorithms in Theory and Practice*. Oxford University Press, 1996.Google Scholar - 3.K.A. DeJong and W.M. Spears, “Using genetic algorithms to solve NP-complete problems,” in
*International Conference on Genetic Algorithms*, 1989, pp. 124-132.Google Scholar - 4.D.E. Goldberg, “Genetic and evolutionary algorithms come of age,”
*Communications of the ACM*, vol. 37, pp. 113-119, 1994.Google Scholar - 5.K.A. DeJong, W.M. Spears, and D.F. Gordon, “Using genetic algorithms for concept learning,”
*Machine Learning*, vol. 13, pp. 161-188, 1993.Google Scholar - 6.A. Giordana and L. Saitta, “Regal: An integrated system for learning relations using genetic algorithms,” in
*Proceedings of 2nd InternationalWorkshop on Multistrategy Learning*, Morgan Kaufmann, pp. 234-249, 1993.Google Scholar - 7.A.C. Schultz and J.J. Grefenstette, “Improving tactical plans with genetic algorithms,” in
*Proceedings of the 2nd International IEEE Conference on Tools for Artificial Intelligence*, number IEEE Cat. No. 90CH2915-7, Herndon,VA, 6-9Nov. 1990. IEEE Computer Society Press: Los Alamitos, CA, pp. 328-334.Google Scholar - 8.M.L. Wong and K.S. Leung, “Inductive logic programming using genetic algorithms,” in
*Advances in Artificial Intelligence-Theory and Application II*, edited by J.W. Brahan and G.E. Lasker, pp. 119-124, 1994.Google Scholar - 9.J.R. Koza,
*Genetic Programming: On the Programming of Computers by Natural Selection*, MIT Press: Cambridge, MA, 1992.Google Scholar - 10.N.J. Radcliffe and P.D. Surry, “Cooperation through hierarchical competition in genetic data mining,” Technical Report TR9409, University of Edinburgh, Parallel Computing Centre, Edinburgh, 1994.Google Scholar
- 11.D.J. Montana, “Strongly typed genetic programming,” BBN Technical Report #7866, Bolt Beranek and Newman, Inc., 10 Moulton Street, Cambridge, MA 02138, 1994.Google Scholar
- 12.T.D. Haynes, D.A. Schoenefeld, and R.L. Wainwright, “Type inheritance in strongly typed genetic programming,” in
*Advances in Genetic Programming 2*, edited by P.J. Angeline and K.E. Kinnear, Jr., MIT Press: Cambridge, MA, Chap. 18, pp. 359-376, 1996.Google Scholar - 13.M.L. Wong and K.S. Leung, “An induction system that learns programs in different programming languages using genetic programming and logic grammars,” in
*Seventh International Conference on Tools with Artificial Intelligence*, Herndon, Virginia, Nov. 1995, The Chinese University of Hong Kong, IEEE Computer Society Press, pp. 380-387.Google Scholar - 14.M.L. Wong and K.S. Leung, “Combining genetic programming and inductive logic programming using logic grammars,” in
*Proceedings of the IEEE International Conference on Evolutionary Computation*, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 733-736.Google Scholar - 15.M.L. Wong and K.S. Leung, “Applying logic grammars to induce sub-functions in genetic programming,” in
*Proceedings of the IEEE International Conference on Evolutionary Computation*, Perth, Western Australia, Nov. 1995, IEEE Computer Society Press, pp. 737-740.Google Scholar - 16.M.L. Wong and K.S. Leung, “Evolving recursive functions for the even-parity problem using genetic programming,” in
*Advances in Genetic Programming 2*, edited by P.J. Angeline and K.E. Kinnear Jr., MIT Press, Cambridge, MA, 1996.Google Scholar - 17.P.A. Whigham, “Grammatically-based genetic programming,” in
*Proc. Workshop on Genetic Programming: From Theory to Real-World Applications*, edited by J. Rosca, Morgan Kaufmann, pp. 33-41, 1995.Google Scholar - 18.P.A. Whigham, “Search bias, language bias, and genetic programming,” in
*Genetic Programming 1996: Proceedings of the First Conference*, Stanford University, CA, 28-31 July 1996, edited by John R. Koza, David E. Goldberg, David B. Fogel, and Rick L. Riolo, MIT Press.Google Scholar - 19.F. Gruau, “On using syntactic constraints with genetic programming,” in
*Advances in Genetic Programming III*, edited by P.J. Angeline and K.E. Kinnear, MIT Press: Cambridge, MA, Chap. 19, pp. 377-394, 1996.Google Scholar - 20.P.S. Ngan, M.L. Wong, W. Lam, K.S. Leung, and J.C.Y. Cheng, “Medical data mining using evolutionary computation,”
*Artificial Intelligence in Medicine*, vol. 16, pp. 73-96, 1999.Google Scholar - 21.P.A. Whigham and P.F. Crapper, “Time series modelling using genetic programming: An application to rainfall-runoff models,” in
*Advances in Genetic Programming*, edited by L. Spector, W.B. Langdon, U.-M. O'Reilly, and P.J. Angeline, MIT Press: Cambridge, MA, Chap. 5, pp. 89-104. 1999.Google Scholar - 22.K.A. DeJong and W.M. Spears, “Learning concept classification rules using genetic algorithms,” in
*Proceedings of the Twelfth International Joint Conference on Artificial Intelligence*, 1991, pp. 651-656.Google Scholar - 23.C.Z. Janikow, “A knowledge-intensive genetic algorithm for supervised learning,”
*Machine Learning*, vol. 13, pp. 189-228, 1993.Google Scholar - 24.J.J. Grefenstette, “A system for learning control strategies with genetic algorithms,” George Mason University, pp. 183-190. Naval Research Laboratory, 1989Google Scholar
- 25.J.J. Grefenstette, “Learning decision strategies with genetic algorithms,” in
*Proceedings of the International Workshop on Analogical and Inductive Inference*, Lecture Notes in Artificial Intelligence 642, pp. 35-50, 1992.Google Scholar - 26.J.J. Grefenstette, “The evolution of strategies for multiagent environments,”
*Adaptive Behaviour*, vol. 1, no. 1, pp. 65-90, 1992.Google Scholar - 27.J.J. Grefenstette, “Lamarckian learning in multiagent environments,” in
*Proceedings of 4th International Conference of*Scaling Up Inductive Logic Programming 197*Genetic Algorithms*, Morgan Kaufmann, pp. 303-310, 1992.Google Scholar - 28.S. Augier, G. Venturini, and Y. Kodratoff, “Learning first order logic rules with a genetic algorithm,” in
*Proceedings of the First International Conference on Knowledge Discovery and Data Mining*, 1995, pp. 21-26.Google Scholar - 29.A. Giordana, L. Saitta, and F. Zini, “Learning disjunctive concepts by means of genetic algorithms,” in
*Proceedings of the 11th International Conference on Machine Learning*, 1994, pp. 96-104.Google Scholar - 30.F. Neri, “First order logic concept learning by means of a distributed genetic algorithm,” Ph.D. dissertation, Università of Torino, Dipartimento di Informatica, 1997.Google Scholar
- 31.L. Davis,
*Handbook of Genetic Algorithms*, Van Nostrand Reinhold: New York, 1991.Google Scholar - 32.W.E. Hart and R.K. Belew, “Optimization with genetic algorithm hybrids that use local search,” in
*Adaptive Individuals in Evolving Populations: Models and Algorithms*, edited by R.K. Belew and M. Mitchell, SFI Studies in the Sciences of Complexity, vol. 26, Chap. 27, pp. 483-496, 1996.Google Scholar - 33.D.E. Goldberg,
*Genetic Algorithms in Search, Optimization, and Machine Learning*, Addison-Wesley: Reading, MA, 1989.Google Scholar - 34.S. Muggleton, “Inductive logic programming,”
*New Generation Computing*, vol. 8, no. 4, pp. 295-318, 1991.Google Scholar - 35.S. Muggleton, “Inductive logic programming: Derivations, successes and shortcomings,”
*SIGART Bulletin*, vol. 5, no. 1, pp. 5-11, 1994.Google Scholar - 36.S. Muggleton, “Inverse entailment and progol,”
*New Generation Computing*, vol. 13, 1995.Google Scholar - 37.U. Pompe, M. Kovačič, and I. Kononenko, “SFOIL: Stochastic approach to inductive logic programming,” in
*Proceedings of the 2nd Electrotechnical and Computer Science Conference*, Portoroz, Slovenija, p. 31, 1993.Google Scholar - 38.M. Kovačič, “Stochastic inductive logic programming,” Ph.D. Thesis, University of Ljubljana, Slovenia, 1994.Google Scholar
- 39.L. Breiman, “Bagging predictors,” Technical Report No. 421, University of California, Berkeley, Department of Statistics, 1994.Google Scholar
- 40.S.H. Muggleton, M. Bain, J. Hayes-Michie, and D. Michie, “An experimental comparison of human and machine learning formalisms,” in
*Proc. Sixth International Workshop on Machine Learning*, Morgan Kaufmann, San Mateo, CA, 1989. pp. 113-118.Google Scholar - 41.J. Cussens, “Part-of-speech tagging using progol,” in
*Inductive Logic Programming: Proceedings of the 7th InternationalWorkshop (ILP-97)*, 1997, volume 1297 of Lecture Notes in Artificial Intelligence, Springer pp. 93-108.Google Scholar - 42.M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz, “Building a large annotated corpus of English: the Penn treebank,”
*Computational Linguistics*, vol. 19, 1993.Google Scholar - 43.J. Cussens, “Part-of-speech disambiguation using ILP,” Technical Report PRG-TR-25-96, Oxford University Computing laboratory, 1996.Google Scholar
- 44.J.R. Quinlan, “Discovering rules from large collections of examples: A case study,” in
*Expert Systems in the Micro-electronic Age*, edited by D. Michie, Edinburgh University Press, Edinburgh, pp. 168-201, 1979.Google Scholar - 45.J.W. Lloyd,
*Foundations of Logic Programming*, Springer-Verlag: Berlin, 1984.Google Scholar - 46.T.G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,”
*Neural Computation*, vol. 10, no. 7, pp. 1895-1924, 1998.Google Scholar - 47.D. Fogel and C. deSilva, eds.
*Proceedings of the IEEE International Conference on Evolutionary Computation*, Perth,Western Australia, Nov. 1995, IEEE Computer Society Press.Google Scholar