Abstract
In this chapter, we present a system that enhances the representational capabilities of decision and regression tree learning by extending it to first-order logic, i.e., relational representations as commonly used in Inductive Logic Programming. We describe an algorithm named Structural Classification and Regression Trees (S-Cart), which is capable of inducing first-order trees for both classification and regression problems, i.e., for the prediction of either discrete classes or numerical values. We arrive at this algorithm by a strategy called upgrading — we start from a propositional induction algorithm and turn it into a relational learner by devising suitable extensions of the representation language and the associated algorithms. In particular, we have upgraded Cart, the classical method for learning classification and regression trees, to handle relational examples and background knowledge. The system constructs a tree containing a literal (an atomic formula or its negation) or a conjunction of literals in each node, and assigns either a discrete class or a numerical value to each leaf. In addition, we have extended the Cart methodology by adding linear regression models to the leaves of the trees; this does not have a counterpart in Cart, but was inspired by its approach to pruning. The regression variant of S-Cart is one of the few systems applicable to Relational Regression problems. Experiments in several real-world domains demonstrate that the approach is useful and competitive with existing methods, indicating that the advantage of relatively small and comprehensible models does not come at the expense of predictive accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
P. Auer, W. Maass, and R. Holte. Theory and applications of agnostic PAC-learning with small decision trees. In Proceedings of the Twelfth International Conference on Machine Learning, pages 21–29. Morgan Kaufmann, San Francisco, CA, 1995.
H. Blockeel and L. De Raedt. Top-down induction of first order logical decision trees. Artificial Intelligence, 101(1–2):285–297, 1998.
H. Boström. Covering vs. divide-and-conquer for top-down induction of logic programs. In Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pages 1194–1200. Morgan Kaufmann, San Francisco, CA, 1995.
L. Breiman, J. Friedman, R. Olshen, and C. Stone. Classification and Regression Trees. Wadsworth, Belmont, CA, 1984.
W. Buntine. A theory of learning classification rules. PhD thesis, School of Computing Science, University of Technology, Sydney, Australia, 1992.
P. Clark and T. Niblett. The CN2 induction algorithm. Machine Learning, 3:261–283, 1989.
W. Cohen. Grammatically biased learning: Learning logic programs using an explicit antecedent description language. Artificial Intelligence, 68(2):303–366, 1994.
J. Cussens and S. Džeroski, editors. Learning Language in Logic. Springer, Berlin, 2000.
L. Dehaspe and H. Toivonen. Discovery of frequent datalog patterns. Data Mining and Knowledge Discovery, 3(1):7–36, 1999.
L. De Raedt and W. Van Laer. Inductive constraint logic. In Proceedings of the Fifth Workshop on Algorithmic Learning Theory, pages 80–94. Springer, Berlin, 1995.
T. Dietterich, M. Kearns, and Y. Mansour. Applying the weak learning framework to understand and improve C4.5. In Proceedings of the Thirteenth International Conference on Machine Learning, pages 96–104. Morgan Kaufmann, San Francisco, CA, 1996.
S. Džeroski, H. Blockeel, B. Kompare, S. Kramer, B. Pfahringer, and W. Van Laer. Experiments in predicting biodegradability. In Proceedings of the Ninth International Workshop on Inductive Logic Programming, pages 80–91. Springer, Berlin, 1999.
S. Džeroski, L. Todorovski, and T. Urbančič. Handling real numbers in inductive logic programming: A step towards better behavioural clones. Proceedings of the Eighth European Conference on Machine Learning, pages 283–286. Springer, Berlin, 1995.
R. Freund and W. Wilson. Regression analysis. Academic Press, London, 1998.
D. Heath, S. Kasif, and S. Salzberg. Induction of oblique decision trees. In Proceedings of the Thirteenth International Joint Conference on Artificial Intelligence, pages 1002–1007. Morgan Kaufmann, San Francisco, CA, 1993.
A. Karalić. Employing linear regression in regression tree leaves. In Proceedings of the Tenth European Conference on Artificial Intelligence, pages 440–441. John Wiley and Sons, 1992.
A. Karalić and I. Bratko. First order regression. Machine Learning, 26(2/3): 147–176, 1997.
M. Kearns. Boosting theory towards practice: Recent developments in decision tree induction and the weak learning framework. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 1337–1339. AAAI Press, Menlo Park, CA, 1996.
M. Kearns and Y. Mansour. A fast, bottom-up decision tree pruning algorithm with near-optimal generalization. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 269–277. Morgan Kaufmann, San Francisco, CA, 1998.
R. King and A. Srinivasan. Prediction of rodent carcinogenicity bioassays from molecular structure using Inductive Logic Programming. Environmental Health Perspectives, 104(5):1031–1040, 1996.
S. Kramer. Structural regression trees. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 812–819. AAAI Press, Menlo Park, CA, 1996.
S. Kramer. Relational learning vs. propositionalization: investigations in inductive logic programming and propositional machine learning. PhD thesis, Vienna University of Technology, Vienna, Austria, 1999.
N. Lavrač and S. Džeroski. Inductive Logic Programming: Techniques and Applications. Ellis Horwood, Chichester, 1994. Freely available at http://www-ai.ijs.si/SasoDzeroski/ILPBook/.
M. Manago. Knowledge-intensive induction. In Proceedings of the Sixth International Workshop on Machine Learning, pages 151–155. Morgan Kaufman, San Francisco, CA, 1989.
G. Pagallo and D. Haussler. Boolean feature discovery in empirical learning. Machine Learning, 5 (1):71–100, 1990.
H. Pirker and S. Kramer. Listening to lists: Studying durational phenomena in enumerations. In Proceedings of the Fourteenth International Conference of Phonetic Sciences, pages 273–276. Department of Linguistics, University of California, Berkeley, CA, 1999.
J. Quinlan. Learning with continuous classes. Proceedings of the Second Australian Conference on Artificial Intelligence, pages 343–348. World Scientific, Singapore, 1992.
J. Quinlan. C4.5: programs for machine learning. Morgan Kaufmann, San Francisco, CA, 1993.
J. Quinlan. Bagging, boosting and C4.5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 725–730. AAAI Press, Menlo Park, CA, 1996.
G. Silverstein and M. Pazzani. Relational clichés: Constraining constructive induction during relational learning. In Proceedings of the Eighth International Workshop on Machine Learning, pages 203–207. Morgan Kaufmann, San Francisco, CA, 1991.
A. Srinivasan, S. Muggleton, R. King, and M. Sternberg. Theories for mutagenicity: a study of first-order and feature based induction. Artificial Intelligence, 85 (1–2):277–299, 1996.
Y. Wang and I. Witten. Inducing model trees for continuous classes. In Poster Papers of the Ninth European Conference on Machine Learning, pages 128– 137. Faculty of Informatics and Statistics, University of Economics, Prague, 1997.
L. Watanabe and L. Rendell. Learning structural decision trees from examples. In Proceedings of the Twelfth International Joint Conference on Artificial Intelligence, pages 770–776. Morgan Kaufmann, San Francisco, CA, 1991.
S. Weiss and N. Indurkhya. Rule-based machine learning methods for functional prediction. Journal of Artificial Intelligence Research, 3:383–403, 1995.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Kramer, S., Widmer, G. (2001). Inducing Classification and Regression Trees in First Order Logic. In: Džeroski, S., Lavrač, N. (eds) Relational Data Mining. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-04599-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-662-04599-2_6
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-07604-6
Online ISBN: 978-3-662-04599-2
eBook Packages: Springer Book Archive