Abstract
It is well-known that heuristic search in ILP is prone to plateau phenomena. An explanation can be given after the work of Giordana and Saitta: the ILP covering test is NP-complete and therefore exhibits a sharp phase transition in its coverage probability. As the heuristic value of a hypothesis depends on the number of covered examples, the regions “yes” and “no” represent plateaus that need to be crossed during search without an informative heuristic value. Several subsequent works have extensively studied this finding by running several learning algorithms on a large set of artificially generated problems and argued that the occurrence of this phase transition dooms every learning algorithm to fail to identify the target concept. We note however that only generate-and-test learning algorithms have been applied and that this conclusion has to be qualified in the case of data-driven learning algorithms. Mostly building on the pioneering work of Winston on near-miss examples, we show that, on the same set of problems, a top-down data-driven strategy can cross any plateau if near-misses are supplied in the training set, whereas they do not change the plateau profile and do not guide a generate-and-test strategy. We conclude that the location of the target concept with respect to the phase transition alone is not a reliable indication of the learning problem difficulty as previously thought.
Article PDF
Similar content being viewed by others
References
Ales-Bianchetti, J., Rouveirol, C., & Sebag, M. (2002). Constraint-based learning of long relational concepts. In M. Kaufmann (Ed.), Nineteenth international conference on machine learning (ICML-2002) (pp. 35–42), Sydney, NSW, Australia. Los Altos: Kaufmann.
Alphonse, E. (2004). Macro-operators revisited in inductive logic programming. In Proceedings of the conference on inductive logic programming (pp. 8–25), Porto, Portugal. Berlin: Springer.
Alphonse, E., & Rouveirol, C. (2006). Extension of the top-down data-driven strategy to ILP. In Proceedings of the conference on inductive logic programming, Santiago de Compostela, Spain. Berlin: Springer.
Blockeel, H., & Raedt, L. D. (1997). Lookahead and discretization in ILP. In N. Lavrač & S. Džeroski (Eds.), Proceedings of the conference on inductive logic programming (Vol. 1297, pp. 77–84), 17–20 September 1997. Berlin: Springer.
Blockeel, H., & Raedt, L. D. (1998). Top-down induction of first order decision trees. Artificial Intelligence, 101, 285–297.
Botta, M., Giordana, A., Saitta, L., & Sebag, M. (2003). Relational learning as search in a critical region. Journal of Machine Learning Research, 4, 431–463.
Cheeseman, P., Kanefsky, B., & Taylor, W. (1991). Where the really hard problems are. In R. Myopoulos (Ed.), Proceedings of the 12th international joint conference on artificial intelligence (pp. 331–340), Sydney, Australia, August 1991. Los Altos: Kaufmann.
Clark, P., & Niblett, T. (1989). The CN2 induction algorithm. Machine Learning, 3, 261–283.
Cohen, W. W. (1993). Learnability of restricted logic programs. In S. Muggleton (Ed.), Proceedings of the conference on inductive logic programming (pp. 41–72). Szeged: J. Stefan Institute.
Cohen, W. W. (1995). Fast effective rule induction. In Proceedings of the 12th international conference on machine learning (pp. 115–123), Tahoe City, CA. Los Altos: Kaufmann.
Fürnkranz, J., & Flach, P. (2005). Roc’n’ rule learning-towards a better understanding of covering algorithms. Machine Learning, 58, 39–77.
Giordana, A., & Saitta, L. (2000). Phase transitions in learning relations. Machine Learning, 41, 217–25.
Giordana, A., Saitta, L., Sebag, M., & Botta, M. (2000). Analyzing relational learning in the phase transition framework. In 17th international conference on machine learning (pp. 311–318), Stanford, CA, USA. Los Altos: Kaufmann.
Gottlob, G., Leone, N., & Scarcello, F. (1997). On the complexity of some inductive logic programming problems. In N. Lavrač & S. Džeroski (Eds.), Proceedings of the 7th international workshop on inductive logic programming (Vol. 1297, pp. 17–32). Berlin: Springer.
Haussler, D. (1989). Learning conjunctive concepts in structural domains. Machine Learning, 4(1), 7–40.
Hayes-Roth, F., & McDermott, J. (1977). Knowledge acquisition from structural descriptions. In R. Reddy (Ed.), Proceedings of the 5th international joint conference on artificial intelligence (pp. 356–362). Cambridge: Kaufmann.
Kearns, M. J., & Vazirani, U. V. (1994). An introduction to computational learning theory. Cambridge: MIT Press.
Korf, R. E. (1985a). Depth-first iterative-deepening: an optimal admissible tree search. Artificial Intelligence, 27(1), 97–109.
Korf, R. E. (1985b). Macro-operators: a weak method for learning. Artificial Intelligence, 26, 35–77.
Laird, P. D. (1986). Inductive inference by refinement. In T. Kehler & S. Rosenschein (Eds.), Proceedings of the 5th national conference on artificial intelligence (Vol. 1, pp. 472–476), August 1986. Los Altos: Kaufmann.
Michalski, R. S. (1983). A theory and methodology of inductive learning. Palo Alto: Kaufmann.
Mitchell, T. M. (1982). Generalization as search. Artificial Intelligence, 18, 203–226.
Muggleton, S. (1995). Inverse entailment and PROGOL. New Generation Computing, 13, 245–286.
Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs: Prentice-Hall.
Peña-Castillo, L., & Wrobel, S. (2002). On the stability of example-driven learning systems: a case study in multirelational learning. In MICAI 2002: advances in artificial intelligence (pp. 321–330). Berlin: Springer.
Pearl, J. (1985). Heuristics. Reading: Addison-Wesley.
Plotkin, G. (1970). A note on inductive generalization. In Machine intelligence (pp. 153–163). Edinburgh: Edinburgh University Press.
Quinlan, J. R. (1986). Induction of decision trees. Machine Learning, 1(1), 81–106.
Quinlan, J. R. (1990). Learning logical definitions from relations. Machine Learning, 5(3), 239–266.
Quinlan, J. R. (1991). Determining literals in inductive logic programming. In Proceedings of the 12th international joint conference on artificial intelligence (pp. 746–750), Sydney, New South Wales, Australia. Berlin: Springer.
Quinlan, J. R. (1993). C4.5: programs for machine learning. San Mateo: Kaufmann.
Richards, B., & Mooney, R. (1992). Learning relations by pathfinding. In Proceedings of the tenth national conference on artificial intelligence (pp. 723–738). San Jose: AAAI Press/MIT Press.
Russell, S., & Norvig, P. (1995). Artificial intelligence: a modern approach. Englewood Cliffs: Prentice Hall.
Serra, A., Giordana, A., & Saitta, L. (2001). Learning on the phase transition edge. In B. Nebel (Ed.), Proceedings. of the 7th int. conference. on artificial intelligence (IJCAI-01) (pp. 921–926), Seattle, Washington, USA. Los Altos: Kaufmann.
Shapiro, E. (1983). Algorithmic program debugging. Cambridge: MIT Press.
Silverstein, G., & Pazzani, M. J. (1991). Relational cliches: constraining constructive induction during relational learning. In L. Birnbaum & G. Collins (Eds.), Proceedings of the 8th international workshop on machine learning (pp. 203–207), University of California, Irvine. Los Altos: Kaufmann.
Smith, B. D., & Rosenbloom, P. S. (1990). Incremental non-backtracking focusing: a polynomially bounded generalization algorithm for version spaces. In Proceedings of the 8th national conference on artificial intelligence (pp. 848–853). Boston: AAAI Press/MIT Press.
Srinivasan, A. (1999). A learning engine for proposing hypotheses (Aleph). http://web.comlab.ox.ac.uk/oucl/research/areas/machlearn/Aleph.
van der Laag, P., & Nienhuys-Cheng, S. H. (1994). A note on ideal refinement operators in ILP. In S. Wrobel (Ed.), Proceedings of the 4th international workshop on inductive logic programming (Vol. 237, pp. 247–262). Bad Honnef/Bonn: Gesellschaft für Mathematik und Datenverarbeitung MBH.
Winston, P. H. (1975). Learning structural descriptions from examples. In P. H. Winston (Ed.), The psychology of computer vision (pp. 157–209). New York: McGraw-Hill.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editors: Stephen Muggleton, Ramon Otero, Simon Colton.
Rights and permissions
About this article
Cite this article
Alphonse, E., Osmani, A. On the connection between the phase transition of the covering test and the learning success rate in ILP. Mach Learn 70, 135–150 (2008). https://doi.org/10.1007/s10994-007-5031-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-007-5031-9