Learning Ensembles of First-Order Clauses for Recall-Precision Curves: A Case Study in Biomedical Information Extraction

  • Mark Goadrich
  • Louis Oliphant
  • Jude Shavlik
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3194)


Many domains in the field of Inductive Logic Programming (ILP) involve highly unbalanced data. Our research has focused on Information Extraction (IE), a task that typically involves many more negative examples than positive examples. IE is the process of finding facts in unstructured text, such as biomedical journals, and putting those facts in an organized system. In particular, we have focused on learning to recognize instances of the protein-localization relationship in Medline abstracts. We view the problem as a machine-learning task: given positive and negative extractions from a training corpus of abstracts, learn a logical theory that performs well on a held-aside testing set. A common way to measure performance in these domains is to use precision and recall instead of simply using accuracy. We propose Gleaner, a randomized search method which collects good clauses from a broad spectrum of points along the recall dimension in recall-precision curves and employs an “at least N of these M clauses” thresholding method to combine the selected clauses. We compare Gleaner to ensembles of standard Aleph theories and find that Gleaner produces comparable testset results in a fraction of the training time needed for ensembles.


Information Extraction Learn Ensemble Inductive Logic Programming Name Entity Recognition Candidate Clause 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aitken, S.: Learning Information Extraction Rules: An Inductive Logic Programming Approach. In: van Harmelen, F. (ed.) Proceedings of the 15th European Conference on Artificial Intelligence, Amsterdam (2002)Google Scholar
  2. 2.
    Ben Taskar, M.-F.W., Abbeeland, P., Koller, D.: Label and Link Prediction in Relational Data. In: IJCAI Workshop on Learning Statistical Models from Relational Data (2003)Google Scholar
  3. 3.
    Blaschke, C., Hirschman, L., Valencia, A.: Information Extraction in Molecular Biology. Briefings in Bioinformatics 3(2), 154–165 (2002)CrossRefGoogle Scholar
  4. 4.
    Blaschke, C., Valencia, A.: Can Bibliographic Pointers for Known Biological Data be Found Automatically? Protein Interactions as a Case Study. Comparative and Functional Genomics 2, 196–206 (2001)CrossRefGoogle Scholar
  5. 5.
    Bradley, A.: The Use of the Area Under the ROC Curve in the Evaluation of Machine Learning Algorithms. Pattern Recognition 30(7), 1145–1159 (1997)CrossRefGoogle Scholar
  6. 6.
    Breiman, L.: Bagging Predictors. Machine Learning 24(2), 123–140 (1996)zbMATHMathSciNetGoogle Scholar
  7. 7.
    Bunescu, R., Ge, R., Kate, R., Marcotte, E., Mooney, R., Ramani, A., Wong, Y.: Comparative Experiments on Learning Information Extractors for Proteins and their Interactions. Journal of Artificial Intelligence in Medicine (2004)Google Scholar
  8. 8.
    Califf, M., Mooney, R.: Relational Learning of Pattern-Match Rules for Information Extraction. In: Working Notes of AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, pp. 6–11. AAAI Press, Menlo Park (1998)Google Scholar
  9. 9.
    Cortes, C., Mohri, M.: AUC Optimization vs. Error Rate Minimization. In: Neural Information Processing Systems NIPS2003 (2003)Google Scholar
  10. 10.
    Craven, M., Slattery, S.: Relational Learning with Statistical Predicate Invention: Better Models for Hypertext. Machine Learning 43(1/2), 97–119 (2001)zbMATHCrossRefGoogle Scholar
  11. 11.
    de Castro Dutra, I., Page, D., Costa, V.S., Shavlik, J.: An Empirical Evaluation of Bagging in Inductive Logic Programming. In: Twelfth International Conference on Inductive Logic Programming, Sydney, Australia, pp. 48–65 (2002)Google Scholar
  12. 12.
    Dietterich, T.: Machine-Learning Research: Four Current Directions. The AI Magazine 18(4), 97–136 (1998)Google Scholar
  13. 13.
    Eliassi-Rad, T., Shavlik, J.: A Theory-Refinement Approach to Information Extraction. In: Proceedings of the 18th International Conference on Machine Learning (2001)Google Scholar
  14. 14.
    Freitag, D., Kushmerick, N.: Boosted Wrapper Induction. In: AAAI/IAAI, pp. 577–583 (2000)Google Scholar
  15. 15.
    Hoche, S., Wrobel, S.: Relational Learning Using Constrained Confidence-Rated Boosting. In: 11th International Conference on Inductive Logic Programming, Strasbourg, France (2001)Google Scholar
  16. 16.
    Hu, Z.: Guidelines for Protein Name Tagging. Technical report, Georgetown University (2003)Google Scholar
  17. 17.
    Kauchak, D., Smarr, J., Elkan, C.: Sources of Success for Boosted Wrapper Induction. Journal of Machine Learning Research 5, 499–527 (2004)MathSciNetGoogle Scholar
  18. 18.
    Lavrac, N., Zelezny, F., Flach, P.: RSD: Relational Subgroup Discovery through First-order Feature Construction. In: Matwin, S., Sammut, C. (eds.) ILP 2002. LNCS (LNAI), vol. 2583, pp. 149–165. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  19. 19.
    Manning, C., Schutze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  20. 20.
    Michalski, R., Larson, J.: Inductive Inference of VL Decision Rules. In: Proceedings of the Workshop in Pattern-Directed Inference Systems (May 1977)Google Scholar
  21. 21.
    Popescul, L., Ungar, S.: Lawrence, and D. Pennock. Statistical Relational Learning for Document Mining. In: IEEE International Conference on Data Mining, ICDM 2003 (2003)Google Scholar
  22. 22.
    Porter, M.: An Algorithm for Suffix Stripping. Program 14(3), 130–137 (1980)Google Scholar
  23. 23.
    Ray, S., Craven, M.: Representing Sentence Structure in Hidden Markov Models for Information Extraction. In: Proceedings of the 17th International Joint Conference on Artificial Intelligence, IJCAI 2001 (2001)Google Scholar
  24. 24.
    Riloff, E.: The Sundance Sentence Analyzer (1998),
  25. 25.
    Rindflesch, T., Tanabe, T., Weinstein, L., Hunter, J.: Edgar: Extraction of drugs, genes and relations from the biomedical literature. In: Proceedings of the Pacific Symposium on Biocomputing (2000)Google Scholar
  26. 26.
    Shatkay, H., Feldman, R.: Mining the Biomedical Literature in the Genomic Era: An Overview. Journal of Computational Biology 10(6), 821–855 (2003)CrossRefGoogle Scholar
  27. 27.
    Srinivasan, A.: The Aleph Manual Version 4 (2003),
  28. 28.
    Srinivasan, A., King, R.: Feature Construction with Inductive Logic Programming: A Study of Quantitative Predictions of Biological Activity Aided by Structural Attributes. In: Muggleton, S. (ed.) Proceedings of the 6th International Workshop on Inductive Logic Programming, Stockholm University, Royal Institute of Technology, pp. 352–367 (1996)Google Scholar
  29. 29.
    Srinivasan, A., Muggleton, S., Sternberg, M., King, R.: Theories for Mutagenicity: A Study in First-Order and Feature-Based Induction. Artificial Intelligence 85(1- 2), 277–299 (1996)CrossRefGoogle Scholar
  30. 30.
    Tang, L., Mooney, R., Melville, P.: Scaling up ILP to Large Examples: Results on Link Discovery for Counter-Terrorism. In: KDD Workshop on Multi-Relational Data Mining (2003)Google Scholar
  31. 31.
    Temkin, J., Gilder, M.: Extraction of Protein Interaction Information From Unstructured Text Using a Context-Free Grammar. Bioinformatics 19(16), 2046–2053 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Mark Goadrich
    • 1
  • Louis Oliphant
    • 1
  • Jude Shavlik
    • 1
  1. 1.Department of Biostatistics and Medical Informatics and Department of Computer SciencesUniversity of Wisconsin-MadisonUSA

Personalised recommendations