Evaluating Hybrid Versus Data-Driven Coreference Resolution

  • Iris Hendrickx
  • Veronique Hoste
  • Walter Daelemans
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4410)


In this paper, we present a systematic evaluation of a hybrid approach of combined rule-based filtering and machine learning to Dutch coreference resolution. Through the application of a selection of linguistically-motivated negative and positive filters, which we apply in isolation and combined, we study the effect of these filters on precision and recall using two different learning techniques: memory-based learning and maximum entropy modeling. Our results show that by using the hybrid approach, we can reduce up to 92 % of the training material without performance loss. We also show that the filters improve the overall precision of the classifiers leading to higher F-scores on the test set.


Noun Phrase Positive Instance Negative Instance Maximum Entropy Modeling Negative Label 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Cardie, C., Wagstaff, K.: Noun phrase coreference as clustering. In: Proceedings of the 1999 joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pp. 82–89 (1999)Google Scholar
  2. 2.
    Quinlan, J.: C4.5: Programs for machine learning. Morgan Kaufmann, San Mateo (1993)Google Scholar
  3. 3.
    Aone, C., Bennett, S.: Evaluating automated and manual acquisition of anaphora resolution strategies. In: Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL-1995), pp. 122–129 (1995)Google Scholar
  4. 4.
    McCarthy, J.: A Trainable Approach to Coreference Resolution for Information Extraction. PhD thesis, Department of Computer Science, University of Massachusetts, Amherst MA (1996)Google Scholar
  5. 5.
    Soon, W., Ng, H., Lim, D.: A machine learning approach to coreference resolution of noun phrases. Computational Linguistics 27, 521–544 (2001)CrossRefGoogle Scholar
  6. 6.
    Cohen, W.W.: Fast effective rule induction. In: Proceedings of the 12th International Conference on Machine Learning (ICML-1995), pp. 115–123 (1995)Google Scholar
  7. 7.
    Ng, V., Cardie, C.: Combining sample selection and error-driven pruning for machine learning of coreference rules. In: Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), pp. 55–62 (2002)Google Scholar
  8. 8.
    Strube, M., Rapp, S., Müller, C.: The influence of minimum edit distance on reference resolution. In: Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP-2002), pp. 312–319 (2002)Google Scholar
  9. 9.
    Yang, X., et al.: Coreference resolution using competition learning approach. In: Proceedings of the 41th Annual Meeting of the Association for Compuatiational Linguistics (ACL-03), pp. 176–183 (2003)Google Scholar
  10. 10.
    Harabagiu, S., Bunescu, R., Maiorano, S.: Text and knowledge mining for coreference resolution. In: Proceedings of the 2nd Meeting of the North American Chapter of the Association of Computational Linguistics (NAACL-2001), pp. 55–62 (2001)Google Scholar
  11. 11.
    Uryupina, O.: Linguistically motivated sample selection for coreference resolution. In: Proceedings of DAARC-2004 (2004)Google Scholar
  12. 12.
    Hoste, V.: Optimization Issues in Machine Learning of Coreference Resolution. PhD thesis, Antwerp University (2005)Google Scholar
  13. 13.
    MUC-7: Muc-7 coreference task definition, version 3.0. In: Proceedings of the Seventh Message Understanding Conference (MUC-7) (1998)Google Scholar
  14. 14.
    Davies, S., et al.: Annotating coreference in dialogues: Proposal for a scheme for mate (1998),
  15. 15.
    van Deemter, K., Kibble, R.: On coreferring: Coreference in muc and related annotation schemes. Computational Linguistics 26, 629–637 (2000)CrossRefGoogle Scholar
  16. 16.
    Daelemans, W., et al.: Mbt: A memory-based part of speech tagger generator. In: Proceedings of the 4th ACL/SIGDAT Workshop on Very Large Corpora, pp. 14–27 (1996)Google Scholar
  17. 17.
    Tjong Kim Sang, E., Daelemans, W., Höthker, A.: Reduction of dutch sentences for automatic subtitling. In: Computational Linguistics in the Netherlands 2003, Selected Papers from the Fourteenth CLIN Meeting, pp. 109–123 (2004)Google Scholar
  18. 18.
    Daelemans, W., van den Bosch, A.: Memory-based Language Processing. Cambridge University Press, Cambridge (2005)Google Scholar
  19. 19.
    Guiasu, S., Shenitzer, A.: The principle of maximum entropy. The Mathematical Intelligencer 7 (1985)Google Scholar
  20. 20.
    Berger, A., Della Pietra, S., Della Pietra, V.: Maximum Entropy Approach to Natural Language Processing. Computational linguistics 22 22 (1996)Google Scholar
  21. 21.
    Le, Z.: Maximum Entropy Modeling Toolkit for Python and C++ (version 20041229). Natural Language Processing Lab, Northeastern University, China (2004)Google Scholar
  22. 22.
    van den Bosch, A.: Wrapped progressive sampling search for optimizing learning algorithm parameters. In: Proceedings of the 16th Belgian-Dutch Conference on Artificial Intelligence, pp. 219–226 (2004)Google Scholar
  23. 23.
    Kohavi, R., John, G.H.: Wrappers for feature subset selection. Artificial Intelligence 97, 273–323 (1997)zbMATHCrossRefGoogle Scholar
  24. 24.
    Provost, F., Jensen, D., Oates, T.: Efficient progressive sampling. In: Proceedings of the Fifth International Conference on Knowledge Discovery and Data Mining, pp. 23–32 (1999)Google Scholar
  25. 25.
    Vilain, M., et al.: A model-theoretic coreference scoring scheme. In: Proceedings of the Sixth Message Understanding Conference (MUC-6), pp. 45–52 (1995)Google Scholar
  26. 26.
    Connolly, D., Burger, J., Day, D.: A machine learning approach to anaphoric reference. In: Proceedings of the International Conference on ‘New Methods in Language Processing’ (1994)Google Scholar
  27. 27.
    Daelemans, W., van den Bosch, A., Zavrel, J.: Forgetting exceptions is harmful in language learning. Machine Learning 34, 11–41 (1999)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer Berlin Heidelberg 2007

Authors and Affiliations

  • Iris Hendrickx
    • 1
  • Veronique Hoste
    • 2
  • Walter Daelemans
    • 1
  1. 1.University of Antwerp, CNTS - Language Technology Group, Universiteitsplein 1, AntwerpBelgium
  2. 2.University College Ghent, LT3 - Language and Translation Technology Team 

Personalised recommendations