Language Resources and Evaluation

, Volume 44, Issue 4, pp 315–345 | Cite as

AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan

  • Marta RecasensEmail author
  • M. Antònia Martí


This article describes the enrichment of the AnCora corpora of Spanish and Catalan (400 k each) with coreference links between pronouns (including elliptical subjects and clitics), full noun phrases (including proper nouns), and discourse segments. The coding scheme distinguishes between identity links, predicative relations, and discourse deixis. Inter-annotator agreement on the link types is 85–89% above chance, and we provide an analysis of the sources of disagreement. The resulting corpora make it possible to train and test learning-based algorithms for automatic coreference resolution, as well as to carry out bottom-up linguistic descriptions of coreference relations as they occur in real data.


Coreference Anaphora Corpus annotation Annotation scheme Reliability study 



This work was supported by the FPU Grant (AP2006-00994) from the Spanish Ministry of Education and Science, and the Lang2World (TIN2006-15265-C06-06) and Ancora-Nom (FFI2008-02691-E/FILO) projects. Special thanks to Mariona Taulé for her invaluable advice, Manuel Bertran for customising the AnCoraPipe annotation tool, and the annotators who participated in the development of AnCora-CO and the reliability study: Oriol Borrega, Isabel Briz, Irene Carbó, Sandra García, Iago González, Esther López, Jesús Martínez, Laura Muñoz, Montse Nofre, Lourdes Puiggròs, Lente Van Leeuwen, and Rita Zaragoza. We are indebted to three anonymous reviewers for their comments on earlier versions of this work.


  1. Ariel, M. (1988). Referring and accessibility. Journal of Linguistics 24(1), 65–87.CrossRefGoogle Scholar
  2. Artstein, R., & Poesio, M. (2005). Bias decreases in proportion to the number of annotators. In Proceedings of FG-MoL 2005 (pp. 141–150). Edinburgh.Google Scholar
  3. Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4), 555–596.CrossRefGoogle Scholar
  4. Baldwin, B. (1997). CogNIAC: High precision coreference with limited knowledge and linguistic resources. In Proceedings of the ACL-EACL 1997 workshop on operational factors in practical, robust anaphor resolution for unrestricted texts (pp. 38–45). Madrid.Google Scholar
  5. Bertran, M., Borrega, O., Recasens, M., & Soriano, B. (2008). AnCoraPipe: A tool for multilevel annotation. Procesamiento del Lenguaje Natural, 41, 291–292.Google Scholar
  6. Blackwell, S. (2003). Implicatures in discourse: The case of Spanish NP anaphora. Amsterdam: John Benjamins.Google Scholar
  7. Borrega, O., Taulé, M., & Martí, M. A. (2007). What do we mean when we talk about named entities?. In Proceedings of the 4th corpus linguistics conference, Birmingham.Google Scholar
  8. Bosque, I., & Demonte, V. (Eds.) (1999). Gramática descriptiva de la lengua española. Madrid: Real Academia Española/Espasa Calpe.Google Scholar
  9. Carletta, J. (1996). Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2), 249–254.Google Scholar
  10. Clark, H. H. (1977). Bridging. In P. Johnson-Laird, & P. C. Wason (Eds.), Thinking: Readings in cognitive science (pp. 411–420). Cambridge: Cambridge University Press.Google Scholar
  11. Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., & Weischedel, R. (2004). The Automatic Content Extraction (ACE) program—Tasks, data, and evaluation. In Proceedings of LREC 2004 (pp. 837–840). Lisbon.Google Scholar
  12. Eckert, M., & Strube, M. (2000). Dialogue acts, synchronising units and anaphora resolution. Journal of Semantics, 17(1), 51–89.CrossRefGoogle Scholar
  13. Fraurud, K. (1990). Definiteness and the processing of NPs in natural discourse. Journal of Semantics, 7, 395–433.CrossRefGoogle Scholar
  14. Gundel, J., Hedberg, N., & Zacharski, R. (1993). Cognitive status and the form of referring expressions in discourse. Language, 69(2), 274–307.CrossRefGoogle Scholar
  15. Halliday, M. A., & Hasan, R. (1976). Cohesion in English. London: Longman.Google Scholar
  16. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89.Google Scholar
  17. Hinrichs, E., Kübler, S., Naumann, K., Telljohann, H., & Trushkina, J. (2004). Recent developments in linguistic annotations of the TüBa-D/Z treebank. In Proceedings of TLT 2004, Tübingen.Google Scholar
  18. Hirschman, L., & Chinchor, N. (1997). MUC-7 coreference task definition—Version 3.0.Google Scholar
  19. Hobbs, J. R. (1978). Resolving pronoun references. Lingua, 44, 311–338.CrossRefGoogle Scholar
  20. Hoste, V. (2005). Optimization issues in machine learning of coreference resolution. PhD Thesis, University of Antwerp.Google Scholar
  21. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006). OntoNotes: The 90% solution. In Proceedings of HLT-NAACL 2006 (pp. 57–60). New York.Google Scholar
  22. Ide, N. (2000). Searching annotated language resources in XML: A statement of the problem. In Proceedings of the ACM SIGIR 2000 workshop on XML and information retrieval, Athens.Google Scholar
  23. Kilgarriff, A. (1999). 95% replicability for manual word sense tagging. In Proceedings of EACL 1999 (pp. 277–278). Bergen.Google Scholar
  24. Kripke, S. (1977). Speaker’s reference and semantic reference. Midwest Studies in Philosophy, 2, 255–276.CrossRefGoogle Scholar
  25. Krippendorff, K. (2004 [1980]). Content Analysis: An Introduction to its Methodology (2nd ed.). Thousand Oaks, CA: Sage. Chapter 11.Google Scholar
  26. Kučová, L., & Hajičová, E. (2004). Coreferential relations in the Prague dependency treebank. In Proceedings of DAARC 2004 (pp. 97–102). San Miguel, Azores.Google Scholar
  27. Lappin, S., & Leass, H. J. (1994). An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4), 535–561.Google Scholar
  28. Luo, X., Ittycheriah, A., Jing, H., Kambhatla, N., & Roukos, S. (2004). A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of ACL 2004 (pp. 21–26). Barcelona.Google Scholar
  29. McCarthy, J. F., & Lehnert, W. G. (1995). Using decision trees for coreference resolution. In Proceedings of IJCAI 1995 (pp. 1050–1055). Montréal.Google Scholar
  30. Mengel, A., Dybkjaer, L., Garrido, J. M., Heid, U., Klein, M., Pirrelli, V., et al. (2000). MATE deliverable D2.1 – MATE dialogue annotation guidelines.
  31. Mitkov, R. (1998). Robust pronoun resolution with limited knowledge. In Proceedings of COLING-ACL 1998 (pp. 869–875). Montréal.Google Scholar
  32. Mitkov, R., Evans, R., Orasan, C., Barbu, C., Jones, L., & Sotirova, V. (2000). Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies. In Proceedings of DAARC 2000 (pp. 49–58). Lancaster.Google Scholar
  33. Morton, T. S. (1999). Using coreference in question answering. In Proceedings of TREC-8 (pp. 85–89). Gaithersburg, MD.Google Scholar
  34. MŸller, C., & Strube, M. (2006). Multi-level annotation of linguistic data with MMAX2. In S. Braun, K. Kohn, & J. Mukherjee (Eds.), Corpus technology and language pedagogy: New resources, new tools, new methods (pp. 197–214). Frankfurt: Peter Lang.Google Scholar
  35. Navarretta, C. (2007). A contrastive analysis of abstract anaphora in Danish, English and Italian. In Proceedings of DAARC 2007 (pp. 103–109). Lagos.Google Scholar
  36. Ng, V., & Cardie, C. (2002). Improving machine learning approaches to coreference resolution. In Proceedings of ACL 2002 (pp. 104–111). Philadelphia.Google Scholar
  37. Orasan, C. (2003). PALinkA: A highly customisable tool for discourse annotation. In Proceedings of the 4th SIGdial workshop on discourse and dialogue (pp. 39–43). Sapporo.Google Scholar
  38. Orasan, C., Cristea, D., Mitkov, R., & Branco, A. (2008). Anaphora resolution exercise: An overview. In Proceedings of LREC 2008, Marrakech.Google Scholar
  39. Passonneau, R. (2004). Computing reliability for coreference annotation. In Proceedings of LREC 2004 (pp. 1503–1506). Lisbon.Google Scholar
  40. Passonneau, R. (2006). Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of LREC 2006 (pp. 831–836). Genoa.Google Scholar
  41. Poesio, M. (2004a). Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the ACL 2004 workshop on discourse annotation (pp. 72–79). Barcelona.Google Scholar
  42. Poesio, M. (2004b). The MATE/GNOME proposals for anaphoric annotation, revisited. In Proceedings of the 5th SIGdial workshop at HLT-NAACL 2004 (pp. 154–162). Boston.Google Scholar
  43. Poesio, M., & Artstein, R. (2008). Anaphoric annotation in the ARRAU corpus. In Proceedings of LREC 2008, Marrakech.Google Scholar
  44. Poesio, M., & Vieira, R. (1998). A corpus-based investigation of definite description use. Computational Linguistics 24(2), 183–216.Google Scholar
  45. Pradhan, S. S., Ramshaw, L., Weischedel, R., MacBride, J., & Micciulla, L. (2007). Unrestricted coreference: Identifying entities and events in OntoNotes. In Proceedings of ICSC 2007 (pp. 446–453). Irvine, CA.Google Scholar
  46. Recasens, M., Martí, M. A., & Taulé, M. (2009a). First-mention definites: More than exceptional cases. In S. Featherston, & S. Winkler (Eds.), The fruits of empirical linguistics (pp. 169–189). Berlin: De Gruyter.Google Scholar
  47. Recasens, M., Martí, M. A., Taulé, M., Màrquez, L., & Sapena, E. (2009b). SemEval-2010 Task 1: Coreference resolution in multiple languages. In Proceedings of the NAACL 2009 workshop on semantic evaluations: Recent achievements and future directions (pp. 70–75). Boulder, CO.Google Scholar
  48. Siegel, S., & Castellan, N. J. (1988). Nonparametric statistics for the behavioral sciences, Chap. 9.8 (2nd ed.). New York: McGraw Hill.Google Scholar
  49. Solà, J. (Ed.). (2002). Gramàtica del català contemporani. Barcelona: Empúries.Google Scholar
  50. Soon, W. M., Ng, H. T., & Lim, D. C. Y. (2001). A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4), 521–544.CrossRefGoogle Scholar
  51. Stede, M. (2004). The potsdam commentary corpus. In Proceedings of the ACL 2004 workshop on discourse annotation (pp. 96–102). Barcelona.Google Scholar
  52. Steinberger, J., Poesio, M., Kabadjov, M. A., & Jeek, K. (2007). Two uses of anaphora resolution in summarization. Information Processing and Management: an International Journal, 43(6), 1663–1680.CrossRefGoogle Scholar
  53. Taboada, M. (2008). Reference, centers and transitions in spoken Spanish. In J. Gundel & N. Hedberg (Eds.), Reference and reference processing (pp. 176–215). Oxford: Oxford University Press.Google Scholar
  54. Taulé, M., Martí, M. A., & Recasens, M. (2008). AnCora: Multilevel annotated corpora for Catalan and Spanish. In Proceedings of LREC 2008, Marrakech.Google Scholar
  55. van Deemter, K., & Kibble, R. (2000). On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4), 629–637.CrossRefGoogle Scholar
  56. Webber, B. L. (1979). A formal approach to discourse anaphora. New York: Garland Press.Google Scholar
  57. Webber, B. L. (1988). Discourse deixis: Reference to discourse segments. In Proceedings of ACL 1988 (pp. 113–122). Buffalo, New York.Google Scholar
  58. Zaenen, A. (2006). Mark-up barking up the wrong tree. Computational Linguistics, 32(4), 577–580.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.Centre de Llenguatge i Computació (CLiC)University of BarcelonaBarcelonaSpain

Personalised recommendations