Adjudication of Coreference Annotations via Answer Set Optimization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10377)


We describe the first automatic approach for merging coreference annotations obtained from multiple annotators into a single gold standard. Merging is subject to hard constraints (consistency) and optimization criteria (minimal divergence from annotators) and involves an equivalence relation over a large number of elements. We describe two representations of the problem in Answer Set Programming and four objective functions suitable for the task. We provide two structurally different real-world benchmark datasets based on the METU-Sabanci Turkish Treebank, and we report our experiences in using the Gringo, Clasp, and Wasp tools for computing optimal adjudication results on these datasets.


Coreference resolution Adjudication Answer set programming 


  1. 1.
    Alviano, M., Dodaro, C., Marques-Silva, J., Ricca, F.: Optimum stable model search: algorithms and implementation. J. Logic Comput. (2015). doi: 10.1093/logcom/exv061
  2. 2.
    Alviano, M., Dodaro, C.: Anytime answer set optimization via unsatisfiable core shrinking. Theor. Pract. Logic Program. 16(5–6), 533–551 (2016)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Alviano, M., Dodaro, C., Leone, N., Ricca, F.: Advances in WASP. In: Calimeri, F., Ianni, G., Truszczynski, M. (eds.) LPNMR 2015. LNCS (LNAI), vol. 9345, pp. 40–54. Springer, Cham (2015). doi: 10.1007/978-3-319-23264-5_5 CrossRefGoogle Scholar
  4. 4.
    Andres, B., Kaufmann, B., Matheis, O., Schaub, T.: Unsatisfiability-based optimization in clasp. In: International Conference on Logic Programming (ICLP), Technical Communications, pp. 212–221 (2012)Google Scholar
  5. 5.
    Ansótegui, C., Bonet, M.L., Levy, J.: SAT-based MaxSAT algorithms. Artif. Intell. 196, 77–105 (2013)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Baral, C.: Knowledge Representation, Reasoning, and Declarative Problem Solving. Cambridge University Press, New York (2004)MATHGoogle Scholar
  7. 7.
    Brewka, G., Eiter, T., Truszczynski, M.: Answer set programming at a glance. Commun. ACM 54(12), 92–103 (2011)CrossRefGoogle Scholar
  8. 8.
    Calimeri, F., Faber, W., Gebser, M., Ianni, G., Kaminski, R., Krennwallner, T., Leone, N., Ricca, F., Schaub, T.: ASP-Core-2 Input language format. Technical report, ASP Standardization Working Group (2012)Google Scholar
  9. 9.
    Chomicki, J., Marcinkowski, J.: Minimal-change integrity maintenance using tuple deletions. Inf. Comput. 197(1), 90–121 (2005)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Clark, J.H., González-Brenes, J.P.: Coreference resolution: current trends and future directions. Lang. Stat. II Lit. Rev. (2008).
  11. 11.
    Denis, P., Baldridge, J.: Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Nat. 42, 87–96 (2009)Google Scholar
  12. 12.
    Eiter, T., Fink, M., Schüller, P., Weinzierl, A.: Finding explanations of inconsistency in multi-context systems. Artif. Intell. 216, 233–274 (2014)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Eiter, T., Fink, M., Stepanova, D.: Towards practical deletion repair of inconsistent DL-programs. In: European Conference on Artificial Intelligence, pp. 285–290 (2014)Google Scholar
  14. 14.
    Erdem, E., Lifschitz, V.: Tight logic programs. Theor. Pract. Logic Program. 3(4–5), 499–518 (2003)MathSciNetCrossRefMATHGoogle Scholar
  15. 15.
    Gebser, M., Janhunen, T., Rintanen, J.: ASP encodings of acyclicity properties. J. Logic Comput. 25, 613–638 (2015)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Gebser, M., Kaminski, R., Kaufmann, B., Schaub, T.: Answer Set Solving in Practice. Morgan Claypool (2012)Google Scholar
  17. 17.
    Gebser, M., Kaminski, R., König, A., Schaub, T.: Advances in gringo series 3. In: Delgrande, J.P., Faber, W. (eds.) LPNMR 2011. LNCS (LNAI), vol. 6645, pp. 345–351. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-20895-9_39 CrossRefGoogle Scholar
  18. 18.
    Gebser, M., Kaufmann, B., Schaub, T.: Conflict-driven answer set solving: from theory to practice. Artif. Intell. 187–188, 52–89 (2012)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Hirst, G.: Anaphora in Natural Language Understanding: A Survey. Springer, New York (1981)CrossRefGoogle Scholar
  20. 20.
    Kehler, A., Kertz, L., Rohde, H., Elman, J.L.: Coherence and coreference revisited. J. Semant. 25(1), 1–44 (2008)CrossRefGoogle Scholar
  21. 21.
    Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., Jurafsky, D.: Deterministic coreference resolution based on entity-centric, precision-ranked rules. Comput. Linguist. 39(4), 885–916 (2013)CrossRefGoogle Scholar
  22. 22.
    Lifschitz, V.: What is answer set programming? In: AAAI Conference on Artificial Intelligence pp. 1594–1597 (2008)Google Scholar
  23. 23.
    Ng, V.: Supervised noun phrase coreference research: the first fifteen years. In: Association for Computational Linguistics (ACL), pp. 1396–1411 (2010)Google Scholar
  24. 24.
    Recasens, M., Hovy, E.: BLANC: implementing the rand index for coreference evaluation. Nat. Lang. Eng. 17(4), 485–510 (2010)CrossRefGoogle Scholar
  25. 25.
    Sapena, E., Padró, L., Turmo, J.: Coreference Resolution. Technical report, TALP Research Center, UPC (2008)Google Scholar
  26. 26.
    Say, B., Zeyrek, D., Oflazer, K., Özge, U.: Development of a corpus and a treebank for present day written Turkish. In: Current Research in Turkish Linguistics (Proceedings of the Eleventh International Conference of Turkish Linguistics, 2002), pp. 183–192. Eastern Mediterranean University Press (2004)Google Scholar
  27. 27.
    Schüller, P.: Adjudication of coreference annotations via finding optimal repairs of equivalence relations. In: International Workshop on Experimental Evaluation of Algorithms for Solving Problems with Combinatorial Explosion (RCRA), pp. 57–71 (2016)Google Scholar
  28. 28.
    Schüller, P.: Modeling variations of first-order horn abduction in answer set programming. Fund. Inform. 149, 159–207 (2016)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Stoyanov, V., Eisner, J.: Easy-first coreference resolution. In: International Conference on Computational Linguistics (COLING), pp. 2519–2534 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Computer Engineering Department, Faculty of EngineeringMarmara UniversityIstanbulTurkey

Personalised recommendations