Advertisement

Language Resources and Evaluation

, Volume 52, Issue 3, pp 673–706 | Cite as

Annotation of semantic roles for the Turkish Proposition Bank

  • Gözde Gül Şahin
  • Eşref Adalı
Original Paper

Abstract

In this work, we report large-scale semantic role annotation of arguments in the Turkish dependency treebank, and present the first comprehensive Turkish semantic role labeling (SRL) resource: Turkish Proposition Bank (PropBank). We present our annotation workflow that harnesses crowd intelligence, and discuss the procedures for ensuring annotation consistency and quality control. Our discussion focuses on syntactic variations in realization of predicate-argument structures, and the large lexicon problem caused by complex derivational morphology. We describe our approach that exploits framesets of root verbs to abstract away from syntax and increase self-consistency of the Turkish PropBank. The issues that arise in the annotation of verbs derived via valency changing morphemes, verbal nominals, and nominal verbs are explored, and evaluation results for inter-annotator agreement are provided. Furthermore, semantic layer described here is aligned with universal dependency (UD) compliant treebank and released to enable more researchers to work on the problem. Finally, we use PropBank to establish a baseline score of 79.10 F1 for Turkish SRL using the mate-tool (an open-source SRL tool based on supervised machine learning) enhanced with basic morphological features. Turkish PropBank and the extended SRL system are made publicly available.

Keywords

PropBank Semantic role annotation Derivational morphology Turkish Crowdsourcing Semantic role labeling 

Notes

Acknowledgements

The first author is funded by Tubitak 2211 domestic Ph.D. Scholarship Program. We would like to sincerely thank Gülşen Eryiğit for fruitful discussions, guidance throughout this work and allowing access to the IMST corpus prior to its public release. Furthermore, we are more than grateful to Celal Şahin, İsmail Hakkı Yadigar, Kurt Bandy and Erdem İşgüder for their invaluable feedback. We are thankful to our anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions. Finally, this work would not be possible without our anonymous crowdworkers from all around the country.

References

  1. Akbik, A., Chiticariu, L., Danilevsky, M., Li, Y., Vaithyanathan, S., & Zhu, H. (2015). Generating high quality proposition banks for multilingual semantic role labeling. In Proceedings of the 53rd annual meeting of the association for computational linguistics (pp. 397–407), Beijing, China, July 2015. Association for Computational Linguistics.Google Scholar
  2. Akbik, A., & Li, Y. (2016). Towards semi-automatic generation of proposition banks for low-resource languages. In EMNLP.Google Scholar
  3. Baker, C. F., Fillmore, C. J., & Lowe, J. B. (1998). The berkeley framenet project. In Proceedings of the 36th annual meeting of the association for computational linguistics and 17th international conference on computational linguistics-Volume 1 (pp. 86–90). Association for Computational Linguistics.Google Scholar
  4. Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., et al. (2012). Abstract meaning representation (amr) 1.0 specification. Parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing pp. 1533–1544. Seattle: ACL.Google Scholar
  5. Basile, V., Bos, J., Evang, K., & Venhuizen, N. (2012). Developing a large semantically annotated corpus. LREC, 12, 3196–3200.Google Scholar
  6. Björkelund, A., Hafdell, L., & Nugues, P. (2009). Multilingual semantic role labeling. In Proceedings of the thirteenth conference on computational natural language learning: Shared task (pp. 43–48). Association for Computational Linguistics.Google Scholar
  7. Callison-Burch, C., Ungar, L., & Pavlick, E. (2015). Crowdsourcing for NLP. In Proceedings of NAACL 2015. North America Association for Computational Linguistics.Google Scholar
  8. Carreras, X., & Màrquez, L. (2004). Introduction to the conll-2004 shared task: Semantic role labeling. In Proceedings of the ninth conference on computational natural language learning. Association for Computational Linguistics.Google Scholar
  9. Carreras, X., & Màrquez, L. (2005). Introduction to the conll-2005 shared task: Semantic role labeling. In Proceedings of the ninth conference on computational natural language learning (pp. 152–164). Association for Computational Linguistics.Google Scholar
  10. Chang, N., Paritosh, P., Huynh, D., & Baker, C. F. (2015). Scaling semantic frame annotation. In The 9th linguistic annotation workshop held in conjuncion with NAACL 2015 (p. 1).Google Scholar
  11. Christensen, J., Soderland, S., Etzioni, O., et al. (2010). Semantic role labeling for open information extraction. In Proceedings of the NAACL HLT 2010 first international workshop on formalisms and methodology for learning by reading, (pp. 52–60). Association for Computational Linguistics.Google Scholar
  12. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12, 2461–2505.Google Scholar
  13. Deschacht, K., & Moens, M.-F. (2009). Semi-supervised semantic role labeling using the latent words language model. In Proceedings of the 2009 conference on empirical methods in natural language processing: Volume 1–Volume 1 (pp. 21–29). Association for Computational Linguistics.Google Scholar
  14. Duran, M. S., & Aluísio, S. M. (2012). Propbank-br: A Brazilian treebank annotated with semantic role labels. In LREC.Google Scholar
  15. Exner, P., & Nugues, P. (2011). Using semantic role labeling to extract events from wikipedia. In Proceedings of the workshop on detection, representation, and exploitation of events in the semantic web (DeRiVE 2011). Workshop in conjunction with the 10th international semantic web conference (pp. 23–24).Google Scholar
  16. Feizabadi, P. S., & Padó, S. (2014). Crowd sourcing annotation of non-local semantic roles. In EACL (pp. 226–230).Google Scholar
  17. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378.CrossRefGoogle Scholar
  18. Fossati, M., Giuliano, C., & Tonelli, S. (2013a). Outsourcing framenet to the crowd. ACL, 2, 742–747.Google Scholar
  19. Fossati, M., Tonelli, S., & Giuliano, C. (2013b). Frame semantics annotation made easy with DBpedia. In Crowdsourcing, the, semantic, web.Google Scholar
  20. Fürstenau, H., & Lapata, M. (2009). Graph alignment for semi-supervised semantic role labeling. In Proceedings of the 2009 conference on empirical methods in natural language processing Volume 1-EMNLP ’09 (pp. 11–20).Google Scholar
  21. Gao, Q., & Vogel, S. (2011). Corpus expansion for statistical machine translation with semantic role label substitution rules. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies: Short papers-Volume 2 (pp. 294–298). Association for Computational Linguistics.Google Scholar
  22. Gildea, D. (2002). Automatic labeling of semantic roles. Computational Linguistics, 28(3), 245–288.CrossRefGoogle Scholar
  23. Hajič, J., Ciaramita, M., Johansson, R., Kawahara, D., Martí, M. A., Màrquez, L., et al. (2009). The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the thirteenth conference on computational natural language learning: Shared task, CoNLL ’09 (pp. 1–18), Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.Google Scholar
  24. Haverinen, K., Kanerva, J., Kohonen, S., Missila, A., Ojala, S., Viljanen, T., et al. (2015). The Finnish Proposition Bank. Language Resources and Evaluation, 49(4), 907–926.CrossRefGoogle Scholar
  25. He, L., Lewis, M., & Zettlemoyer, L. (2015). Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 643–653), Lisbon, Portugal, Association for Computational Linguistics.Google Scholar
  26. Hengirmen, M. (2006). Türkçe temel dilbilgisi. Cankaya: Engin Yayınevi.Google Scholar
  27. Hong, J., & Baker, C. F. (2011). How good is the crowd at real wsd? In Proceedings of the 5th linguistic annotation workshop (pp. 30–37). Association for Computational Linguistics.Google Scholar
  28. Hung, S.-H., Lin, C.-H., & Hong, J.-S. (2010). Web mining for event-based commonsense knowledge using lexico-syntactic pattern matching and semantic role labeling. Expert Systems with Applications, 37(1), 341–347.CrossRefGoogle Scholar
  29. Irvine, A., & Klementiev, A. (2010). Using mechanical turk to annotate lexicons for less commonly used languages. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with amazon’s mechanical turk (pp. 108–113). Association for Computational Linguistics.Google Scholar
  30. Kaisser, M., & Webber, B. (2007). Question answering based on semantic roles. In Proceedings of the workshop on deep linguistic processing (pp. 41–48). Association for Computational Linguistics.Google Scholar
  31. Kozhevnikov, M., & Titov, I. (2013). Cross-lingual transfer of semantic role labeling models. ACL, 1, 1190–1200.Google Scholar
  32. Lewis, G. L. (1985). Turkish grammar. Oxford: Oxford University Press.Google Scholar
  33. Liu, D., & Gildea, D. (2010). Semantic role features for machine translation. In Proceedings of the 23rd international conference on computational linguistics (pp. 716–724). Association for Computational Linguistics.Google Scholar
  34. Lo, C.-k., Addanki, K., Saers, M., & Wu, D. (2013). Improving machine translation by training against an automatic semantic frame based evaluation metric. In ACL (pp. 375–381).Google Scholar
  35. Madnani, N., Tetreault, J., Chodorow, M., & Rozovskaya, A. (2011). They can help: Using crowdsourcing to improve the evaluation of grammatical error detection systems. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies: Short papers-Volume 2 (pp. 508–513). Association for Computational Linguistics.Google Scholar
  36. May, J. (2016). Semeval-2016 task 8: Meaning representation parsing. In Proceedings of SemEval (pp. 1063–1073).Google Scholar
  37. Mellebeek, B., Benavent, F., Grivolla, J., Codina, J., Costa-Jussa, M. R., & Banchs, R. (2010). Opinion mining of spanish customer comments with non-expert annotations on mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon’s mechanical turk (pp. 114–121). Association for Computational Linguistics.Google Scholar
  38. Meyers, A., Reeves, R., Macleod, C., Szekely, R., Zielinska, V., Young, B., et al. (2004). The nombank project: An interim report. In HLT-NAACL 2004 workshop: Frontiers in corpus annotation (Vol. 24, p. 31).Google Scholar
  39. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3), 436–465.CrossRefGoogle Scholar
  40. Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Cinková, S., Flickinger, D., et al. (2015). Semeval 2015 task 18: Broad-coverage semantic dependency parsing. In SemEval@NAACL-HLT.Google Scholar
  41. Oepen, S., Kuhlmann, M., Miyao, Y., Zeman, D., Flickinger, D., Hajic, J., et al. (2014). Semeval 2014 task 8: Broad-coverage semantic dependency parsing. In Proceedings of the 8th international workshop on semantic evaluation (SemEval 2014) (pp. 63–72).Google Scholar
  42. Oflazer, K. (2014). Turkish and its challenges for language processing. Language Resources and Evaluation, 48(4), 639–653.CrossRefGoogle Scholar
  43. Oflazer, K., & El-Kahlout, I. D. (2007). Exploring different representational units in english-to-turkish statistical machine translation. In Proceedings of the second workshop on statistical machine translation (pp. 25–32). Association for Computational Linguistics.Google Scholar
  44. Oflazer, K., Say, B., Hakkani-Tür, D. Z., & Tür, G. (2003). Building a Turkish treebank. In A. Abeillé (ed) Treebanks: Building and Using Parsed Corpora (pp. 261–277). Dordrecht: Springer. doi: 10.1007/978-94-010-0201-1_15.CrossRefGoogle Scholar
  45. Padó, S., & Lapata, M. (2009). Cross-lingual annotation projection for semantic roles. Journal of Artificial Intelligence Research, 36(1), 307–340.CrossRefGoogle Scholar
  46. Palmer, M., Bhatt, R., Narasimhan, B., Rambow, O., Sharma, D. M., & Xia, F. (2009). Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In Proceedings of the 7th international conference on natural language processing, ICON’09 (pp. 261–268).Google Scholar
  47. Palmer, M., Gildea, D., & Kingsbury, P. (2005). The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1), 71–106.CrossRefGoogle Scholar
  48. Pavlick, E., Post, M., Irvine, A., Kachaev, D., & Callison-Burch, C. (2014). The language demographics of amazon mechanical turk. Transactions of the Association for Computational Linguistics, 2, 79–92.Google Scholar
  49. Reisinger, D., Rudinger, R., Ferraro, F., Harman, C., Rawlins, K., & Van Durme, B. (2015). Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3, 475–488.Google Scholar
  50. Rim, K. (2015). Event-participant linking-a crowdsourcing approach to semantic role labeling. Master’s thesis, Brandeis University.Google Scholar
  51. Sabou, M., Bontcheva, K., Derczynski, L., & Scharl, A. (2014). Corpus annotation through crowdsourcing: Towards best practice guidelines. In Proceedings of LREC.Google Scholar
  52. Saffran, J. R. (2003). Statistical language learning mechanisms and constraints. Current Directions in Psychological Science, 12(4), 110–114.CrossRefGoogle Scholar
  53. Saffran, J. R., Johnson, E. K., Aslin, R. N., & Newport, E. L. (1999). Statistical learning of tone sequences by human infants and adults. Cognition, 70(1), 27–52.CrossRefGoogle Scholar
  54. Sahin, G. G. (2016a). Framing of verbs for turkish propbank. In Proceedings of 1st international conference on turkic computational linguistics, TurCLing.Google Scholar
  55. Sahin, G. G. (2016b). Verb sense annotation for turkish propbank via crowdsourcing. In Proceedings of 17th international conference on intelligent text processing and computational linguistics. CICLING 2016.Google Scholar
  56. Sahin, G. G., & Adalı, E. (2014). Using morphosemantic information in construction of a pilot lexical semantic resource for Turkish. In Workshop on lexical and grammatical resources for language processing (p. 46).Google Scholar
  57. Sammons, M., Vydiswaran, V. V., Vieira, T., Johri, N., Chang, M.-W., Goldwasser, D., et al. (2009). Relation alignment for textual entailment recognition. In Text analysis conference (TAC).Google Scholar
  58. Schuster, S., & Manning, C. D. (2016). Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In Proceedings of the tenth international conference on language resources and evaluation (LREC 2016).Google Scholar
  59. Shen, D., & Lapata, M. (2007). Using semantic roles to improve question answering. In EMNLP-CoNLL (pp. 12–21).Google Scholar
  60. Snow, R., O’Connor, B., Jurafsky, D., & Ng, A. Y. (2008). Cheap and fast—but is it good?: Evaluating non-expert annotations for natural language tasks. In Proceedings of the conference on empirical methods in natural language processing (pp. 254–263). Association for Computational Linguistics.Google Scholar
  61. Sulubacak, U., Gökırmak, M., & Eryiğit, G. (2016a). Universal dependencies for Turkish. In COLING.Google Scholar
  62. Sulubacak, U., Pamay, T., & Eryigit, G. (2016b). Imst: A revisited turkish dependency treebank. In Proceedings of 1st international conference on turkic computational linguistics, TurCLing.Google Scholar
  63. Surdeanu, M., Johansson, R., Meyers, A., Màrquez, L., & Nivre, J. (2008). The conll-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the twelfth conference on computational natural language learning (pp. 159–177). Association for Computational Linguistics.Google Scholar
  64. Titov, I., & Klementiev, A. (2011). A bayesian model for unsupervised semantic parsing. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-Volume 1 (pp. 1445–1455). Association for Computational Linguistics.Google Scholar
  65. Toutanova, K., Haghighi, A., & Manning, C. D. (2005). Joint learning improves semantic role labeling. In Proceedings of the 43rd annual meeting on association for computational linguistics (pp. 589–596). Association for Computational Linguistics.Google Scholar
  66. Wu, D., & Fung, P. (2009). Can semantic role labeling improve SMT. In Proceedings of the 13th annual conference of the EAMT (pp. 218–225).Google Scholar
  67. Xue, N., & Palmer, M. (2008). Adding semantic roles to the chinese treebank. Natural Language Engineering, 15(1), 143. (ISSN 1351-3249) .CrossRefGoogle Scholar
  68. Xue, N., & Palmer, M. (2009). Adding semantic roles to the chinese treebank. Natural Language Engineering, 15(01), 143–172.CrossRefGoogle Scholar
  69. Zaghouani, W., Diab, M., Mansouri, A., Pradhan, S., & Palmer, M. (2010). The revised Arabic PropBank. In Proceedings of the fourth linguistic annotation workshop (pp. 222–226).Google Scholar
  70. Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-Volume 1 (pp. 1220–1229). Association for Computational Linguistics.Google Scholar
  71. Zhai, H., Lingren, T., Deleger, L., Li, Q., Kaiser, M., Stoutenborough, L., et al. (2013). Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. Journal of medical Internet research, 15(4), e73.CrossRefGoogle Scholar
  72. Zhou, J., & Xu, W. (2015). End-to-end learning of semantic role labeling using recurrent neural networks. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers) (pp. 1127–1137).Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  1. 1.Department of Computer EngineeringIstanbul Technical UniversityIstanbulTurkey

Personalised recommendations