VerbNet/OntoNotes-Based Sense Annotation

  • Meredith Green
  • Orin Hargraves
  • Claire Bonial
  • Jinying Chen
  • Lindsay Clark
  • Martha PalmerEmail author


In this chapter, we present our challenges and successes in producing the OntoNotes word sense groupings [41], which represent a slightly more coarse-grained set of English verb senses drawn from WordNet [13], and which have provided the foundation for our VerbNet sense annotation. These sense groupings were based on the successive merging of WordNet senses into more coarse-grained senses according to the results of inter-annotator agreement [10]. We find that the sense granularity, or level of semantic specificity found in this inventory, reflects sense distinctions that can be made consistently and accurately by human annotators, who achieve a high inter-annotator agreement rate of 89\(\%\). This, in turn, leads to a correspondingly high system performance for automatic WSD: sense distinctions with this level of granularity can be detected automatically at 87–89\(\%\) accuracy, making them effective for NLP applications [9].


VerbNet OntoNotes PropBank Word Sense Disambiguation WordNet Polysemy Sense tagging Classifiers 


  1. 1.
    Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Pasca, M., Soroa, A.: A study on similarity and relatedness using distributional and wordnet-based approaches. In: Proceedings of NAACL (2009)Google Scholar
  2. 2.
    Atkins, S.: Tools for computer-aided corpus lexicography: the hector project. Acta Linguistica Hung. 41, 5–72 (1993)Google Scholar
  3. 3.
    Bian, J., Gao, B., Liu, T.-Y.: Knowledge-powered deep learning for word embedding. Machine Learning and Knowledge Discovery in Databases. Springer, Heidelberg (2014)Google Scholar
  4. 4.
    Bonial, C., Stowe, K., Palmer, M.: Renewing and revising SemLink. In: The GenLex Workshop on Linked Data in Linguistics, GenLex-13, Pisa, Italy, September 2013 (2013)Google Scholar
  5. 5.
    Chen, J., Palmer, M.: Towards robust high performance word sense disambiguation of English verbs using rich linguistic features. In: International Joint Conference for Natural Language Processing, IJCNLP-05., Jeju Island, Korea, 11–13 October 2005Google Scholar
  6. 6.
    Chen, J., Dligach, D., Palmer, M.: Towards large-scale high-performance English verb sense disambiguation by using linguistically motivated features. In: International Conference on Semantic Computing, 2007. ICSC 2007, pp. 378–388. IEEE (2007)Google Scholar
  7. 7.
    Collobert, R., Weston, J.: A unified architecture for natural language processing: deep neural networks with multitask learning. In: Proceedings of ICML (2008)Google Scholar
  8. 8.
    Dang, H.T., Palmer, M.: Combining contextual features for word sense disambiguation. In: SIGLEX Workshop on Word Sense Disambiguation in Conjunction with the 40th Meeting of the Association for Computational Linguistics, ACL 02, Philadelphia, 7–12 July 2002Google Scholar
  9. 9.
    Dligach, D., Palmer, M.: Good seed makes a good crop: accelerating active learning using language modeling. In: ACL ’11: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, 19–24 June 2011Google Scholar
  10. 10.
    Duffield, C.J., Hwang, J.D., Brown, S.W., Dligach, D., Vieweg, S.E., Davis, J., Palmer, M.: Criteria for the manual grouping of verb senses. In: Proceedings of the Linguistic Annotation Workshop held in conjunction with ACL-2007 (2007)Google Scholar
  11. 11.
    Edmonds, P., Cotton, S.: Senseval-2: overview. In: Proceedings of Senseval-2: Second International Workshop on Evaluating Word Sense Disambiguation Systems, Toulouse, France, pp. 1–5 (2001)Google Scholar
  12. 12.
    Faruqui, M., Dodge, J., Jauhar, S.K., Dyer, C., Hovy, H., Smith, N.A.: Retrofitting word vectors to semantic lexicons. In: Proceedings of NAACL (2015)Google Scholar
  13. 13.
    Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)Google Scholar
  14. 14.
    Fellbaum, C., Palmer, M., Dang, H.T., Delfs, L., Wolf, S.: Manual and automatic semantic annotation with WordNet. In: Proceedings of the Workshop on WordNet and Other Lexical Resource, Pittsburgh, PA (2001)Google Scholar
  15. 15.
    Francis, N.W., Kucera., H.: Frequency analysis of English usage: Lexicon and grammar. Houghton Mifflin, Boston (1982)Google Scholar
  16. 16.
    Guo, J., Che, W., Wang, H., Liu, T.: Revisiting embedding features for simple semi-supervised learning. In: Proceedings of Empirical Methods in Natural Language Processing (EMNLP), pp. 110–120 (2014).
  17. 17.
    Ide, N., Véronis, J.: Word sense disambiguation: the state of the art. Comput. Linguist. 24(1), 1–40 (1998)Google Scholar
  18. 18.
    Kawahara, D., Peterson, D.W., Palmer, M.: A step-wise usage-based method for inducing polysemy-aware verb classes. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL2014), Baltimore, MD (2014)Google Scholar
  19. 19.
    Kilgarriff, A.: English lexical sample task description. In: Proceedings of Senseval-2: Second International Workshop on Evaluating Word Sense Disambiguation Systems, Toulouse, France, pp. 17–20 (2001)Google Scholar
  20. 20.
    Kilgarriff, A., Palmer, M.: Introduction to the special issue on senseval. Comput. Humanit. 34(1–2), 1–13 (2000). Special Issue on SENSEVALCrossRefGoogle Scholar
  21. 21.
    Kilgarriff, A., Rosenzweig, J.: Framework and results for English SENSEVAL. Comput. Humanit. 34(1–2), 15–48 (2000)CrossRefGoogle Scholar
  22. 22.
    Kilgarriff, A., Rychly, P., Smrz, P., Tugwell, D.: The sketch engine. In: Proceedings of EURALEX, Lorient, France (2004)Google Scholar
  23. 23.
    Kipper, K., Korhonen, A., Ryant, N., Palmer, M.: Extending VerbNet with novel verb classes. In: Fifth International Conference on Language Resources and Evaluation (LREC 2006), Italy, June (2006)Google Scholar
  24. 24.
    Kipper, K., Korhonen, A., Ryant, N., Palmer, M.: A large-scale classification of English verbs. Lang. Resour. Eval. J. 42, 21–40 (2008)CrossRefGoogle Scholar
  25. 25.
    Levin, B.: English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago (1993)Google Scholar
  26. 26.
    Lin, D.: Automatic retrieval and clustering of similar words. In: Proceedings of the 17th International Conference on Computational Linguistics (COLING-ACL-98), Canada, pp. 768–774 (1998)Google Scholar
  27. 27.
    McCallum, A.K.: MALLET: A Machine Learning for Language Toolkit (2002).
  28. 28.
    Melamed, I.D., Resnik, P.: Tagger evaluation given hierarchical tag sets. Comput. Humanit. 34(1–2), 79–84 (2000)CrossRefGoogle Scholar
  29. 29.
    Mihalcea, R., Moldovan, D.I.: Automatic generation of a coarse grained WordNet. In: Proceedings of NAACL-2001 Workshop on WordNet and Other Lexical Resources, Pittsburgh, pp. 35–41 (2001)Google Scholar
  30. 30.
    Mihalcea, R., Chklovski, T., Kilgarriff, A.: The Senseval-3 English lexical sample task. In: Senseval-3: Third International Workshop on the Evaluation of Systems for the Semantic Analysis of Text, pp. 25–28 (2004)Google Scholar
  31. 31.
    Miller, G.A. (ed.): WordNet: an on-line lexical database. International Journal of Lexicography 3(4), 235–312 (1990). Special IssueGoogle Scholar
  32. 32.
    Miller, G.A.: WordNet: a lexical database for English. Commun. ACM 38(11), 39–41 (1995)CrossRefGoogle Scholar
  33. 33.
    Miller, G.A., Fellbaum, C.: Semantic networks of English. Cognition 41(1–3), 197–229 (1991)CrossRefGoogle Scholar
  34. 34.
    Miller, G.A., Chodorow, M., Landes, S., Leacock, C., Thomas, R.G.: Using a semantic concordance for sense identification. In: Proceedings of the Human Language Technology Workshop, Princeton (1994)Google Scholar
  35. 35.
    Palmer, M.: SemLink: linking PropBank, VerbNet and FrameNet. In: Proceedings of the Generative Lexicon Conference, GenLex-09, September, Pisa, Italy (2009)Google Scholar
  36. 36.
    Palmer, M., Fellbaum, C., Cotton, S., Delfs, L., Dang, H.T.: English tasks: all-words and verb lexical sample. In: Proceedings of Senseval-2: Second International Workshop on Evaluating Word Sense Disambiguation Systems, Toulouse, pp. 21–24 (2001)Google Scholar
  37. 37.
    Palmer, M., Babko-Malaya, O., Dang, H.T.: Different sense granularities for different applications. In: Proceedings of the 2\(^{nd}\) Workshop of Scalable Natural Language Understanding Systems (HLT-NAACL 2004), Boston (2004)Google Scholar
  38. 38.
    Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: a corpus annotated with semantic roles. Comput. Linguist. J. 31(1), 71–105 (2005)CrossRefGoogle Scholar
  39. 39.
    Palmer, M., Dang, H.T., Fellbaum, C.: Making fine-grained and coarse-grained sense distinctions, both manually an automatically. J. Nat. Lang. Eng. 13(2), 137–163 (2007)Google Scholar
  40. 40.
    Popescu, O., Palmer, M., Hanks, P.: Mapping CPA patterns onto OntoNotes senses. In: Proceedings of LREC 2014, Iceland (2014)Google Scholar
  41. 41.
    Pradhan, S., Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R.: OntoNotes: a unified relational semantic representation. In: Proceedings of the First IEEE International Conference on Semantic Computing (2007)Google Scholar
  42. 42.
    Resnik, P., Yarowsky, D.: Distinguishing systems and distinguishing senses: new evaluation methods for word sense disambiguation. Nat. Lang. Eng. 5(2), 113–133 (2000)CrossRefGoogle Scholar
  43. 43.
    Turian, J., Ratinov, L., Bengio, Y.: Word representations: a simple and general method for semi-supervised learning. In: Proceedings of ACL (2010)Google Scholar
  44. 44.
    Turney, P.D.: Similarity of semantic relations. Comput. Linguist. 32(3), 379–416 (2006)CrossRefGoogle Scholar
  45. 45.
    Weischedel, R., Hovy, E., Marcus, M., Belvin, R., Palmer, M., Pradhan, S., Ramshaw, L., Xue, N.: OntoNotes: a large training corpus for enhanced processings. In: Olive, J., Christianso, C., McCary, J. (eds.) Handbook of Natural Language Processing and Machine Translation. Springer, New York (2011)Google Scholar
  46. 46.
    Xu, C., Bai, Y., Bian, J., Gao, B., Wang, G., Liu, X., Liu, T.-Y.: Rc-net: a general framework for incorporating knowledge into word representations. In: Proceedings of CIKM (2014)Google Scholar
  47. 47.
    Yarowsky, D., Florian, R., Cucerzan, S., Schafer, C.: The Johns Hopkins Senseval-2 system description. In: Proceedings of Senseval-2: Second International Workshop on Evaluating Word Sense Disambiguation Systems, Toulouse, pp. 163–166 (2001)Google Scholar
  48. 48.
    Yu, M., Dredze, M.: Improving lexical embeddings with semantic knowledge. In: Proceedings of ACL (2014)Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  • Meredith Green
    • 1
  • Orin Hargraves
    • 1
  • Claire Bonial
    • 2
  • Jinying Chen
    • 3
  • Lindsay Clark
    • 4
  • Martha Palmer
    • 1
    Email author
  1. 1.University of ColoradoBoulderUSA
  2. 2.U.S. Army Research LaboratoryAdelphiUSA
  3. 3.University of Massachusetts Medical SchoolWorcesterUSA
  4. 4.SDLSuperiorUSA

Personalised recommendations