Language Resources and Evaluation

, Volume 43, Issue 2, pp 181–208 | Cite as

Improving English verb sense disambiguation performance with linguistically motivated features and clear sense distinction boundaries



This paper presents a high-performance broad-coverage supervised word sense disambiguation (WSD) system for English verbs that uses linguistically motivated features and a smoothed maximum entropy machine learning model. We describe three specific enhancements to our system’s treatment of linguistically motivated features which resulted in the best published results on SENSEVAL-2 verbs. We then present the results of training our system on OntoNotes data, both the SemEval-2007 task and additional data. OntoNotes data is designed to provide clear sense distinctions, based on using explicit syntactic and semantic criteria to group WordNet senses, with sufficient examples to constitute high quality, broad coverage training data. Using similar syntactic and semantic features for WSD, we achieve performance comparable to that of human taggers, and competitive with the top results for the SemEval-2007 task. Empirical analysis of our results suggests that clarifying sense boundaries and/or increasing the number of training instances for certain verbs could further improve system performance.


Word sense disambiguation Sense granularity Maximum entropy Linguistically motivated features Linear regression 



We gratefully acknowledge the support of the National Science Foundation Grant NSF-0415923, Word Sense Disambiguation, and Defense Advanced Research Projects Agency (DARPA/IPTO) under the GALE program, DARPA/CMO Contract No. HR0011-06-C-0022, subcontract from BBN, Inc. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.


  1. Agirre, E., & Edmonds, P. (2007). Word sense disambiguation: Algorithms and applications. Text, Speech and Language Technology Series (Vol. 33). Springer, Netherland. ISBN: 978-1-4020-6870-6.Google Scholar
  2. Berger, A. L., Della Piertra, S. A., & Della Pietra, V. J. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22(1), 39–71.Google Scholar
  3. Bikel, D. M. (2002). Design of a multi-lingual, parallel-processing statistical parsing engine. In Proceedings of HLT 2002, San Diego, CA.Google Scholar
  4. Bikel, D. M., Schwartz, R., & Weischedel, R. M. (1999). An algorithm that learns what’s in a name. Machine Learning, 34(1–3). Special Issue on Natural Language Learning.Google Scholar
  5. Cai, J. F., Lee, W. S., & Teh, Y. W. (2007). NUS-ML: Improving word sense disambiguation using topic features. In Proceedings of the 4th international workshop on Semantic Evaluations (SemEval 2007), Prague, Czech Republic (pp. 249–252).Google Scholar
  6. Carpuat, M., & Wu, D. (2007). Improving statistical machine translation using word sense disambiguation. In Proceedings of the 2007 joint conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL) (pp. 61–72).Google Scholar
  7. Chan, Y. S., Ng, H. T., & Chiang, D. (2007, June). Word sense disambiguation improves statistical machine translation. In Proceedings of the 45th annual meeting of the association of computational linguistics, Prague, Czech Republic. Association for Computational Linguistics (pp. 33–40).Google Scholar
  8. Chen, J. (2006). Towards high-performance word sense disambiguation by combining rich linguistic knowledge and machine learning approaches. PhD Thesis, University of Pennsylvania.Google Scholar
  9. Chen, J., Dligach, D., & Palmer, M. (2007). Towards large-scale high-performance English verb sense disambiguation by using linguistically motivated features. In Proceedings of the international conference on semantic computing (ICSC 2007). Irvine, CA.Google Scholar
  10. Chen, J., & Palmer, M. (2005, October 11–13). Towards robust high performance word sense disambiguation of English verbs using rich linguistic features. In Proceedings of the 2nd international joint conference on natural language processing, Jeju Island, Korea.Google Scholar
  11. Chen, S. F., & Rosenfeld, R. (1999). A Gaussian prior for smoothing maximum entropy models. Technical Report CMU-CS-99-108, CMU.Google Scholar
  12. Chen, J., Schein, A., Ungar, L., & Palmer, M. (2006). An empirical study of the behavior of word sense disambiguation. In Proceedings of NAACL-HLT 2006, NY, 2006.Google Scholar
  13. Dang, H. T. (2004). Investigations into the role of lexical semantics in word sense disambiguation. PhD Thesis, University of Pennsylvania.Google Scholar
  14. Dang, H. T., & Palmer, M. (2005, June 26–28). The role of semantic roles in disambiguating verb senses. In Proceedings of the 43rd annual meeting of the association for computational linguistics, Ann Arbor, MI.Google Scholar
  15. Dang, H. T., & Palmer, M. (2002). Combining contextual features for word sense disambiguation. In Proceedings of the SIGLEX/SENSEVAL workshop on WSD: Recent successes and future directions, in conjunction with ACL-02, Philadelphia.Google Scholar
  16. Duffield, C. J., Hwang, J. D., Brown, S. W., Dligach, D., Vieweg, S. E., Davis, J., & Palmer, M. (2007). Criteria for the manual grouping of verb senses. In Linguistics annotation workshop, held in conjunction with ACL-2007, Prague, The Czech Republic.Google Scholar
  17. Edmonds, P., & Cotton, S. (2001). SENSEVAL-2: Overview. In Proceedings of SENSEVAL-2: 2nd international workshop on evaluating WSD systems. ACL-SIGLEX, Toulouse France.Google Scholar
  18. Fellbaum, C. (1998). WordNet—an electronic lexical database. Cambridge, MA/London: The MIT Press.Google Scholar
  19. Fellbaum, C., Delfs, L., Wolff, S., & Palmer, M. (2005). Word meaning in Dictionaries, corpora, and the speaker’s mind. In G. Barnbrook, P. Danielsson, & M. Mahlberg (Eds.), Meaningful texts: The extraction of semantic information from monolingual and multilingual corpora (pp. 31–38). Birmingham, UK: Birmingham University Press.Google Scholar
  20. Fellbaum, C., Palmer, M., Dang, H. T., Delfs, L., & Wolf, S. (2001, June 2, 3). Manual and automatic semantic annotation with WordNet. In SIGLEX workshop on WordNet and other lexical resources (NAACL-01), Invited talk, Pittsburgh, PA.Google Scholar
  21. Gonzalo, J., Verdejo, F., Chugur, I., & Cigarran, J. (1998). Indexing with WordNet synsets can improve text retrieval. In Proceedings of the COLING/ACL’98 workshop on usage of WordNet for NLP, Montreal, Canada.Google Scholar
  22. Hanks, P. (1996). Contextual dependencies and lexical sets. The International Journal of Corpus Linguistics, 1, 1.Google Scholar
  23. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006). OntoNotes: The 90% solution. In Proceedings of HLT-NAACL06, New York.Google Scholar
  24. Ide, N., & Veronis, J. (1998). Introduction to the special issue on word sense disambiguation: The state of the art. Computational Linguistics, 24(1), 140. Google Scholar
  25. Kipper, K., Korhonen, A., Ryant, N., & Palmer, M. (2006). Extensive classifications of English verbs. In Proceedings of the 12th EURALEX international congress, Turin, Italy.Google Scholar
  26. Lappin, S., & Leass, H. (1994). An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4), 535–561.Google Scholar
  27. Lee, Y. K., & Ng, H. T. (2002). An empirical evaluation of knowledge sources and learning algorithms for word sense disambiguation. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 41–48).Google Scholar
  28. Lee, Y. K., Ng, H. T., & Chia, T. K. (2004). Supervised word sense disambiguation with support vector machines and multiple knowledge sources. In Proceedings of SENSEVAL-3: Third international workshop on the evaluation of systems for the semantic analysis of text, Barcelona, Spain (pp. 137–140).Google Scholar
  29. Levin, B. (1993). English verb classes and alternations: A preliminary investigation. Chicago: University of Chicago Press.Google Scholar
  30. Lucke, J. F., & Embretson, S. (1984). The biases and mean squared errors of estimators of multinormal squared multiple correlation. Journal of Educational Statistics, 9(3), 183–192. doi: 10.2307/1165005.CrossRefGoogle Scholar
  31. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Ferguson, M., Katz, K., et al. (1994). The Penn Treebank: Annotating predicate argument structure. In Proceedings of the ARPA’94 HLT workshop.Google Scholar
  32. McCallum, A. K. (2002). MALLET: A machine learning for language toolkit.
  33. Mihalcea, R., Chklovski, T., & Kilgarriff, A. (2004, July). The Senseval-3 English lexical sample task. In Proceedings of Senseval-3: The third international workshop on the evaluation of systems for the semantic analysis of text, Barcelona, Spain.Google Scholar
  34. Navigli, R. (2006, July 17–18). Meaningful clustering of senses helps boost word sense disambiguation performance. In Proceedings of the 21st international conference on computational linguistics and the 44th annual meeting of the ACL, Sydney, Australia (pp. 105–112).Google Scholar
  35. Navigli, R., Litkowski, K. C., & Hargraves, O. (2007, June). SemEval-2007 Task 07: Coarse-grained English all-words task. In Proceedings of SemEval, held in conjunction with ACL 2007, Prague, Czech Republic.Google Scholar
  36. Palmer, M., Babko-Malaya, O., & Dang, H. T. (2004). Different sense granularities for different applications. In Proceedings of the 2nd workshop on scalable natural language understanding systems (HLT/NAACL 2004). Boston, MA.Google Scholar
  37. Palmer, M., Dang, H., & Fellbaum, C. (2007, June). Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Journal of Natural Language Engineering, 13(2), 137–163.Google Scholar
  38. Palmer, M., Fellbaum, C., Cotton, S., Delfs, L., & Dang, H. T. (2001, July 5–6). English tasks: All-words and verb lexical sample. In Proceedings of SENSEVAL-2: Second international workshop on evaluating word sense disambiguation systems. Toulouse, France.Google Scholar
  39. Palmer, M., Gildea, D., & Kingsbury, P. (2005). The proposition bank: A corpus annotated with semantic roles. Computational Linguistics, 31, 1. doi: 10.1162/0891201053630264.CrossRefGoogle Scholar
  40. Philpot, A., Hovy, E., & Pantel, P. (2005). The omega ontology. In Proceedings of the ONTOLEX workshop at the International Conference on Natural Language Processing (IJCNLP05). Jeju Island, Korea.Google Scholar
  41. Pradhan, S., Loper, E., Dligach, D., & Palmer, M. (2007, June). SemEval-2007 task-17: English lexical sample, SRL and all words. In Proceedings of SemEval, held in conjunction with ACL 2007, Prague, Czech Republic.Google Scholar
  42. Ratnaparkhi, A. (1998). Maximum entropy models for natural language ambiguity resolution. Ph.D. Thesis, University of Pennsylvania.Google Scholar
  43. Sanderson, M. (1994). Word sense disambiguation and information retrieval. In Proceedings of the 17th International ACM SIGIR, Dublin, IrelandGoogle Scholar
  44. Sanderson, M. (2000). Retrieving with good sense. Information Retrieval, 2(1), 49–69.CrossRefGoogle Scholar
  45. Snyder, B., & Palmer, M. (2004, July). The English all-words task. In Proceedings of Senseval-3: The third international workshop on the evaluation of systems for the semantic analysis of text. Barcelona, Spain.Google Scholar
  46. Stokoe, C., Oakes, M. P., & Tait, J. (2003). Word sense disambiguation and information retrieval revisited. In Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval, Toronto, Canada.Google Scholar
  47. Yarowsky, D. (1993). One sense per collocation. In Proceedings of the 5th DARPA speech and natural language workshop.Google Scholar
  48. Yarowsky, D., Cucerzan, S., Florian, R., Schafer, C., & Wicentowski, R. (2001). The Johns Hopkins SENSEVAL2 system description. In Proceedings of SENSEVAL-2: 2nd international workshop on evaluating WSD systems, Toulouse France.Google Scholar
  49. Yarowsky, D., & Florian, R. (2002). Evaluating sense disambiguation across diverse parameter spaces. Journal of Natural Language Engineering, 8(4), 293–310.CrossRefGoogle Scholar
  50. Yi, S.-t., Loper, E., & Palmer, M. (2007, April). Can semantic roles generalize across genres? In Proceedings of NAACL 2007, Rochester, NY.Google Scholar
  51. Zhong, Z., Tou Ng, H., & Chan, Y. S. (2008, October). Word sense disambiguation using OntoNotes: An empirical study. In Proceedings of EMNLP 2008, Waikiki, Honolulu, HI.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.BBN TechnologiesCambridgeUSA
  2. 2.University of ColoradoBoulderUSA

Personalised recommendations