Language Resources and Evaluation

, Volume 43, Issue 2, pp 139–159 | Cite as

The English lexical substitution task

Article

Abstract

Since the inception of the Senseval series there has been a great deal of debate in the word sense disambiguation (WSD) community on what the right sense distinctions are for evaluation, with the consensus of opinion being that the distinctions should be relevant to the intended application. A solution to the above issue is lexical substitution, i.e. the replacement of a target word in context with a suitable alternative substitute. In this paper, we describe the English lexical substitution task and report an exhaustive evaluation of the systems participating in the task organized at SemEval-2007. The aim of this task is to provide an evaluation where the sense inventory is not predefined and where performance on the task would bode well for applications. The task not only reflects WSD capabilities, but also can be used to compare lexical resources, whether man-made or automatically created, and has the potential to benefit several natural-language applications.

Keywords

Lexical substitution Word sense disambiguation SemEval-2007 

References

  1. Barnard, J. (Ed.) (1986). Macquarie Thesaurus. Sydney: Macquarie Library.Google Scholar
  2. Brants, T., & Franz, A. (2006). Web 1T 5-gram corpus version 1.1. Technical Report.Google Scholar
  3. Briscoe, E., & Carroll, J. (2002). Robust accurate statistical annotation of general text. In Proceedings of the third international conference on Language Resources and Evaluation (LREC) (pp. 1499–1504). Las Palmas, Canary Islands, Spain.Google Scholar
  4. Carpuat, M., & Wu, D. (2007). Improving statistical machine translation using word sense disambiguation. In Proceedings of the joint conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007) (pp. 61–72). Prague, Czech Republic.Google Scholar
  5. Chan, Y. S., Ng, H. T., & Chiang, D. (2007). Word sense disambiguation improves statistical machine translation. In Proceedings of the 45th annual meeting of the association for computational linguistics (pp. 33–40). Prague, Czech Republic.Google Scholar
  6. Dagan, I., Glickman, O., & Magnini, B. (2005). The PASCAL recognising textual entailment challenge. In Proceedings of the PASCAL first challenge workshop (pp. 1–8). Southampton, UK.Google Scholar
  7. Dahl, G., Frassica, A.-M., & Wicentowski, R. (2007). SW-AG: Local context matching for English lexical substitution. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 304–307). Prague, Czech Republic.Google Scholar
  8. Fellbaum, C. (Ed.) (1998). WordNet, an electronic lexical database. Cambridge, MA: The MIT Press.Google Scholar
  9. Giuliano, C., Gliozzo, A., & Strapparava, C. (2007). FBK-irst: Lexical substitution task exploiting domain and syntagmatic coherence. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 145–148). Prague, Czech Republic.Google Scholar
  10. Graff, D. (2003). English Gigaword. Philadelphia: Linguistic Data Consortium.Google Scholar
  11. Hanks, P. (2000). Do word meanings exist?. Computers and the Humanities. Senseval Special Issue, 34(1–2), 205–215.Google Scholar
  12. Hassan, S., Csomai, A., Banea, C., Sinha, R., & Mihalcea, R. (2007). UNT: SubFinder: Combining knowledge sources for automatic lexical substitution. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 410–413). Prague, Czech Republic.Google Scholar
  13. Hawker, T. (2007). USYD: WSD and lexical substitution using the Web1T corpus. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 446–453). Prague, Czech Republic.Google Scholar
  14. Ide, N., & Véronis, J. (1998). Introduction to the special issue on word sense disambiguation: The state of the art. Computational Linguistics, 24(1), 1–40.Google Scholar
  15. Ide, N., & Wilks, Y. (2006). Making sense about sense. In E. Agirre & P. Edmonds (Eds.), Word sense disambiguation, algorithms and applications (pp. 47–73). Springer.Google Scholar
  16. Kilgarriff, A. (2004). How dominant is the commonest sense of a word? In Proceedings of text, speech, dialogue. Brno, Czech Republic.Google Scholar
  17. Kilgarriff, A. (2006). Word senses. In E. Agirre & P. Edmonds (Eds.), Word sense disambiguation, algorithms and applications (pp. 29–46). Springer.Google Scholar
  18. Lee, L. (1999). Measures of distributional similarity. In Proceedings of the 37th annual meeting of the association for computational linguistics (pp. 25–32).Google Scholar
  19. Leech, G. (1992). 100 million words of English: The British National Corpus. Language Research, 28(1), 1–13.Google Scholar
  20. Lin, D. (1998). An information-theoretic definition of similarity. In Proceedings of the 15th international conference on machine learning. Madison, WI.Google Scholar
  21. Lindberg, C. (Ed.) (2004). The Oxford American Writer’s Thesaurus. Oxford: Oxford University Press.Google Scholar
  22. Martinez, D., Kim, S. N., & Baldwin, T. (2007). MELB-MKB: Lexical substitution system based on relatives in context. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 237–240). Prague, Czech Republic.Google Scholar
  23. McCarthy, D. (2002). Lexical substitution as a task for WSD evaluation. In Proceedings of the ACL workshop on word sense disambiguation: Recent successes and future directions (pp. 109–115). Philadelphia, USA.Google Scholar
  24. McCarthy, D. (2008). Lexical substitution as a framework for multiword evaluation. In Proceedings of the sixth international conference on Language Resources and Evaluation (LREC 2008). Marrakech, Morocco.Google Scholar
  25. McCarthy, D., & Navigli, R. (2007). SemEval-2007 Task 10: English lexical substitution task. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 48–53). Prague, Czech Republic.Google Scholar
  26. Mihalcea, R., & Chklovski, T. (2003). Open mind word expert: Creating large annotated data collections with Web Users’ help. In Proceedings of the EACL 2003 workshop on linguistically annotated corpora (pp. 53–60). Budapest.Google Scholar
  27. Mihalcea, R., & Csomai, A. (2005). SenseLearner: Word sense disambiguation for all words in unrestricted text. In Proceedings of the 43rd annual meeting of the association for computational linguistics. University of Michigan, USA.Google Scholar
  28. Miller, G. A., Leacock, C., Tengi, R., & Bunker, R. T. (1993). A semantic concordance. In Proceedings of the ARPA workshop on human language technology (pp. 303–308).Google Scholar
  29. Mohammad, S., Hirst, G., & Resnik, P. (2007). Tor, TorMd: Distributional profiles of concepts for unsupervised word sense disambiguation. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 326–333). Prague, Czech Republic.Google Scholar
  30. Navigli, R. (2006). Meaningful clustering of senses helps boost word sense disambiguation performance. In Proceedings of the 44th annual meeting of the association for Computational Linguistics joint with the 21st International Conference on Computational Linguistics (COLING-ACL 2006) (pp. 105–112). Sydney, Australia.Google Scholar
  31. Navigli, R., Litkowski, K. C., & Hargraves, O. (2007). SemEval-2007 Task 7: Coarse-grained English all-words task. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 30–35). Prague, Czech Republic.Google Scholar
  32. Palmer, M. (2000). Consistent criteria for sense distinctions. Computers and the Humanities. Senseval Special Issue, 34(1–2), 217–222.Google Scholar
  33. Palmer M., Dang, H. T., & Fellbaum, C. (2007). Making fine-grained and coarse-grained sense distinctions, both manually and automatically. Natural Language Engineering, 13(02), 137–163.Google Scholar
  34. Pantel, P., & Lin, D. (2002). Discovering word senses from text. In Proceedings of ACM SIGKDD conference on knowledge discovery and data mining (pp. 613–619). Edmonton, Canada.Google Scholar
  35. Resnik, P., & Yarowsky, D. (2000). Distinguishing systems and distinguishing senses: New evaluation methods for word sense disambiguation. Natural Language Engineering, 5(3), 113–133.Google Scholar
  36. Roget, P. M. (1911). Roget’s International Thesaurus (1st ed.). New York, USA: Cromwell.Google Scholar
  37. Schütze, H. (1998). Automatic word sense discrimination. Computational Linguistics, 24(1), 97–123.Google Scholar
  38. Schütze, H., & Pederson, J. O. (1995). Information retrieval based on word senses. In Proceedings of the fourth annual symposium on document analysis and information retrieval (pp. 161–175). Las Vegas, NV.Google Scholar
  39. Sharoff, S. (2006). Open-source corpora: Using the net to fish for linguistic data. International Journal of Corpus Linguistics, 11(4), 435–462.CrossRefGoogle Scholar
  40. Stokoe, C. (2005). Differentiating homonymy and polysemy in information retrieval. In Proceedings of the joint conference on human language technology and empirical methods in natural language processing (pp. 403–410). Vancouver, BC, Canada.Google Scholar
  41. Stokoe, C., Oakes, M. P., & Tait, J. (2003). Word sense disambiguation in information retrieval revisited. In Proceedings of SIGIR (pp. 159–166).Google Scholar
  42. Thesaurus.com. (2007). Roget’s New Millennium TM Thesaurus(ist ed., v 1.3.1). Lexico Publishing Group, LLC. http://thesaurus.reference.co.
  43. Yuret, D. (2007). KU: Word sense disambiguation by substitution. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 207–214). Prague, Czech Republic.Google Scholar
  44. Zhao, S., Zhao, L., Zhang, Y., Liu, T., & Li, S. (2007). HIT: Web based scoring method for English lexical substitution. In Proceedings of the 4th workshop on Semantic Evaluations (SemEval-2007) (pp. 173–176). Prague, Czech Republic.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.The University of SussexFalmer, East SussexUK
  2. 2.The University of Rome “La Sapienza”RomeItaly

Personalised recommendations