Advertisement

Evaluating Learning Language Representations

  • Jussi Karlgren
  • Jimmy Callin
  • Kevyn Collins-Thompson
  • Amaru Cuba Gyllensten
  • Ariel Ekgren
  • David Jurgens
  • Anna Korhonen
  • Fredrik Olsson
  • Magnus Sahlgren
  • Hinrich Schütze
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9283)

Abstract

Machine learning offers significant benefits for systems that process and understand natural language: (a) lower maintenance and upkeep costs than when using manually-constructed resources, (b) easier portability to new domains, tasks, or languages, and (c) robust and timely adaptation to situation-specific settings. However, the behaviour of an adaptive system is less predictable than when using an edited, stable resource, which makes quality control a continuous issue. This paper proposes an evaluation benchmark for measuring the quality, coverage, and stability of a natural language system as it learns word meaning. Inspired by existing tests for human vocabulary learning, we outline measures for the quality of semantic word representations, such as when learning word embeddings or other distributed representations. These measures highlight differences between the types of underlying learning processes as systems ingest progressively more data.

Keywords

Language representations Semantic spaces Word embeddings Machine learning Evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Baroni, M., Lenci, A.: How we BLESSed distributional semantic evaluation. In: Proceedings of the 2011 Workshop on GEometrical Models of Natural Language Semantics, pp. 1–10. ACL (2011)Google Scholar
  2. 2.
    Cook, P., Lau, J.H., McCarthy, D., Baldwin, T.: Novel word-sense identification. In: Proceedings of COLING, pp. 1624–1635 (2014)Google Scholar
  3. 3.
    Frishkoff, G.A., Collins-Thompson, K., Perfetti, C.A., Callan, J.: Measuring incremental changes in word knowledge: Experimental validation and implications for learning and assessment. Behavior Research Methods 40(4), 907–925 (2008)CrossRefGoogle Scholar
  4. 4.
    Frishkoff, G.A., Perfetti, C.A., Collins-Thompson, K.: Predicting robust vocabulary growth from measures of incremental learning. Scientific Studies of Reading 15(1), 71–91 (2011)CrossRefGoogle Scholar
  5. 5.
    Hill, F., Reichart, R., Korhonen, A.: Simlex-999: Evaluating semantic models with (genuine) similarity estimation (2014). arXiv preprint arXiv:1408.3456
  6. 6.
    Karlgren, J. (ed.): Proceedings of the EACL workshop on New Text: Wikis and blogs and other dynamic text sources, EACL 2006Google Scholar
  7. 7.
    Landauer, T., Dumais, S.: A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review 104(2), 211–240 (1997)CrossRefGoogle Scholar
  8. 8.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of NIPS, pp. 3111–3119 (2013)Google Scholar
  9. 9.
    Turney, P.D., Pantel, P.: From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research 37, 141–188 (2010)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Jussi Karlgren
    • 1
    • 2
  • Jimmy Callin
    • 1
  • Kevyn Collins-Thompson
    • 3
  • Amaru Cuba Gyllensten
    • 1
  • Ariel Ekgren
    • 1
  • David Jurgens
    • 4
  • Anna Korhonen
    • 5
  • Fredrik Olsson
    • 1
  • Magnus Sahlgren
    • 1
  • Hinrich Schütze
    • 6
  1. 1.GavagaiStockholmSweden
  2. 2.Kungl Tekniska HögskolanStockholmSweden
  3. 3.University of MichiganAnn ArborUSA
  4. 4.McGill UniversityMontréalCanada
  5. 5.University of CambridgeCambridgeUK
  6. 6.Ludwig-Maximilians-UniversitätMünchenGermany

Personalised recommendations