Skip to main content
SpringerLink
Log in

Enhancing the Numeracy of Word Embeddings: A Linear Algebraic Perspective

  • Conference paper
  • First Online:
Book cover Natural Language Processing and Chinese Computing (NLPCC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12430))

  • 3034 Accesses

Abstract

To reason over the embeddings of numbers, they should capture numeracy information. In this work, we consider the magnitude aspect of numeracy information. We could find a vector in a high dimensional space and a subspace of original space. After projecting the original embeddings of numbers onto that vector or subspace, the magnitude information could be significantly enhanced. Therefore, this paper proposes a new angle to study numeracy of word embeddings, which is interpretable and has nice mathematical formulations.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    In our work, we restrict our scope to Arabic numbers.

  2. 2.

    Embeddings are available at http://vectors.nlpl.eu/repository with ids 5, 11, 7, 13, 9, and 15 [8].

References

  1. Arora, S., Li, Y., Liang, Y., Ma, T., Risteski, A.: Linear algebraic structure of word senses, with applications to polysemy. Trans. Assoc. Comput. Linguist. 6, 483–495 (2018)

    Article  Google Scholar 

  2. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  3. Cantlon, J.F., Brannon, E.M.: Shared system for ordering small and large numbers in monkeys and humans. Psychol. Sci. 17(5), 401–406 (2006)

    Article  Google Scholar 

  4. Chen, D., Manning, C.: A fast and accurate dependency parser using neural networks. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 740–750 (2014)

    Google Scholar 

  5. Dehaene, S., Dehaene-Lambertz, G., Cohen, L.: Abstract representations of numbers in the animal and human brain. Trends Neurosci. 21(8), 355–361 (1998)

    Article  Google Scholar 

  6. Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., Gardner, M.: Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161 (2019)

  7. Dyer, C., Ballesteros, M., Ling, W., Matthews, A., Smith, N.A.: Transition-based dependency parsing with stack long short-term memory. arXiv preprint arXiv:1505.08075 (2015)

  8. Fares, M., Kutuzov, A., Oepen, S., Velldal, E.: Word vectors, reuse, and replicability: towards a community repository of large-text resources. In: Proceedings of the 21st Nordic Conference on Computational Linguistics, pp. 271–276. Association for Computational Linguistics, Gothenburg, Sweden, May 2017

    Google Scholar 

  9. Frazier, P.I.: A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811 (2018)

  10. Gatt, A., Krahmer, E.: Survey of the state of the art in natural language generation: core tasks, applications and evaluation. J. Artif. Intell. Res. 61, 65–170 (2018)

    Article  MathSciNet  Google Scholar 

  11. Glavaš, G., Vulić, I.: Explicit retrofitting of distributional word vectors. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 34–45 (2018)

    Google Scholar 

  12. Higham, N.J.: Matrix nearness problems and applications

    Google Scholar 

  13. Horn, B.K., Hilden, H.M., Negahdaripour, S.: Closed-form solution of absolute orientation using orthonormal matrices. JOSA A 5(7), 1127–1135 (1988)

    Article  MathSciNet  Google Scholar 

  14. Jiang, C., et al.: Learning numeral embeddings. arXiv preprint arXiv:2001.00003 (2019)

  15. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  16. Kutuzov, A., Velldal, E., Øvrelid, L.: Redefining part-of-speech classes with distributional semantic models. arXiv preprint arXiv:1608.03803 (2016)

  17. Luong, M.T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025 (2015)

  18. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  19. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018) (2018)

    Google Scholar 

  20. Mimno, D., Thompson, L.: The strange geometry of skip-gram with negative sampling. In: Empirical Methods in Natural Language Processing (2017)

    Google Scholar 

  21. Naik, A., Ravichander, A., Rose, C., Hovy, E.: Exploring numeracy in word embeddings. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3374–3380 (2019)

    Google Scholar 

  22. Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  23. Ravichander, A., Naik, A., Rose, C., Hovy, E.: Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference. arXiv preprint arXiv:1901.03735 (2019)

  24. Saxton, D., Grefenstette, E., Hill, F., Kohli, P.: Analysing mathematical reasoning abilities of neural models. arXiv preprint arXiv:1904.01557 (2019)

  25. Şenel, L.K., Utlu, I., Yücesoy, V., Koc, A., Cukur, T.: Semantic structure and interpretability of word embeddings. IEEE/ACM Trans. Audio Speech Lang. Process. 26(10), 1769–1779 (2018)

    Article  Google Scholar 

  26. Trask, A., Hill, F., Reed, S.E., Rae, J., Dyer, C., Blunsom, P.: Neural arithmetic logic units. In: Advances in Neural Information Processing Systems, pp. 8035–8044 (2018)

    Google Scholar 

  27. Wallace, E., Wang, Y., Li, S., Singh, S., Gardner, M.: Do NLP models know numbers? Probing numeracy in embeddings. arXiv preprint arXiv:1909.07940 (2019)

  28. Wen, T.H., Gasic, M., Mrksic, N., Su, P.H., Vandyke, D., Young, S.: Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745 (2015)

  29. Whalen, J., Gallistel, C.R., Gelman, R.: Nonverbal counting in humans: the psychophysics of number representation. Psychol. Sci. 10(2), 130–137 (1999)

    Article  Google Scholar 

  30. Wu, Y., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)

  31. Yang, Y., Birnbaum, L., Wang, J.P., Downey, D.: Extracting commonsense properties from embeddings with limited human guidance. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 644–649 (2018)

    Google Scholar 

Download references

Acknowledgements

We would like to thank the anonymous reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, China

    Yuanhang Ren

  2. Southwestern University of Finance and Economics, Chengdu, China

    Ye Du

Authors
  1. Yuanhang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ye Du
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ye Du .

Editor information

Editors and Affiliations

  1. ECE & Ingenuity Labs Research Institute, Queen’s University, Kingston, ON, Canada

    Xiaodan Zhu

  2. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Min Zhang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Yu Hong

  4. College of Intelligence and Computing, Tianjin University, Tianjin, China

    Ruifang He

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ren, Y., Du, Y. (2020). Enhancing the Numeracy of Word Embeddings: A Linear Algebraic Perspective. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60450-9_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60449-3

  • Online ISBN: 978-3-030-60450-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation