Advertisement

Sidecar: Augmenting Word Embedding Models with Expert Knowledge

  • Mathieu Lemay
  • Daniel ShapiroEmail author
  • Mary Kate MacPherson
  • Kieran Yee
  • Hamza Qassoud
  • Miodrag Bolic
Conference paper
  • 19 Downloads
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1130)

Abstract

This work investigates a method for enriching pre-trained word embeddings with domain-specific information using a small, custom word embedding. For a classification task on text containing out-of-vocabulary expert jargon, this new approach improves the prediction accuracy when using popular models such as Word2Vec (71.5% to 76.6%), GloVe (73.5% to 77.2%), and fastText (75.8% to 79.6%). Furthermore, an analysis of the approach demonstrates that expert knowledge is improved in terms of discrimination and inconsistency. Another advantage of this word embedding augmentation technique is that it is computationally inexpensive and leverages the general syntactic information encoded in large pre-trained word embeddings.

Keywords

Transfer learning Word embedding Expertise Knowledge Embedding retraining 

References

  1. 1.
    Anders Ericsson, K., Charness, N.: Expert performance: its structure and acquisition. Am. Psychol. 49, 725–747 (1994)CrossRefGoogle Scholar
  2. 2.
    Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606 (2016)
  3. 3.
    Dong, J., Huang, J.: Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus. CoRR abs/1802.02614 (2018). http://arxiv.org/abs/1802.02614
  4. 4.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)CrossRefGoogle Scholar
  5. 5.
    Kiros, R., Zhu, Y., Salakhutdinov, R.R., Zemel, R., Urtasun, R., Torralba, A., Fidler, S.: Skip-thought vectors. In: Advances in Neural Information Processing Systems, pp. 3294–3302 (2015)Google Scholar
  6. 6.
    Limsopatham, N., Collier, N.: Modelling the combination of generic and target domain embeddings in a convolutional neural network for sentence classification. Association for Computational Linguistics (2016)Google Scholar
  7. 7.
    McBride, M.F., Burgman, M.A.: What is expert knowledge, how is such knowledge gathered, and how do we use it to address questions in landscape ecology? In: Perera, A., Drew, C., Johnson, C. (eds.) Expert Knowledge and Its Application in Landscape Ecology, pp. 11–38. Springer, New York (2012)CrossRefGoogle Scholar
  8. 8.
    Merriam-Webster Online: Merriam-Webster Online Dictionary (2009). http://www.merriam-webster.com
  9. 9.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  10. 10.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  11. 11.
    Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
  12. 12.
    Saal, F.E., Downey, R.G., Lahey, M.A.: Rating the ratings: assessing the psychometric quality of rating data. Psychol. Bull. 88(2), 413 (1980)CrossRefGoogle Scholar
  13. 13.
    Shanteau, J., Weiss, D., Thomas, R., Pounds, J.: Performance-based assessment of expertise: how to decide if someone is an expert or not. Eur. J. Oper. Res. 136(2), 253–263 (2002)CrossRefGoogle Scholar
  14. 14.
    Shapiro, D.: Composing recommendations using computer screen images: a deep learning recommender system for PC users. Ph.D. thesis, Université d’Ottawa/University of Ottawa (2017)Google Scholar
  15. 15.
    Shapiro, D., Qassoud, H., Lemay, M., Bolic, M.: Visual deep learning recommender system for personal computer users. In: The Second International Conference on Applications and Systems of Visual Paradigms, VISUAL 2017, pp. 1–10 (2017). https://www.thinkmind.org/index.php?view=article&articleid=visual_2017_1_10_70006
  16. 16.
    Speer, R., Chin, J.: An ensemble method to produce high-quality word embeddings. arXiv preprint arXiv:1604.01692 (2016)
  17. 17.
    Xu, C., Bai, Y., Bian, J., Gao, B., Wang, G., Liu, X., Liu, T.Y.: RC-NET: a general framework for incorporating knowledge into word representations. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, pp. 1219–1228. ACM (2014)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Mathieu Lemay
    • 1
  • Daniel Shapiro
    • 1
    • 2
    Email author
  • Mary Kate MacPherson
    • 1
  • Kieran Yee
    • 1
  • Hamza Qassoud
    • 2
  • Miodrag Bolic
    • 2
  1. 1.Lemay.ai Inc.OttawaCanada
  2. 2.School of Electrical Engineering and Computer ScienceUniversity of OttawaOttawaCanada

Personalised recommendations