Skip to main content
SpringerLink
Log in
Book cover

International Semantic Web Conference

ISWC 2020: The Semantic Web – ISWC 2020 pp 619–635Cite as

Detecting Different Forms of Semantic Shift in Word Embeddings via Paradigmatic and Syntagmatic Association Changes

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12506)

Abstract

Automatically detecting semantic shifts (i.e., meaning changes) of single words has recently received strong research attention, e.g., to quantify the impact of real-world events on online communities. These computational approaches have introduced various measures, which are intended to capture the somewhat elusive and undifferentiated concept of semantic shift. On the other hand, there is a longstanding and well established distinction in linguistics between a word’s paradigmatic (i.e., terms that can replace a word) and syntagmatic associations (i.e., terms that typically occur next to a word). In this work, we join these two lines of research by introducing a method that captures a measure’s sensitivity for paradigmatic and/or syntagmatic (association) shifts. For this purpose, we perform synthetic distortions on textual corpora that in turn induce shifts in word embeddings trained on them. We find that the Local Neighborhood is sensitive to paradigmatic and the Global Semantic Displacement is sensitive to syntagmatic shift in word embeddings. By applying the newly validated paradigmatic and syntagmatic measures on three real-world datasets (Amazon, Reddit and Wikipedia) we find examples of words that undergo paradigmatic and syntagmatic shift both separately and at the same time. With this more nuanced understanding of semantic shift on word embeddings, we hope to analyze a similar concept of semantic shift on RDF graph embeddings in the future.

Keywords

  • Semantic shift detection
  • Paradigmatic associations
  • Syntagmatic associations
  • RDF embedding shift

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Baumgartner, J.: Reddit dataset, https://files.pushshift.io/reddit/, (accessed on 2019-09-25.

  2. 2.

    wikimedia: wikipedia snapshots on archive.org, https://archive.org/download/enwiki-20150112, https://archive.org/download/enwiki-20160113, https://archive.org/download/enwiki-20170101, https://archive.org/download/enwiki-20180101, https://archive.org/details/enwiki-20190120, (accessed on 2019-09-25).

  3. 3.

    google: word2vec documentation, https://code.google.com/archive/p/word2vec/, (accessed on 2019-09-25).

  4. 4.

    Wikipedia: Fifty shades of grey, https://en.wikipedia.org/wiki/Fifty_Shades_of_Grey (accessed on 2019-10-14).

  5. 5.

    Amazon: amazon search for “fifty”, https://www.amazon.com/s?k=fifty&ref=nb_sb_noss (accessed on 2019-09-18).

  6. 6.

    Darksouls.fandom.com: Shulva, Sanctum City, https://darksouls.fandom.com/wiki/Shulva,_Sanctum_City (accessed on 2019-09-30).

  7. 7.

    Darksouls.fandom.com: Forest of fallen giants, https://darksouls.fandom.com/wiki/Forest_of_Fallen_Giants (accessed on 2019-09-30).

References

  1. Bloomfield, L.: Language. Allen & Unwin, London (1933)

    Google Scholar 

  2. Bréal, M.: Essai de sémantique. Lambert-Lucas (1897)

    Google Scholar 

  3. Camacho-Collados, J., Pilehvar, M.T.: On the role of text preprocessing in neural network architectures: an evaluation study on text categorization and sentiment analysis. In: EMNLP Workshop, pp. 40–46 (2018)

    Google Scholar 

  4. Cochez, M., Ristoski, P., Ponzetto, S.P., Paulheim, H.: Global RDF vector space embeddings. In: d’Amato, C., et al. (eds.) ISWC 2017. LNCS, vol. 10587, pp. 190–207. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68288-4_12

    CrossRef  Google Scholar 

  5. Cruse, A.: Meaning in Language. Oxford Linguistics (2004)

    Google Scholar 

  6. Dell, G.S., Oppenheim, G.M., Kittredge, A.K.: Saying the right word at the right time: syntagmatic and paradigmatic interference in sentence production. Lang. Cogn. Processes 23(4), 583–608 (2008)

    CrossRef  Google Scholar 

  7. Dubossarsky, H., Grossman, E., Weinshal, D.: Outta control: laws of semantic change and inherent biases in word representation models. In: EMNLP, pp. 1136–1145 (2017)

    Google Scholar 

  8. Eger, S., Mehler, A.: On the linearity of semantic change: investigating meaning variation via dynamic graph models. In: ACL, pp. 52–58 (2016)

    Google Scholar 

  9. Frermann, L., Lapata, M.: A Bayesian model of diachronic meaning change. In: ACL, pp. 31–45 (2016)

    Google Scholar 

  10. Gutierrez, C., Hurtado, C., Vaisman, A.: Temporal RDF. In: Gómez-Pérez, A., Euzenat, J. (eds.) ESWC 2005. LNCS, vol. 3532, pp. 93–107. Springer, Heidelberg (2005). https://doi.org/10.1007/11431053_7

    CrossRef  Google Scholar 

  11. Hamilton, W.L., Leskovec, J., Jurafsky, D.: Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In: EMNLP, pp. 2116–2121 (2016)

    Google Scholar 

  12. Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. In: ACL, pp. 1489–1501 (2016)

    Google Scholar 

  13. He, R., McAuley, J.: Ups and downs: modeling the visual evolution of fashion trends with one-class collaborative filtering. In: WWW, pp. 507–517 (2016)

    Google Scholar 

  14. Jatowt, A., Duh, K.: A framework for analyzing semantic change of words across time. In: JCDL, pp. 229–238 (2014)

    Google Scholar 

  15. Khurana, U., Deshpande, A.: Efficient snapshot retrieval over historical graph data. In: ICDE, pp. 997–1008 (2005)

    Google Scholar 

  16. Kim, Y., Chiu, Y., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: ACL Workshop, pp. 61–65 (2014)

    Google Scholar 

  17. Kulkarni, V., Al-Rfou, R., Perozzi, B., Skiena, S.: Statistically significant detection of linguistic change. In: WWW, pp. 625–635 (2015)

    Google Scholar 

  18. Kutuzov, A., Øvrelid, L., Szymanski, T., Velldal, E.: Diachronic word embeddings and semantic shifts: a survey. In: COLING, pp. 1384–1397 (2018)

    Google Scholar 

  19. Mikolov, T., Sutskever, I., Chen, K., Corradom, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: NIPS, pp. 3111–3119 (2013)

    Google Scholar 

  20. Pierrejean, B., Tanguy, L.: Predicting word embeddings variability. In: SEM, pp. 154–159 (2018)

    Google Scholar 

  21. del Prado Martin, F., Brendel, C.: Case and cause in Icelandic: reconstructing causal networks of cascaded language changes. In: ACL, pp. 2421–2430 (2016)

    Google Scholar 

  22. Ristoski, P., Rosati, J., Di Noia, T., De Leone, R., Paulheim, H.: RDF2Vec: RDF graph embeddings and their applications. Semant. Web J. 10, 721–752 (2019)

    CrossRef  Google Scholar 

  23. Rohrdantz, C., Hautli, A., Mayer, T., Butt, M., Keim, D., Plank, F.: Towards tracking semantic change by visual analytics. In: ACL, pp. 305–310 (2011)

    Google Scholar 

  24. Rosenfeld, A., Erk, K.: Deep neural models of semantic shift. In: NAACL-HLT, pp. 474–484 (2018)

    Google Scholar 

  25. Sahlgren, M.: The word-space model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Ph.D. dissertation (2006)

    Google Scholar 

  26. de Saussure, F.: Cours de linguistique generale. Payot, Paris (1916)

    Google Scholar 

  27. Schütze, H., Pedersen, J.: A vector model for syntagmatic and paradigmatic relatedness. In: Conference of the UW Centre for the New OED and Text Research, pp. 104–113 (1993)

    Google Scholar 

  28. Shoemark, P., Liza, F.F., Nguyen, D., Hale, S.A., McGillivray, B.: Room to Glo: a systematic comparison of semantic change detection approaches with word embeddings. In: EMNLP and IJCNLP, pp. 66–76 (2019)

    Google Scholar 

  29. Sokolova, L.V., Cherkasova, A.S.: Spatiotemporal organization of bioelectrical brain activity during reading of syntagmatic and paradigmatic collocations by students with different foreign language proficiency. Hum. Physiol. 41(6), 583–592 (2015). https://doi.org/10.1134/S0362119715060092

    CrossRef  Google Scholar 

  30. Stewart, I., Arendt, D., Bell, E., Volkova, S.: Measuring, predicting and visualizing short-term change in word representation and usage in VKontakte social network. In: ICWSM, pp. 672–675 (2017)

    Google Scholar 

  31. Sun, F., Guo, J., Lan, Y., Xu, J., Cheng, X.: Learning word representations by jointly modeling syntagmatic and paradigmatic relations. In: ACL and IJCNLP, pp. 136–145 (2015)

    Google Scholar 

  32. Tahmasebi, N., Borin, L., Jatowt, A.: Survey of computational approaches to lexical semantic change detection. In: ACL, pp. 31–45 (2018)

    Google Scholar 

  33. Traugott, E.C., Dasher, R.B.: Regularity in Semantic Change. Cambridge University Press, Cambridge (2001)

    CrossRef  Google Scholar 

  34. Wendlandt, L., Kummerfeld, J.K., Mihalcea, R.: Factors influencing the surprising instability of word embeddings. In: NAACL-HLT, pp. 2092–2102 (2018)

    Google Scholar 

Download references

Acknowledgments

Part of the simulations were performed with computing resources granted by RWTH Aachen University. We thank Dong Nguyen for providing advise regarding this work and our (meta-) reviewers for their constructive feedback.

Author information

Authors and Affiliations

  1. RWTH Aachen University, 52062, Aachen, Germany

    Anna Wegmann, Florian Lemmerich & Markus Strohmaier

  2. GESIS - Leibniz Institute for the Social Sciences, 50667, Cologne, Germany

    Markus Strohmaier

Authors
  1. Anna Wegmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Florian Lemmerich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markus Strohmaier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anna Wegmann .

Editor information

Editors and Affiliations

  1. University of Edinburgh, Edinburgh, UK

    Jeff Z. Pan

  2. University of Liverpool, Liverpool, UK

    Valentina Tamma

  3. University of Bari, Bari, Italy

    Claudia d’Amato

  4. University of California, Santa Barbara, Santa Barbara, CA, USA

    Krzysztof Janowicz

  5. California State University, Long Beach, Long Beach, CA, USA

    Bo Fu

  6. Vienna University of Economics and Business, Vienna, Austria

    Axel Polleres

  7. Rensselaer Polytechnic Institute, Troy, NY, USA

    Oshani Seneviratne

  8. Massachusetts Institute of Technology, Cambridge, MA, USA

    Lalana Kagal

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wegmann, A., Lemmerich, F., Strohmaier, M. (2020). Detecting Different Forms of Semantic Shift in Word Embeddings via Paradigmatic and Syntagmatic Association Changes. In: Pan, J.Z., et al. The Semantic Web – ISWC 2020. ISWC 2020. Lecture Notes in Computer Science(), vol 12506. Springer, Cham. https://doi.org/10.1007/978-3-030-62419-4_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-62419-4_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-62418-7

  • Online ISBN: 978-3-030-62419-4

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation