Skip to main content
SpringerLink
Log in

Knowledge-enhanced temporal word embedding for diachronic semantic change estimation

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Historical, social and linguistic factors cause semantic changes that can narrow, broaden or completely alter the meanings of words. Frequency, syntactic and semantic variations are to be studied to examine such changes. Syntactic changes cannot be observed in many cases, if words have no POS variation. Context and connotation contribute more to semantic alteration. In addition, words have similar or related meanings in certain contexts, and the context is considered with diverse features such as co-occurrence and word association. The semantic change is generally related to the variation in n-gram context with a maximum of 5 g. However, distant context terms also play a prominent role in semantic change. There is also a link between the type of change and the use of lexical relations. This paper builds a knowledge-enhanced temporal word embedding model that utilizes ‘word-centric dependency relations’ for capturing context words irrespective of their n-gram position and ‘syntactic patterns for lexical relations’ for determining the type of semantic change. The joint learning of contexts with both dependency and lexical relations from diachronic corpora is performed to obtain temporal word embedding vectors. The proposed model outperforms other n-gram-based approaches when evaluated with standard diachronic corpora.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://google.com/ngram/viewer, http://google.com/ngram/trends.

  2. http://storage.googleapis.com/books/ngrams/books/datasetsv2.html.

  3. https://googlebooks.byu.edu/x.asp.

  4. https://corpus.byu.edu/Times.

  5. https://corpus.byu.edu/coha/.

  6. https://gate.ac.uk/.

  7. http://bitbucket.org/yoavgo//word2vecf.

  8. http://mattmahoney.net/dc/text8.zip.

  9. http://www.atlas-semantiques.eu/?l=EN.

  10. https://code.google.com/archive/p/word2vec/source/default/source/browse/trunk/questions-words.txt.

  11. https://devopedia.org/wordnet.

References

  • Allen TT, Sui Z, Parker NL (2017) Timely decision analysis enabled by efficient social media modeling. Decis Anal 14(4):250–260

    Article  MathSciNet  MATH  Google Scholar 

  • Allen TT, Sui Z, Akbari K (2018) Exploratory text data analysis for quality hypothesis generation. Qual Eng 30(4):701–712

    Article  Google Scholar 

  • Bansal M, Gimpel K, Livescu K (2014) Tailoring continuous word representations for dependency parsing. In: ACL (2) (pp. 809–815)

  • Berland M, Charniak E (1999) Finding parts in very large corpora. In: Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics. Association for Computational Linguistics, pp 57–64

  • Bian J, Gao B, Liu TY (2014) Knowledge-powered deep learning for word embedding. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, Berlin, pp 132–148

  • Boukhaled M, Fagard B, Poibeau T (2019) Modelling the semantic change dynamics using diachronic word embedding

  • Church KW, Hanks P (1990) Word association norms, mutual information, and lexicography. Comput Linguist 16(1):22–29

    Google Scholar 

  • Cook P, Stevenson S (2010) Automatically identifying changes in the semantic orientation of words. In: LREC

  • Cui H, Sun R, Li K, Kan MY, Chua TS (2005) Question answering passage retrieval using dependency relations. In: Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval. ACM, pp 400–407

  • Davies M (2014) Making Google Books n-grams useful for a wide range of research on language change. Int J Corpus Linguist 19(3):401–416

    Article  Google Scholar 

  • De Marneffe MC, Manning CD (2008) Stanford typed dependencies manual. Technical report, Stanford University, pp 338–345

  • Dubossarsky H, Tsvetkov Y, Dyer C, Grossman E (2015) A bottom up approach to category mapping and meaning change. In: NetWordS, pp 66–70

  • Gulordava K, Baroni M (2011) A distributional similarity approach to the detection of semantic change in the Google Books Ngram corpus. In: Proceedings of the GEMS 2011 workshop on geometrical models of natural language semantics. Association for Computational Linguistics, pp 67–71

  • Gupta S, MacLean DL, Heer J, Manning CD (2014) Induced lexico-syntactic patterns improve information extraction from online medical forums. J Am Med Inf Assoc 21(5):902–909

    Article  Google Scholar 

  • Hamilton WL, Leskovec J, Jurafsky D (2016a) Diachronic word embeddings reveal statistical laws of semantic change. arXiv preprint arXiv:1605.09096

  • Hamilton WL, Leskovec J, Jurafsky D (2016b) Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In: Proceedings of the conference on empirical methods in natural language processing, vol 2016. NIH Public Access, p 2116

  • Hearst MA (1992) Automatic acquisition of hyponyms from large text corpora. In: Proceedings of the 14th conference on computational linguistics, vol 2. Association for Computational Linguistics, pp 539–545

  • Jatowt A, Duh K (2014) A framework for analyzing semantic change of words across time. In: Proceedings of the 14th ACM/IEEE-CS joint conference on digital libraries. IEEE Press, pp. 229–238

  • Kim Y, Chiu YI, Hanaki K, Hegde D, Petrov S (2014) Temporal analysis of language through neural language models. arXiv preprint arXiv:1405.3515

  • Komninos A, Manandhar S (2016) Dependency based embeddings for sentence classification tasks. In HLT-NAACL, pp 1490–1500

  • Kulkarni V, Al-Rfou R, Perozzi B, Skiena S (2015) Statistically significant detection of linguistic change. In: Proceedings of the 24th international conference on World Wide Web. International World Wide Web Conferences Steering Committee, pp 625–635

  • Kutuzov A, Ovrelid L, Szymanski T, Velldal E (2018) Diachronic word embeddings and semantic shifts: a survey. arXiv preprint arXiv:1806.03537

  • Levy O, Goldberg Y (2014) Dependency-based word embeddings. In: ACL (2), pp 302–308

  • Li Q, Li T, Chang B (2016) Learning word sense embeddings from word sense definitions. In: International conference on computer processing of oriental languages. Springer, Berlin, pp 224–235

  • Lieberman E, Michel JB, Jackson J, Tang T, Nowak MA (2007) Quantifying the evolutionary dynamics of language. Nature 449(7163):713

    Article  Google Scholar 

  • Lim S, Lee C, Ra D (2013) Dependency-based semantic role labeling using sequence labeling with a structural SVM. Pattern Recognit Lett 34(6):696–702

    Article  Google Scholar 

  • Lin D, Zhao S, Qin L, Zhou M (2003) Identifying synonyms among distributionally similar words. In: IJCAI, vol 3, pp 1492–1493

  • Maybaum R (2013) Language change as a social process: diffusion patterns of lexical innovations in Twitter. In: Annual Meeting of the Berkeley Linguistics Society, vol 39, no. 1, pp 152–166

  • Melamud O, McClosky D, Patwardhan S, Bansal M (2016) The role of context types and dimensionality in learning word embeddings (2016). arXiv preprint arXiv:1601.00893

  • Michel JB, Shen YK, Aiden AP, Veres A, Gray MK, Pickett JP, Pinker S (2011) Quantitative analysis of culture using millions of digitized books. Science 331(6014):176–182

    Article  Google Scholar 

  • Mitra S, Mitra R, Riedl M, Biemann C, Mukherjee A, Goyal P (2014) That’s sick dude!: automatic identification of word sense change across different timescales. arXiv preprint arXiv:1405.4392

  • Nastase V, Shirabad JS, Caropreso MF (2007) Using dependency relations for text classification. University of Ottawa SITE Technical Report TR-2007-12, 13

  • Niu L, Dai XY, Huang S, Chen J (2016) A unified framework for jointly learning distributed representations of word and attributes. In: Asian conference on machine learning, pp 143–156

  • Pelevina M, Arefyev N, Biemann C, Panchenko A (2017) Making sense of word embeddings. arXiv preprint arXiv:1708.03390

  • Phillips L, Shaffer K, Arendt D, Hodas N, Volkova S (2017) Intrinsic and extrinsic evaluation of spatiotemporal text representations in Twitter streams. In: Proceedings of the 2nd workshop on representation learning for NLP, pp 201–210

  • Ploux S, Boussidan A, Ji H (2010) The semantic atlas: an interactive model of lexical representation. In: Proceedings of the sixth conference of international language resources (ELRA), pp 1–5

  • Polajnar T, Clark S (2014) Improving distributional semantic vectors through context selection and normalisation. In: EACL, pp 230–238

  • Roller S, Erk K (2016) Relations such as hypernymy: identifying and exploiting hearst patterns in distributional vectors for lexical entailment. arXiv preprint arXiv:1605.05433

  • Sagi E, Kaufmann S, Clark B (2011) Tracing semantic change with latent semantic analysis. Curr Methods Hist Seman 73:161–183

    Google Scholar 

  • Samha AK (2016) Aspect-based opinion mining using dependency relations. Int J Comput Sci Trends Technol 4(1):113–123

    Google Scholar 

  • Schouten K, Baas F, Bus O, Osinga A, van de Ven N, van Loenhout S et al (2016) Aspect-based sentiment analysis using Lexico-semantic patterns. In: International conference on web information systems engineering. Springer, Berlin, pp 35–42

  • Schwartz R, Reichart R, Rappoport A (2015) Symmetric pattern based word embeddings for improved word similarity prediction. In: CoNLL, vol 2015, pp 258–267

  • Seitner J, Bizer C, Eckert K, Faralli S, Meusel R, Paulheim H, Ponzetto SP (2016) A large database of hypernymy relations extracted from the web. In: LREC

  • Stewart I, Arendt D, Bell E, Volkova S (2017) Measuring, predicting and visualizing short-term change in word representation and usage in social network. In: Eleventh international AAAI conference on web and social media

  • Sui Z (2019) Social media text data visualization modeling: a timely topic score technique. Am J Manag Sci Eng 4(3):49–55

    Article  Google Scholar 

  • Tang X (2018) A state-of-the-art of semantic change computation. Nat Lang Eng 24(5):649–676

    Article  Google Scholar 

  • Tang X, Qu W, Chen X (2016) Semantic change computation: a successive approach. World Wide Web 19(3):375–415

    Article  Google Scholar 

  • Tovar M, Pinto D, Montes A, González G, Vilarino D, Beltrán B (2014) Use of lexico-syntactic patterns for the evaluation of taxonomic relations. In: Mexican conference on pattern recognition. Springer, Cham, pp 331–340

  • Wijaya DT, Yeniterzi R (2011) Understanding semantic change of words over centuries. In: Proceedings of the 2011 international workshop on detecting and exploiting cultural diversity on the social web. ACM, pp 35–40

  • Yamaguchi K (2014) How do typological studies explain the semantic changes of english complex prepositions? Top Linguist 13(1):60–66

    Google Scholar 

  • Zhao Y, Huang S, Dai X, Zhang J, Chen J (2014) Learning word embeddings from dependency relations. In: 2014 international conference on Asian language processing (IALP). IEEE, pp 123–127

Download references

Author information

Authors and Affiliations

  1. Department of CSE, Anna University, Chennai, India

    J. Vijayarani & T. V. Geetha

Authors
  1. J. Vijayarani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. V. Geetha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. Vijayarani.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Communicated by V. Loia.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vijayarani, J., Geetha, T.V. Knowledge-enhanced temporal word embedding for diachronic semantic change estimation. Soft Comput 24, 12901–12918 (2020). https://doi.org/10.1007/s00500-020-04714-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-020-04714-0

Keywords

Search

Navigation