Abstract
Language evolves over time with trends and shifts in technological, political, or cultural contexts. Capturing these variations is important to develop better language models. While recent works tackle temporal drifts by learning diachronic embeddings, we instead propose to integrate a temporal component into a recurrent language model. It takes the form of global latent variables, which are structured in time by a learned non-linear transition function. We perform experiments on three time-annotated corpora. Experimental results on language modeling and classification tasks show that our model performs consistently better than temporal word embedding methods in two temporal evaluation settings: prediction and modeling. Moreover, we empirically show that the system is able to predict informative latent representations in the future.
This work has been partially supported by the ANR (French National Research Agency) LOCUST project (ANR-15-CE23-0027).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Supplementary material available at https://github.com/edouardelasalles/drlm/raw/master/supplementary.pdf.
- 2.
Code of the models available at https://github.com/edouardelasalles/drlm.
- 3.
References
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271 (2018)
Bamler, R., Mandt, S.: Dynamic word embeddings. In: ICML (2017)
Blei, D.M., Lafferty, J.D.: Dynamic topic models. In: ICML (2006)
Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A., Jozefowicz, R., Bengio, S.: Generating sentences from a continuous space. In: SIGNLL (2016)
Chiu, C.C., et al.: State-of-the-art speech recognition with sequence-to-sequence models. arXiv:1712.01769 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
Eger, S., Mehler, A.: On the linearity of semantic change: investigating meaning variation via dynamic graph models. In: ACL (2016)
Fedus, W., Goodfellow, I., Dai, A.M.: MaskGAN: better text generation via filling in the \(\_\). In: ICLR (2018)
Fraccaro, M., Sønderby, S.K., Paquet, U., Winther, O.: Sequential neural models with stochastic layers. In: NeurIPS (2016)
Frermann, L., Lapata, M.: A Bayesian model of diachronic meaning change. ACL (2016)
Hamilton, W.L., Leskovec, J., Jurafsky, D.: Diachronic word embeddings reveal statistical laws of semantic change. In: ACL, vol. 1 (2016)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: ACL (2018)
Iwata, T., Yamada, T., Sakurai, Y., Ueda, N.: Sequential modeling of topic dynamics with multiple timescales. ACM Trans. KDD 5, 19 (2012)
Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv:1607.01759 (2016)
Kabán, A., Girolami, M.A.: A dynamic probabilistic model to visualise topic evolution in text streams. J. Intell. Inf. Syst. 18, 107–125 (2002)
Kim, Y., Chiu, Y.I., Hanaki, K., Hegde, D., Petrov, S.: Temporal analysis of language through neural language models. In: ACL Workshop on Language Technologies and Computational Social Science (2014)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)
Krishnan, R.G., Shalit, U., Sontag, D.: Structured inference networks for nonlinear state space models. In: AAAI (2017)
Kulkarni, V., Al-Rfou, R., Perozzi, B., Skiena, S.: Statistically significant detection of linguistic change. In: WWW (2015)
Melis, G., Dyer, C., Blunsom, P.: On the state of the art of evaluation in neural language models. In: ICLR (2018)
Merity, S., Keskar, N.S., Socher, R.: An analysis of neural language modeling at multiple scales. arXiv:1803.08240 (2018)
Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. In: ICRL (2018)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)
Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: ISCA (2010)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: ICML (2014)
Rosenfeld, A., Erk, K.: Deep neural models of semantic shift. In: NAACL (2018)
Rudolph, M., Blei, D.: Dynamic Bernoulli embeddings for language evolution. arXiv:1703.08052 (2017)
Semeniuta, S., Severyn, A., Barth, E.: A hybrid convolutional variational autoencoder for text generation. In: EMNLP (2017)
Serban, I.V., Ororbia, A.G., Pineau, J., Courville, A.: Piecewise latent variables for neural variational text processing. In: EMNLP (2017)
Shen, D., et al.: Baseline needs more love: on simple word-embedding-based models and associated pooling mechanisms. arXiv:1805.09843 (2018)
Tan, C., Lee, L.: All who wander: on the prevalence and characteristics of multi-community engagement. In: WWW (2015)
Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: lessons learned from the 2015 MSCOCO image captioning challenge. PAMI 39, 652–663 (2017)
Wang, C., Blei, D., Heckerman, D.: Continuous time dynamic topic models. arXiv:1206.3298 (2012)
Wang, X., McCallum, A.: Topics over time: a non-Markov continuous-time model of topical trends. In: ACM SIGKDD (2006)
Yang, Z., Hu, Z., Salakhutdinov, R., Berg-Kirkpatrick, T.: Improved variational autoencoders for text modeling using dilated convolutions. In: ICML (2017)
Yao, Z., Sun, Y., Ding, W., Rao, N., Xiong, H.: Dynamic word embeddings for evolving semantic discovery. In: WSDM (2018)
Zaheer, M., Ahmed, A., Smola, A.J.: Latent LSTM allocation joint clustering and non-linear dynamic modeling of sequential data. In: ICML (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Delasalles, E., Lamprier, S., Denoyer, L. (2019). Dynamic Neural Language Models. In: Gedeon, T., Wong, K., Lee, M. (eds) Neural Information Processing. ICONIP 2019. Lecture Notes in Computer Science(), vol 11955. Springer, Cham. https://doi.org/10.1007/978-3-030-36718-3_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-36718-3_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36717-6
Online ISBN: 978-3-030-36718-3
eBook Packages: Computer ScienceComputer Science (R0)