Skip to main content

Seq2VAR: Multivariate Time Series Representation with Relational Neural Networks and Linear Autoregressive Model

  • Conference paper
  • First Online:
Advanced Analytics and Learning on Temporal Data (AALTD 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11986))

  • 778 Accesses

Abstract

Finding understandable and meaningful feature representation of multivariate time series (MTS) is a difficult task, since information is entangled both in temporal and spatial dimensions. In particular, MTS can be seen as the observation of simultaneous causal interactions between dynamical variables. Standard way to model these interactions is the vector linear autoregression (VAR). The parameters of VAR models can be used as MTS feature representation. Yet, VAR cannot generalize on new samples, hence independent VAR models must be trained to represent different MTS. In this paper, we propose to use the inference capacity of neural networks to overpass this limit. We propose to associate a relational neural network to a VAR generative model to form an encoder-decoder of MTS. The model is denoted Seq2VAR for Sequence-to-VAR. We use recent advances in relational neural network to build our MTS encoder by explicitly modeling interactions between variables of MTS samples. We also propose to leverage reparametrization tricks for binomial sampling in neural networks in order to build a sparse version of Seq2VAR and find back the notion of Granger causality defined in sparse VAR models. We illustrate the interest of our approach through experiments on synthetic datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Davis, R.A., Zang, P., Zheng, T.: Sparse vector autoregressive modeling. J. Comput. Graph. Stat. 25(4), 1077–1096 (2016)

    Article  MathSciNet  Google Scholar 

  2. Eichler, M., Didelez, V.: Causal reasoning in graphical time series models. arXiv preprint arXiv:1206.5246 (2012)

  3. Esling, P., Agon, C.: Time-series data mining. ACM Comput. Surv. (CSUR) 45(1), 12 (2012)

    Article  Google Scholar 

  4. Gong, M., Zhang, K., Schoelkopf, B., Tao, D., Geiger, P.: Discovering temporal causal relations from subsampled data. In: International Conference on Machine Learning, pp. 1898–1906 (2015)

    Google Scholar 

  5. Haufe, S., Müller, K.-R., Nolte, G., Krämer, N., Sparse causal discovery in multivariate time series. In: Causality: Objectives and Assessment, pp. 97–106 (2010)

    Google Scholar 

  6. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)

  7. Kaiser, Ł., Bengio, S.: Discrete autoencoders for sequence models. arXiv preprint arXiv:1801.09797 (2018)

  8. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  9. Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., Zemel, R.: Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04687 (2018)

  10. Larochelle, H., Murray, I.: The neural autoregressive distribution estimator. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29–37 (2011)

    Google Scholar 

  11. Li, Z., et al.: Towards binary-valued gates for robust LSTM training. arXiv preprint arXiv:1806.02988 (2018)

  12. Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811.12359 (2018)

  13. Louizos, C., Welling, M., Kingma, D.P.: Learning sparse neural networks through \( l\_0 \) regularization. arXiv preprint arXiv:1712.01312 (2017)

  14. Ma, X., Zhou, C., Hovy, E.: MAE: mutual posterior-divergence regularization for variational autoencoders. arXiv preprint arXiv:1901.01498 (2019)

  15. Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)

  16. Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182 (2017)

  17. Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)

    Google Scholar 

  18. Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems, pp. 4967–4976 (2017)

    Google Scholar 

  19. Seabold, S., Perktold, J.: Statsmodels: econometric and statistical modeling with python. In: Proceedings of the 9th Python in Science Conference, vol. 57, p. 61. Scipy (2010)

    Google Scholar 

  20. Shen, X., Su, H., Niu, S., Demberg, V.: Improving variational encoder-decoders in dialogue generation. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  21. Toda, H.Y., Phillips, P.C.B.: Vector autoregression and causality: a theoretical overview and simulation study. Econ. Rev. 13(2), 259–285 (1994)

    Article  MathSciNet  Google Scholar 

  22. Tschannen, M., Bachem, O., Lucic, M.: Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069 (2018)

  23. Yi, S., Pavlovic, V.: Sparse granger causality graphs for human action classification. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 3374–3377. IEEE (2012)

    Google Scholar 

  24. Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014)

Download references

Acknowledgments

This work is supported by the company Safran through the CIFRE convention 2017/1317.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Edouard Pineau .

Editor information

Editors and Affiliations

A Appendix: Hyperparameters

A Appendix: Hyperparameters

For our Seq2VAR, we used a succession of 2-layers perceptrons for our relational encoder, as in NRI [9]. The parameters to chose concern latent dimension for all layers and the temperature parameter \(\tau \) for the relaxed binary sampling (see Sect. 2.3). They are presented in Table 4.

Table 4. Training parameters.

For NRI, all other parameters are the one of the original paper for the homogeneous springs rigidity except for the dimensionality of the latent space which we set to 64 instead of 256, since in the experimental setup of the original paper, it gives the same results while diminishing computing time and memory needs. For the heterogeneous rigidity, the parameter prediction_steps is set to 5 instead of 10 and \(\tau \) is set to 0.1. These parameters gave the best average results. In fact, due to the highly expressive form of its decoder, NRI was able to build good predictor with not the good graph. We played with parameters to get more stable and better results. For the experiments, we also tried to change the skip first parameter that is set to False or True in the original paper [9], depending on the dataset studied. It did not change the results of the experiments.

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Pineau, E., Razakarivony, S., Bonald, T. (2020). Seq2VAR: Multivariate Time Series Representation with Relational Neural Networks and Linear Autoregressive Model. In: Lemaire, V., Malinowski, S., Bagnall, A., Bondu, A., Guyet, T., Tavenard, R. (eds) Advanced Analytics and Learning on Temporal Data. AALTD 2019. Lecture Notes in Computer Science(), vol 11986. Springer, Cham. https://doi.org/10.1007/978-3-030-39098-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-39098-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-39097-6

  • Online ISBN: 978-3-030-39098-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics