Skip to main content
Book cover

International Workshop on Advanced Analysis and Learning on Temporal Data

AALTD 2019: Advanced Analytics and Learning on Temporal Data pp 126–140Cite as

Seq2VAR: Multivariate Time Series Representation with Relational Neural Networks and Linear Autoregressive Model

  • 586 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 11986)

Abstract

Finding understandable and meaningful feature representation of multivariate time series (MTS) is a difficult task, since information is entangled both in temporal and spatial dimensions. In particular, MTS can be seen as the observation of simultaneous causal interactions between dynamical variables. Standard way to model these interactions is the vector linear autoregression (VAR). The parameters of VAR models can be used as MTS feature representation. Yet, VAR cannot generalize on new samples, hence independent VAR models must be trained to represent different MTS. In this paper, we propose to use the inference capacity of neural networks to overpass this limit. We propose to associate a relational neural network to a VAR generative model to form an encoder-decoder of MTS. The model is denoted Seq2VAR for Sequence-to-VAR. We use recent advances in relational neural network to build our MTS encoder by explicitly modeling interactions between variables of MTS samples. We also propose to leverage reparametrization tricks for binomial sampling in neural networks in order to build a sparse version of Seq2VAR and find back the notion of Granger causality defined in sparse VAR models. We illustrate the interest of our approach through experiments on synthetic datasets.

Keywords

  • Multivariate time series
  • Vector linear autoregression
  • Relational neural networks
  • Granger causality

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-39098-3_10
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   54.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-39098-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

References

  1. Davis, R.A., Zang, P., Zheng, T.: Sparse vector autoregressive modeling. J. Comput. Graph. Stat. 25(4), 1077–1096 (2016)

    CrossRef  MathSciNet  Google Scholar 

  2. Eichler, M., Didelez, V.: Causal reasoning in graphical time series models. arXiv preprint arXiv:1206.5246 (2012)

  3. Esling, P., Agon, C.: Time-series data mining. ACM Comput. Surv. (CSUR) 45(1), 12 (2012)

    CrossRef  Google Scholar 

  4. Gong, M., Zhang, K., Schoelkopf, B., Tao, D., Geiger, P.: Discovering temporal causal relations from subsampled data. In: International Conference on Machine Learning, pp. 1898–1906 (2015)

    Google Scholar 

  5. Haufe, S., Müller, K.-R., Nolte, G., Krämer, N., Sparse causal discovery in multivariate time series. In: Causality: Objectives and Assessment, pp. 97–106 (2010)

    Google Scholar 

  6. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 (2016)

  7. Kaiser, Ł., Bengio, S.: Discrete autoencoders for sequence models. arXiv preprint arXiv:1801.09797 (2018)

  8. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  9. Kipf, T., Fetaya, E., Wang, K.-C., Welling, M., Zemel, R.: Neural relational inference for interacting systems. arXiv preprint arXiv:1802.04687 (2018)

  10. Larochelle, H., Murray, I.: The neural autoregressive distribution estimator. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 29–37 (2011)

    Google Scholar 

  11. Li, Z., et al.: Towards binary-valued gates for robust LSTM training. arXiv preprint arXiv:1806.02988 (2018)

  12. Locatello, F., Bauer, S., Lucic, M., Gelly, S., Schölkopf, B., Bachem, O.: Challenging common assumptions in the unsupervised learning of disentangled representations. arXiv preprint arXiv:1811.12359 (2018)

  13. Louizos, C., Welling, M., Kingma, D.P.: Learning sparse neural networks through \( l\_0 \) regularization. arXiv preprint arXiv:1712.01312 (2017)

  14. Ma, X., Zhou, C., Hovy, E.: MAE: mutual posterior-divergence regularization for variational autoencoders. arXiv preprint arXiv:1901.01498 (2019)

  15. Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: a continuous relaxation of discrete random variables. arXiv preprint arXiv:1611.00712 (2016)

  16. Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. arXiv preprint arXiv:1708.02182 (2017)

  17. Sak, H., Senior, A., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)

    Google Scholar 

  18. Santoro, A., et al.: A simple neural network module for relational reasoning. In: Advances in Neural Information Processing Systems, pp. 4967–4976 (2017)

    Google Scholar 

  19. Seabold, S., Perktold, J.: Statsmodels: econometric and statistical modeling with python. In: Proceedings of the 9th Python in Science Conference, vol. 57, p. 61. Scipy (2010)

    Google Scholar 

  20. Shen, X., Su, H., Niu, S., Demberg, V.: Improving variational encoder-decoders in dialogue generation. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)

    Google Scholar 

  21. Toda, H.Y., Phillips, P.C.B.: Vector autoregression and causality: a theoretical overview and simulation study. Econ. Rev. 13(2), 259–285 (1994)

    CrossRef  MathSciNet  Google Scholar 

  22. Tschannen, M., Bachem, O., Lucic, M.: Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069 (2018)

  23. Yi, S., Pavlovic, V.: Sparse granger causality graphs for human action classification. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 3374–3377. IEEE (2012)

    Google Scholar 

  24. Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. arXiv preprint arXiv:1409.2329 (2014)

Download references

Acknowledgments

This work is supported by the company Safran through the CIFRE convention 2017/1317.

Author information

Authors and Affiliations

  1. Safran Tech, Signal and Information Technologies, Châteaufort, France

    Edouard Pineau & Sébastien Razakarivony

  2. Telecom Paris, Institut Polytechnique de Paris, Paris, France

    Edouard Pineau & Thomas Bonald

Authors
  1. Edouard Pineau
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sébastien Razakarivony
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Bonald
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Edouard Pineau .

Editor information

Editors and Affiliations

  1. Orange Labs, Lannion, France

    Vincent Lemaire

  2. Inria, University of Rennes, Rennes, France

    Simon Malinowski

  3. University of East Anglia, Norwich, UK

    Prof. Anthony Bagnall

  4. Orange Labs, Châtillon, France

    Alexis Bondu

  5. Irisa, Agrocampus Ouest, Rennes, France

    Dr. Thomas Guyet

  6. University of Rennes 2, Rennes, France

    Romain Tavenard

A Appendix: Hyperparameters

A Appendix: Hyperparameters

For our Seq2VAR, we used a succession of 2-layers perceptrons for our relational encoder, as in NRI [9]. The parameters to chose concern latent dimension for all layers and the temperature parameter \(\tau \) for the relaxed binary sampling (see Sect. 2.3). They are presented in Table 4.

Table 4. Training parameters.
Full size table

For NRI, all other parameters are the one of the original paper for the homogeneous springs rigidity except for the dimensionality of the latent space which we set to 64 instead of 256, since in the experimental setup of the original paper, it gives the same results while diminishing computing time and memory needs. For the heterogeneous rigidity, the parameter prediction_steps is set to 5 instead of 10 and \(\tau \) is set to 0.1. These parameters gave the best average results. In fact, due to the highly expressive form of its decoder, NRI was able to build good predictor with not the good graph. We played with parameters to get more stable and better results. For the experiments, we also tried to change the skip first parameter that is set to False or True in the original paper [9], depending on the dataset studied. It did not change the results of the experiments.

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Pineau, E., Razakarivony, S., Bonald, T. (2020). Seq2VAR: Multivariate Time Series Representation with Relational Neural Networks and Linear Autoregressive Model. In: Lemaire, V., Malinowski, S., Bagnall, A., Bondu, A., Guyet, T., Tavenard, R. (eds) Advanced Analytics and Learning on Temporal Data. AALTD 2019. Lecture Notes in Computer Science(), vol 11986. Springer, Cham. https://doi.org/10.1007/978-3-030-39098-3_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-39098-3_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-39097-6

  • Online ISBN: 978-3-030-39098-3

  • eBook Packages: Computer ScienceComputer Science (R0)