Skip to main content

How to Trust Generative Probabilistic Models for Time-Series Data?

  • 258 Accesses

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 12931)

Abstract

Generative machine learning methods deliver unprecedented quality in the fields of computer vision and natural language processing. When comparing models for these task, the user can fast and reliably judge generated data with her bare eye—for humans, it is easy to decide whether an image or a paragraph of text is realistic. However, generative models for time series data from natural or social processes are largely unexplored, partially due to a lack of reliable and practical quality measures. In this work, measures for the evaluation of generative models for time series data are studied—in total, over 1000 models are trained and analyzed. The well-established maximum mean discrepancy (MMD) and our novel proposal: the Hausdorff discrepancy (HD) are considered for quantifying the disagreement between the sample distribution of each generated data set and the ground truth data. While MMD relies on the distance between mean-vectors in an implicit high-dimensional feature space, the proposed HD relies on intuitive and explainable geometric properties of a “typical” sample. Both discrepancies are instantiated for three underlying distance measures, namely Euclidean, dynamic time warping, and Frechét distance. The discrepancies are applied to evaluate samples from generative adversarial networks, variational autoencoders, and Markov random fields. Experiments on real-world energy prices and humidity measurements suggest, that considering a single score is insufficient for judging the quality of a generative model.

Keywords

  • Generative models
  • Time series
  • Deep learning
  • Hausdorff discrepancy
  • MMD

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-92121-7_23
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   69.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-92121-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   89.99
Price excludes VAT (USA)
Fig. 1.

Notes

  1. 1.

    https://github.com/mk406/GenerativeProbabilisticModels.git.

References

  1. Alt, H., Godau, M.: Computing the Fréchet distance between two polygonal curves. Int. J. Comput. Geom. Appl. 5(01n02), 75–91 (1995)

    CrossRef  Google Scholar 

  2. Arlitt, M., Marwah, M., Bellala, G., Shah, A., Healey, J., Vandiver, B.: IoTAbench: an Internet of Things analytics benchmark. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering, pp. 133–144 (2015)

    Google Scholar 

  3. Brophy, E., Wang, Z., Ward, T.E.: Quick and easy time series generation with established image-based GANs. arXiv preprint arXiv:1902.05624 (2019)

  4. Chen, Y., Wang, Y., Kirschen, D., Zhang, B.: Model-free renewable scenario generation using generative adversarial networks. IEEE Trans. Power Syst. 33(3), 3265–3275 (2018)

    CrossRef  Google Scholar 

  5. Chollet, F.: Deep Learning with Python. Manning, Shelter Island (2018). Safari Tech Books Online

    Google Scholar 

  6. Dubuisson, M.P., Jain, A.K.: A modified Hausdorff distance for object matching. In: International Conference on Pattern Recognition, vol. 1, pp. 566–568. IEEE (1994)

    Google Scholar 

  7. Esteban, C., Hyland, S.L., Rätsch, G.: Real-valued (medical) time series generation with recurrent conditional GANs. CoRR abs/1706.02633 (2017)

    Google Scholar 

  8. Fischer, R., Piatkowski, N., Pelletier, C., Webb, G.I., Petitjean, F., Morik, K.: No cloud on the horizon: probabilistic gap filling in satellite image series. In: International Conference on Data Science and Advanced Analytics, pp. 546–555 (2020)

    Google Scholar 

  9. Fortuin, V., Baranchuk, D., Rätsch, G., Mandt, S.: GP-VAE: deep probabilistic time series imputation. In: International Conference on Artificial Intelligence and Statistics, pp. 1651–1661. PMLR (2020)

    Google Scholar 

  10. Goodfellow, I.J., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680 (2014)

    Google Scholar 

  11. Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.J.: A kernel method for the two-sample-problem. In: Advances in Neural Information Processing Systems, vol. 19, pp. 513–520 (2006)

    Google Scholar 

  12. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30, pp. 6626–6637 (2017)

    Google Scholar 

  13. Kaiser, Ł., Bengio, S.: Discrete autoencoders for sequence models. arXiv preprint arXiv:1801.09797 (2018)

  14. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)

    Google Scholar 

  15. Keogh, E.J., Pazzani, M.J.: Derivative dynamic time warping. In: Proceedings of the 2001 SIAM International Conference on Data Mining, pp. 1–11. SIAM (2001)

    Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  17. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)

  18. Kusner, M.J., Paige, B., Hernández-Lobato, J.M.: Grammar variational autoencoder. arXiv preprint arXiv:1703.01925 (2017)

  19. Metz, L., Ibarz, J., Jaitly, N., Davidson, J.: Discrete sequential prediction of continuous actions for deep RL. arXiv preprint arXiv:1705.05035 (2017)

  20. Piatkowski, N.: Hyper-parameter-free generative modelling with deep Boltzmann trees. In: Brefeld, U., Fromont, E., Hotho, A., Knobbe, A., Maathuis, M., Robardet, C. (eds.) ECML PKDD 2019. LNCS (LNAI), vol. 11907, pp. 415–431. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46147-8_25

    CrossRef  Google Scholar 

  21. Piatkowski, N., Lee, S., Morik, K.: Spatio-temporal random fields: compressible representation and distributed estimation. Mach. Learn. 93(1), 115–139 (2013). https://doi.org/10.1007/s10994-013-5399-7

    MathSciNet  CrossRef  MATH  Google Scholar 

  22. Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and approximate inference in deep generative models. In: Proceedings of the 31th International Conference on Machine Learning, pp. 1278–1286 (2014)

    Google Scholar 

  23. Sankoff, D., Kruskal, J.: The symmetric time-warping problem: from continuous to discrete. In: Time Warps, String Edits and Macromolecules: The Theory and Practice of Sequence Comparison, pp. 125–161. Addison Wesley (1983)

    Google Scholar 

  24. Sriperumbudur, B.K., Gretton, A., Fukumizu, K., Schölkopf, B., Lanckriet, G.R.: Hilbert space embeddings and metrics on probability measures. J. Mach. Learn. Res. 11, 1517–1561 (2010)

    MathSciNet  MATH  Google Scholar 

  25. Takahashi, S., Chen, Y., Tanaka-Ishii, K.: Modeling financial time-series with generative adversarial networks. Phys. A 527, 121261 (2019)

    CrossRef  Google Scholar 

  26. Wan, Z., Zhang, Y., He, H.: Variational autoencoder based synthetic data generation for imbalanced learning. In: 2017 Symposium Series on Computational Intelligence (SSCI), pp. 1–7. IEEE (2017)

    Google Scholar 

  27. Wiese, M., Knobloch, R., Korn, R., Kretschmer, P.: Quant GANs: deep generation of financial time series. Quant. Finance 20(9), 1419–1440 (2020)

    MathSciNet  CrossRef  Google Scholar 

  28. Xu, H., et al.: Unsupervised anomaly detection via variational auto-encoder for seasonal KPIs in web applications. In: WWW Conference, pp. 187–196 (2018)

    Google Scholar 

  29. Yoon, J., Jarrett, D., van der Schaar, M.: Time-series generative adversarial networks. In: Advances in Neural Information Processing Systems, pp. 5508–5518 (2019)

    Google Scholar 

  30. Zhang, C., Kuppannagari, S.R., Kannan, R., Prasanna, V.K.: Generative adversarial network for synthetic time series data generation in smart grids. In: 2018 International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp. 1–6. IEEE (2018)

    Google Scholar 

Download references

Acknowledgments

Parts of this work have been funded by the Federal Ministry of Education and Research of Germany as part of the competence center for machine learning ML2R (01IS18038B). Parts of this work have been supported by the Deutsche Forschungsgemeinschaft via SFB 823.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nico Piatkowski .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Piatkowski, N., Posch, P.N., Krause, M. (2021). How to Trust Generative Probabilistic Models for Time-Series Data?. In: Simos, D.E., Pardalos, P.M., Kotsireas, I.S. (eds) Learning and Intelligent Optimization. LION 2021. Lecture Notes in Computer Science(), vol 12931. Springer, Cham. https://doi.org/10.1007/978-3-030-92121-7_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92121-7_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92120-0

  • Online ISBN: 978-3-030-92121-7

  • eBook Packages: Computer ScienceComputer Science (R0)