Skip to main content
View expanded cover

Joint European Conference on Machine Learning and Knowledge Discovery in Databases

ECML PKDD 2021: Machine Learning and Knowledge Discovery in Databases. Research Track pp 116–131Cite as

Principled Interpolation in Normalizing Flows

  • 690 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12976)

Abstract

Generative models based on normalizing flows are very successful in modeling complex data distributions using simpler ones. However, straightforward linear interpolations show unexpected side effects, as interpolation paths lie outside the area where samples are observed. This is caused by the standard choice of Gaussian base distributions and can be seen in the norms of the interpolated samples as they are outside the data manifold. This observation suggests that changing the way of interpolating should generally result in better interpolations, but it is not clear how to do that in an unambiguous way. In this paper, we solve this issue by enforcing a specific manifold and, hence, change the base distribution, to allow for a principled way of interpolation. Specifically, we use the Dirichlet and von Mises-Fisher base distributions on the probability simplex and the hypersphere, respectively. Our experimental results show superior performance in terms of bits per dimension, Fréchet Inception Distance (FID), and Kernel Inception Distance (KID) scores for interpolation, while maintaining the generative performance.

Keywords

  • Generative modeling
  • Density estimation
  • Normalizing flows

S. G. Fadel and S. Mair—Equal contribution.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-86520-7_8
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-86520-7
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

References

  1. Agustsson, E., Sage, A., Timofte, R., Gool, L.V.: Optimal transport maps for distribution preserving operations on latent spaces of generative models. In: International Conference on Learning Representations (2019)

    Google Scholar 

  2. Arvanitidis, G., Hansen, L.K., Hauberg, S.: Latent space oddity: on the curvature of deep generative models. In: International Conference on Learning Representations (2018)

    Google Scholar 

  3. Bińkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying MMD GANs. In: International Conference on Learning Representations (2018)

    Google Scholar 

  4. Bose, J., Smofsky, A., Liao, R., Panangaden, P., Hamilton, W.: Latent variable modelling with hyperbolic normalizing flows. In: International Conference on Machine Learning, pp. 1045–1055. PMLR (2020)

    Google Scholar 

  5. Brehmer, J., Cranmer, K.: Flows for simultaneous manifold learning and density estimation. In: Advances in Neural Information Processing Systems, vol. 33 (2020)

    Google Scholar 

  6. Chen, T.Q., Behrmann, J., Duvenaud, D.K., Jacobsen, J.H.: Residual flows for invertible generative modeling. In: Advances in Neural Information Processing Systems, pp. 9913–9923 (2019)

    Google Scholar 

  7. Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., Ha, D.: Deep learning for classical Japanese literature. CoRR abs/1812.01718 (2018)

    Google Scholar 

  8. Davidson, T.R., Falorsi, L., De Cao, N., Kipf, T., Tomczak, J.M.: Hyperspherical variational auto-encoders. In: Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, pp. 856–865 (2018)

    Google Scholar 

  9. Dinh, L., Krueger, D., Bengio, Y.: NICE: non-linear independent components estimation. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Workshop Track Proceedings (2015)

    Google Scholar 

  10. Dinh, L., Shol-Dickstein, J., Bengio, S.: Density estimation using Real NVP. In: International Conference on Learning Representations (2017)

    Google Scholar 

  11. Gemici, M.C., Rezende, D., Mohamed, S.: Normalizing flows on Riemannian manifolds. CoRR abs/1611.02304 (2016)

    Google Scholar 

  12. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  13. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, pp. 6626–6637 (2017)

    Google Scholar 

  14. Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations (2018)

    Google Scholar 

  15. Kilcher, Y., Lucchi, A., Hofmann, T.: Semantic interpolation in implicit models. In: International Conference on Learning Representations (2018)

    Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015)

    Google Scholar 

  17. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, 14–16 April 2014, Conference Track Proceedings (2014)

    Google Scholar 

  18. Kingma, D.P., Welling, M., et al.: An introduction to variational autoencoders. Found. Trends® Mach. Learn. 12(4), 307–392 (2019)

    CrossRef  Google Scholar 

  19. Kingma, D.P., Dhariwal, P.: Glow: generative flow with invertible 1 x 1 convolutions. In: Advances in Neural Information Processing Systems, pp. 10215–10224 (2018)

    Google Scholar 

  20. Kingma, D.P., Salimans, T., Jozefowicz, R., Chen, X., Sutskever, I., Welling, M.: Improved variational inference with inverse autoregressive flow. In: Advances in Neural Information Processing Systems, pp. 4743–4751 (2016)

    Google Scholar 

  21. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report (2009)

    Google Scholar 

  22. LeCun, Y., Cortes, C., Burges, C.: Mnist handwritten digit database. ATT Labs [Online]. http://yann.lecun.com/exdb/mnist, February 2010

  23. Papamakarios, G., Nalisnick, E., Rezende, D.J., Mohamed, S., Lakshminarayanan, B.: Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res. 22(57), 1–64 (2021)

    MathSciNet  MATH  Google Scholar 

  24. Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)

    Google Scholar 

  25. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: International Conference on Learning Representations, ICLR 2016 (2016)

    Google Scholar 

  26. Rezende, D., Mohamed, S.: Variational inference with normalizing flows. In: International Conference on Machine Learning, pp. 1530–1538. PMLR (2015)

    Google Scholar 

  27. Rezende, D.J., et al.: Normalizing flows on tori and spheres. In: International Conference on Machine Learning, pp. 8083–8092 (2020)

    Google Scholar 

  28. Rippel, O., Adams, R.P.: High-dimensional probability estimation with deep density models. CoRR abs/1302.5125 (2013)

    Google Scholar 

  29. Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)

    Google Scholar 

  30. Shoemake, K.: Animating rotation with quaternion curves. In: Proceedings of the 12th Annual Conference on Computer Graphics and Interactive Techniques, pp. 245–254 (1985)

    Google Scholar 

  31. Tabak, E.G., Turner, C.V.: A family of nonparametric density estimation algorithms. Commun. Pure Appl. Math. 66(2), 145–164 (2013)

    MathSciNet  CrossRef  Google Scholar 

  32. Tabak, E.G., Vanden-Eijnden, E.: Density estimation by dual ascent of the log-likelihood. Commun. Pure Appl. Math. 8(1), 217–233 (2010)

    MathSciNet  MATH  Google Scholar 

  33. White, T.: Sampling generative networks. CoRR abs/1609.04468 (2016)

    Google Scholar 

  34. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. CoRR abs/1708.07747 (2017)

    Google Scholar 

  35. Xu, J., Durrett, G.: Spherical latent spaces for stable variational autoencoders. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4503–4513 (2018)

    Google Scholar 

Download references

Acknowledgements

This research was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES), Brazil, Finance Code 001 and by FAPESP (grants 2017/24005-2 and 2018/19350-5).

Author information

Authors and Affiliations

  1. University of Campinas, Campinas, Brazil

    Samuel G. Fadel

  2. Leuphana University, Lüneburg, Germany

    Samuel G. Fadel, Sebastian Mair & Ulf Brefeld

  3. Norwegian University of Science and Technology, Ålesund, Norway

    Ricardo da S. Torres

Authors
  1. Samuel G. Fadel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Mair
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ricardo da S. Torres
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ulf Brefeld
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samuel G. Fadel .

Editor information

Editors and Affiliations

  1. ELLIS - The European Laboratory for Learning and Intelligent Systems, Alicante, Spain

    Nuria Oliver

  2. ETHZ and EPFL, Zürich, Switzerland

    Fernando Pérez-Cruz

  3. Johannes Gutenberg University of Mainz, Mainz, Germany

    Stefan Kramer

  4. École Polytechnique, Palaiseau, France

    Jesse Read

  5. Basque Center for Applied Mathematics, Bilbao, Spain

    Jose A. Lozano

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3238 KB)

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Fadel, S.G., Mair, S., da S. Torres, R., Brefeld, U. (2021). Principled Interpolation in Normalizing Flows. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12976. Springer, Cham. https://doi.org/10.1007/978-3-030-86520-7_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86520-7_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86519-1

  • Online ISBN: 978-3-030-86520-7

  • eBook Packages: Computer ScienceComputer Science (R0)