Abstract
We introduce a new method for training generative adversarial networks by applying the Wasserstein-2 metric proximal on the generators. The approach is based on Wasserstein information geometry. It defines a parametrization invariant natural gradient by pulling back optimal transport structures from probability space to parameter space. We obtain easy-to-implement iterative regularizers for the parameter updates of implicit deep generative models. Our experiments demonstrate that this method improves the speed and stability of training in terms of wall-clock time and Fréchet Inception Distance.
A. Lin, W. Li and S. Osher were supported by AFOSR MURI FA 9550-18-1-0502, AFOSR FA 9550-18-0167, ONR N00014-18-2527 and NSF DMS 1554564 (STROBE). G. Montúfar has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no 757983).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amari, S.: Natural gradient works efficiently in learning. Neural Comput. 10(2), 251–276 (1998)
Amari, S.: Information Geometry and Its Applications. AMS, vol. 194. Springer, Tokyo (2016). https://doi.org/10.1007/978-4-431-55978-8
Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv:1701.07875 [cs, stat] (2017)
Ay, N., Jost, J., Lê, H.V., Schwachhöfer, L.: Information Geometry. EMGFASMSM, vol. 64. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56478-4
Frogner, C., Zhang, C., Mobahi, H., Araya-Polo, M., Poggio, T.: Learning with a Wasserstein Loss. arXiv:1506.05439 [cs, stat] (2015)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27, pp. 2672–2680. Curran Associates Inc. (2014)
Jordan, R., Kinderlehrer, D., Otto, F.: The variational formulation of the Fokker-Planck equation. SIAM J. Math. Anal. 29(1), 1–17 (1998)
Li, W.: Transport information geometry: Riemannian calculus on probability simplex. arXiv:1803.06360 [math] (2018)
Li, W., Montúfar, G.: Natural gradient via optimal transport. Inf. Geom. 1(2), 181–214 (2018)
Li, W., Montúfar, G.: Ricci curvature for parametric statistics via optimal transport. arXiv:1807.07095 [cs, math, stat] (2018)
Montavon, G., Müller, K.-R., Cuturi, M.: Wasserstein training of restricted Boltzmann machines. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 3718–3726. Curran Associates Inc. (2016)
Ollivier, Y.: Riemannian metrics for neural networks I: feedforward networks. Inf. Infer. J. IMA 4(2), 108–153 (2015)
Otto, F.: The geometry of dissipative evolution equations the porous medium equation. Commun. Partial Differ. Equ. 26(1–2), 101–174 (2001)
Tong Lin, A., Li, W., Osher, S., Montúfar, G.: Wasserstein proximal of GANs. arXiv:2102.06862 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Lin, A.T., Li, W., Osher, S., Montúfar, G. (2021). Wasserstein Proximal of GANs. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2021. Lecture Notes in Computer Science(), vol 12829. Springer, Cham. https://doi.org/10.1007/978-3-030-80209-7_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-80209-7_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80208-0
Online ISBN: 978-3-030-80209-7
eBook Packages: Computer ScienceComputer Science (R0)