Abstract
We investigate evidence lower bound (ELBO) with generalized/deformed entropy and generalized/deformed divergence, in place of Shannon entropy and KL divergence in the standard framework. Two equivalent forms of deformed ELBO have been proposed, suitable for either Tsallis or Rényi deformation that have been unified in the recent framework of \(\lambda \)-deformation (Wong and Zhang, 2022, IEEE Trans Inform Theory). The decomposition formulae are developed for \(\lambda \)-deformed ELBO, or \(\lambda \)-ELBO in short, now for real-valued \(\lambda \) (with \(\lambda = 0\) reducing to the standard case). The meaning of the deformation factor \(\lambda \) in the \(\lambda \)-deformed ELBO and its performance for variational autoencoder (VAE) are investigated. Naturally emerging from our formulation is a deformation homotopy probability distribution function that extrapolates encoder distribution and the latent prior. Results show that \(\lambda \) values around 0.5 generally achieve better performance in image reconstruction for generative models.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Basu, A., Harris, I.R., Hjort, N.L., Jones, M.: Robust and efficient estimation by minimising a density power divergence. Biometrika 85(3), 549–559 (1998)
Burda, Y., Grosse, R., Salakhutdinov, R.: Importance weighted autoencoders (2015). https://doi.org/10.48550/ARXIV.1509.00519
Chen, L., Tao, C., Zhang, R., Henao, R., Duke, L.C.: Variational inference and model selection with generalized evidence bounds. In: International Conference on Machine Learning, pp. 893–902. PMLR (2018)
Chernoff, H.: A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics, pp. 493–507 (1952)
Cichocki, A., Amari, S.I.: Families of alpha-beta-and gamma-divergences: Flexible and robust measures of similarities. Entropy 12(6), 1532–1568 (2010)
Cressie, N., Read, T.R.: Multinomial goodness-of-fit tests. J. Roy. Stat. Soc.: Ser. B (Methodol.) 46(3), 440–464 (1984)
Fujisawa, H., Eguchi, S.: Robust parameter estimation with a small bias against heavy contamination. J. Multivar. Anal. 99(9), 2053–2081 (2008)
Gimenez, J.R., Zou, J.: A unified f-divergence framework generalizing vae and gan (2022). https://doi.org/10.48550/ARXIV.2205.05214
Kingma, D.P., Welling, M.: Auto-encoding variational bayes (2013). https://doi.org/10.48550/ARXIV.1312.6114
Kobayashis, T.: q-VAE for disentangled representation learning and latent dynamical systems. IEEE Robotics Autom. Lett. 5(4), 5669–5676 (2020). https://doi.org/10.1109/lra.2020.3010206
Lafferty, J.: Additive models, boosting, and inference for generalized divergences. In: Proceedings of the Twelfth Annual Conference on Computational Learning Theory, pp. 125–133 (1999)
Li, Y., Turner, R.E.: Rényi divergence variational inference (2016). https://doi.org/10.48550/ARXIV.1602.02311
Mihoko, M., Eguchi, S.: Robust blind source separation by beta divergence. Neural Comput. 14(8), 1859–1886 (2002)
Mollah, M.N.H., Sultana, N., Minami, M., Eguchi, S.: Robust extraction of local structures by the minimum \(\beta \)-divergence method. Neural Netw. 23(2), 226–238 (2010)
Nielsen, F., Nock, R.: On rényi and tsallis entropies and divergences for exponential families. arXiv: 1105.3259 (2011)
Prokhorov, V., Shareghi, E., Li, Y., Pilehvar, M.T., Collier, N.: On the importance of the Kullback-Leibler divergence term in variational autoencoders for text generation. In: Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 118–127. Association for Computational Linguistics, Hong Kong (Nov 2019). https://doi.org/10.18653/v1/D19-5612
Regli, J.B., Silva, R.: Alpha-beta divergence for variational inference (2018)
Sajid, N., Faccio, F., Da Costa, L., Parr, T., Schmidhuber, J., Friston, K.: Bayesian brains and the rényi divergence (2021). https://doi.org/10.48550/ARXIV.2107.05438
Sârbu, S., Malagò, L.: Variational autoencoders trained with q-deformed lower bounds (2019)
Sârbu, S., Volpi, R., Peşte, A., Malagò, L.: Learning in variational autoencoders with kullback-leibler and renyi integral bounds (2018). https://doi.org/10.48550/ARXIV.1807.01889
Taneja, I.J.: New developments in generalized information measures. In: Advances in Imaging and Electron Physics, vol. 91, pp. 37–135. Elsevier (1995)
Tsallis, C.: What are the numbers that experiments provide. Quim. Nova 17, 468–471 (1994)
Wang, Z., et al.: Variational inference mpc using tsallis divergence (2021). https://doi.org/10.48550/ARXIV.2104.00241
Wong, T.K.L., Zhang, J.: Tsallis and rényi deformations linked via a new \(\lambda \)-duality. IEEE Trans. Inf. Theory 68(8), 5353–5373 (2022). https://doi.org/10.1109/TIT.2022.3159385
Zhang, J., Wong, T.K.L.: \(\lambda \)-Deformed probability families with subtractive and divisive normalizations, vol. 45, pp. 187–215 (Jan 2021). https://doi.org/10.1016/bs.host.2021.06.003
Zhang, J., Wong, T.K.L.: λ-deformation: A canonical framework for statistical manifolds of constant curvature. Entropy 24(2) (2022). https://doi.org/10.3390/e24020193
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cheng, K., Zhang, J. (2023). \(\lambda \)-Deformed Evidence Lower Bound (\(\lambda \)-ELBO) Using Rényi and Tsallis Divergence. In: Nielsen, F., Barbaresco, F. (eds) Geometric Science of Information. GSI 2023. Lecture Notes in Computer Science, vol 14071. Springer, Cham. https://doi.org/10.1007/978-3-031-38271-0_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-38271-0_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-38270-3
Online ISBN: 978-3-031-38271-0
eBook Packages: Computer ScienceComputer Science (R0)