Variational Autoencoders

Abstract

Chapter 12 explained that learning models can be divided into discriminative and generative models. The Variational Autoencoder (VAE), introduced in this chapter, is a generative model. Variational inference is a technique that finds a lower bound on the log-likelihood of the data and maximizes the lower bound rather than the log-likelihood in the Maximum Likelihood Estimation (MLE) (see Chap. 12). This lower bound is usually referred to as the Evidence Lower Bound (ELBO). Learning the parameters of latent space can be done using Expectation Maximization (EM), as done in factor analysis (see Chap. 12). The Variational Autoencoder (VAE) implements variational inference in an autoencoder neural network setup, where the encoder and decoder model the E-step (expectation step) and M-step (maximization step) of EM, respectively. However, VAE is usually trained using backpropagation in practice. Variational inference and VAE are found in many Bayesian analysis applications. For example, variational inference has been used in 3D human motion analysis and VAE has been used in forecasting.

Appendix 20.1: Proof for Lemma 20.1

$$\displaystyle \begin{aligned} \text{KL}(p_1\|p_2) &= \int p_1(x) \log\big(\frac{p_1(x)}{p_2(x)}\big) dx \\ &=\int p_1(x) \log(p_1(x))\, dx - \int p_1(x) \log(p_2(x))\, dx. \end{aligned} $$

According to integration by parts:

$$\displaystyle \begin{aligned} \int p_1(x) \log(p_1(x))\, dx = -\frac{1}{2} (1 + \log (2 \pi \sigma_1^2)). \end{aligned} $$

There is also:

$$\displaystyle \begin{aligned} &-\int p_1(x) \log(p_2(x))\, dx = -\int p_1(x) \log\Big( \frac{1}{\sqrt{2 \pi \sigma_2^2}} e^{-\frac{(x - \mu_2)^2}{2 \sigma_2^2}} \Big)\, dx \\ &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) \underbrace{\int p_1(x) dx}_{=1} -\int p_1(x) \log\big( e^{-\frac{(x - \mu_2)^2}{2 \sigma_2^2}} \big)\, dx \\ &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) +\int p_1(x) \frac{(x - \mu_2)^2}{2 \sigma_2^2}\, dx \\ &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) + \frac{1}{2 \sigma_2^2} \Big(\int p_1(x) x^2 dx - \int p_1(x) 2x\mu_2 dx + \int p_1(x) \mu_2^2 dx\Big) \\ &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) + \frac{1}{2 \sigma_2^2} \Big(\mathbb{E}_{\sim p_1(x)} [x^2] - 2 \mu_2 \mathbb{E}_{\sim p_1(x)} [x] + \mu_2^2 \Big). \end{aligned} $$

It is known that:

$$\displaystyle \begin{aligned} \mathbb{V}\text{ar}[x] = \mathbb{E}[x^2] - \mathbb{E}[x]^2 \implies \mathbb{E}[x^2] = \sigma_1^2 + \mu_1^2 \end{aligned} $$

Therefore:

$$\displaystyle \begin{aligned} -\int p_1(x) \log(p_2(x))\, dx &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) + \frac{1}{2 \sigma_2^2} \Big(\sigma_1^2 + \mu_1^2 - 2 \mu_2 \mu_1 + \mu_2^2 \Big) \\ &= \frac{1}{2} \log( 2 \pi \sigma_2^2 ) + \frac{1}{2 \sigma_2^2} \big( \sigma_1^2 + (\mu_1 - \mu_2) \big). \end{aligned} $$

Lastly:

$$\displaystyle \begin{aligned} \text{KL}(p_1\|p_2) &= -\frac{1}{2} (1 + \log (2 \pi \sigma_1^2)) + \frac{1}{2} \log( 2 \pi \sigma_2^2 ) + \frac{1}{2 \sigma_2^2} \big( \sigma_1^2 + (\mu_1 - \mu_2) \big) \\ &= \log(\frac{\sigma_2}{\sigma_1}) + \frac{\sigma_1^2 + (\mu_1 - \mu_2)^2}{2 \sigma_2^2} - \frac{1}{2}. \end{aligned} $$

□