Generative Models for Missing Data

Abstract

Missing data poses an ubiquitous challenge across a wide range of applications, stemming from a multitude of causes that are both diverse and context-dependent. The prevailing issue is that most advanced data analysis techniques are primarily tailored for complete datasets, thereby underscoring the indispensable need for effective imputation methods. In this chapter, we embark on an extensive exploration of missing data from a statistical perspective, offering a holistic review of its intricate nature. Our investigation encompasses a deep dive into the various mechanisms underlying missing data, shedding light on their ignorability and identifiability-fundamental concepts essential for understanding and addressing this pervasive issue. Moreover, we present a succinct yet comprehensive overview of influential classical imputation methods, showcasing their contributions to the field. Building upon this foundation, we delve into the latest advancements in generative models, a burgeoning area that holds great promise for learning from and imputing missing data. By harnessing the power of generative models, we aim to unlock novel insights and methodologies that can tackle the challenges posed by missing data. Furthermore, we introduce an approach that specifically addresses the critical problem of nonparametric identifiability in nonignorable missing data through the innovative use of generative models. This novel approach aims to overcome the limitations associated with alternative generative models and provides a potential solution to this challenging issue. To enhance the clarity of our proposed method, we supplement our discourse with curated numerical examples that distinguish its effectiveness from other baselines in specific scenarios. Through the exploration, we hope to pave the way for further research and advancements in this critical domain, ultimately leading to more accurate and reliable analyses and interpretations of incomplete datasets.

7 Appendix

Summary of Notations See Table 2.

Table 2 Summary of Basic Notations for Missing Data

Proof of Theorem 1

Proof

Under Condition (9),

$$\begin{aligned} p(r|x) & = \int p(r|x, \tilde{z}) p(\tilde{z}) d\tilde{z}\\ & = \int \prod ^p_{j=1} p(r_j|x_{-j}, \tilde{z}) p(\tilde{z}) d\tilde{z}\\ & = \int \prod ^p_{j=1} p(r_j|x_{-j}, R_{-j} = \boldsymbol{1}, \tilde{z}) p(\tilde{z}) d\tilde{z}. \end{aligned}$$

Thus, p(r|x) is a function of the observed data only.

Proof of Corollary 1

Proof

Using the odds ratio parametrization [71, 72] with odds ratio function

$$\begin{aligned} \text {OR}(r,x) \equiv \text {OR}(r,x;r_0 = \boldsymbol{1}, x_0 = \boldsymbol{0}) = \frac{p (r|x)}{p (R=\boldsymbol{1}|x)}\frac{p (R=\boldsymbol{1}|X=\boldsymbol{0})}{p (r|X=\boldsymbol{0})}, \end{aligned}$$

the full-data distribution [1] can be written as

$$\begin{aligned} p (x,r) = \frac{OR(r,x)p (x|R=\boldsymbol{1})p (r|X = \boldsymbol{0})}{\sum _{r'}\mathbb {E}[OR(r',y)|R=\boldsymbol{1}]p (r'|X=\boldsymbol{0})}, \end{aligned}$$

which is also a function of observed data only.

Implementation Details

In the experiments in Sect. 5, for all three generative models, we set the dimension of the latent space to be $p-1$ for both datasets. K is set to be 20 for the importance samples. All three models have the same nonlinear structure for the decoder of missingness indicators. We use one hidden layer for both encoder and decoders with dimension 128. Instead of taking a fixed value, the observational noise for the continuous data variables in the decoder is specified as a learnable parameter as proposed in [69]. All methods are trained with Adam optimizer with batch size 16, and learning rate 0.001 for 30000 epochs. During the imputation, the number of importance samples L is set as 10000. For the two classical imputation methods, the IterativeImputer from the sklearn package in Python is exploited in the experiments, with the default estimator for MICE with maximum iteration as 100 and RandomForestRegressor for missForest with the number of estimators as 100.