Principal Component Analysis in the Presence of Missing Data

Abstract

The aim of this chapter is to provide an overview of recent developments in principal component analysis (PCA) methods when the data are incomplete. Missing data bring uncertainty into the analysis and their treatment requires statistical approaches that are tailored to cope with specific missing data processes (i.e., ignorable and nonignorable mechanisms). Since the publication of the classic textbook by Jolliffe, which includes a short, same-titled section on the missing data problem in PCA, there have been a few methodological contributions that hinge upon a probabilistic approach to PCA. In this chapter, we unify methods for ignorable and nonignorable missing data in a general likelihood framework. We also provide real data examples to illustrate the application of these methods using the R language and environment for statistical computing and graphics.

Appendix – EM Algorithm for PPCA with MNAR Values

In this appendix, we provide additional details on the Monte Carlo EM algorithm introduced in Sect. 3.2 and we derive a simplified E-step where the random effects are integrated out from the complete data log-likelihood.

The Monte Carlo E-step requires sampling from $f\left( \mathbf {z}_{i},\mathbf {u}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right)$. This task can be carried out efficiently via ARMS [12] using the full conditionals

$$\begin{aligned} f\left( \mathbf {z}_{i}|\mathbf {x}_{i},\mathbf {u}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right)&\propto f\left( \mathbf {y}_{i}|\mathbf {u}_{i},\varvec{\lambda }^{(t)}\right) f\left( \mathbf {m}_{i}|\mathbf {y}_{i},\varvec{\lambda }^{(t)}\right) ,\end{aligned}$$

(13)

$$\begin{aligned} f\left( \mathbf {u}_{i}|\mathbf {x}_{i},\mathbf {z}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right)&\propto f\left( \mathbf {y}_{i}|\mathbf {u}_{i},\varvec{\lambda }^{(t)}\right) f\left( \mathbf {u}_{i}\right) . \end{aligned}$$

(14)

An implementation of ARMS is available in the R package HI [35].

A sample $\varvec{\xi }_{i1},\ldots ,\varvec{\xi }_{iK}$ for $i=1,\ldots ,n$ is obtained at each EM iteration t, where the $(s_{i} + q) \times 1$ vector $\varvec{\xi }_{ik} = \left( \tilde{\mathbf {z}}_{ik},\tilde{\mathbf {u}}_{ik}\right)$, $k = 1, \ldots , K$, contains ‘imputed’ values for $\mathbf {z}_{i}$ and $\mathbf {u}_{i}$ (with the understanding that $\varvec{\xi }_{ik} = \tilde{\mathbf {u}}_{ik}$ if $s_{i}=0$). Here the Monte Carlo sample size K is kept constant throughout. Alternative strategies with varying $K^{(t)}$ that may increase the speed or the accuracy of the EM algorithm can be considered [2, 17]. The E-step (11) is approximated by

$$\begin{aligned} Q(\varvec{\lambda }|\varvec{\lambda }^{(t)}) = \frac{1}{K} \sum _{i = 1}^{n}\sum _{k=1}^{K}l\left( \varvec{\lambda }; \varvec{\xi }_{ik}, \mathbf {x}_{i},\mathbf {m}_{i}\right) . \end{aligned}$$

(15)

The maximization of (15) with respect to $\varvec{\lambda }$ is straightforward. Define $\tilde{\mathbf {y}}_{ik} = \left( \tilde{\mathbf {z}}_{ik},\mathbf {x}_{i}\right)$ if $s_{i}>0$ or $\tilde{\mathbf {y}}_{ik} = \mathbf {y}_{i}$ if $s_{i}=0$, $i = 1,\ldots ,n$, $k = 1,\ldots ,K$. The maximum likelihood solution of the M-step at the $(t+1)$th iteration is given by

$$\begin{aligned} \hat{\varvec{\mu }}^{(t+1)}&= \frac{1}{nK}\sum _{i = 1}^{n} \sum _{k=1}^{K}\left( \tilde{\mathbf {y}}_{ik} - \hat{\mathbf {W}}^{(t)}\tilde{\mathbf {u}}_{ik}\right) ,\end{aligned}$$

(16)

$$\begin{aligned} \hat{\mathbf {W}}^{(t+1)}&= \left\{ \sum _{i = 1}^{n}\sum _{k=1}^{K}\left( \tilde{\mathbf {y}}_{ik} - \hat{\varvec{\mu }}^{(t+1)}\right) \tilde{\mathbf {u}}^{\top }_{ik}\right\} \left( \sum _{i = 1}^{n}\sum _{k=1}^{K}\tilde{\mathbf {u}}_{ik}\tilde{\mathbf {u}}^{\top }_{ik}\right) ^{-1},\end{aligned}$$

(17)

$$\begin{aligned} \hat{\psi }^{(t+1)}&= \frac{1}{nKp} \sum _{i = 1}^{n}\sum _{k=1}^{K}\Vert \tilde{\mathbf {y}}_{ik} - \hat{\varvec{\mu }}^{(t+1)} - \hat{\mathbf {W}}^{(t+1)}\tilde{\mathbf {u}}_{ik}\Vert _{2}^2. \end{aligned}$$

(18)

Analogously, the MLE of $\varvec{\eta }$ can be easily obtained using standard results for generalized linear models.

Note that the computational burden can be alleviated by first integrating out the random effects in (11) and then sampling from $f\left( \mathbf {z}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right)$ during the Monte Carlo E-step. We obtain what we call a simplified E-step

$$\begin{aligned} \nonumber \nonumber Q_{i}(\varvec{\lambda }|\varvec{\lambda }^{(t)})&= \int \!\!\! \int \left\{ \log f\left( \mathbf {y}_{i},\mathbf {u}_{i}|\varvec{\theta }\right) + \log f\left( \mathbf {m}_{i}|\mathbf {y}_{i},\varvec{\eta }\right) \right\} f\left( \mathbf {z}_{i},\mathbf {u}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right) d\mathbf {u}_{i}d\mathbf {z}_{i}\\ \nonumber&= \int \!\!\! \int \left\{ \log f\left( \mathbf {y}_{i}|\mathbf {u}_{i},\varvec{\theta }\right) + \log f\left( \mathbf {u}_{i}\right) \right\} f\left( \mathbf {z}_{i},\mathbf {u}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right) d\mathbf {u}_{i}d\mathbf {z}_{i}\\ \nonumber&\qquad {} + \int \log f\left( \mathbf {m}_{i}|\mathbf {y}_{i},\varvec{\eta }\right) f\left( \mathbf {z}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right) d\mathbf {z}_{i}\\ \nonumber&= \int \left\{ -\frac{p}{2} \log (\psi ) - \frac{1}{2\psi } {\text {tr}}\left( \mathbf {W}^{\top }\mathbf {W}\mathbf {B}^{(t)}\right) - \frac{1}{2\psi }\Vert \mathbf {y}_{i} - \varvec{\mu } - \mathbf {W}\mathbf {v}_{i}^{(t)}\Vert _{2}^2 \right. \\ \nonumber&\qquad {} + \log f\left( \mathbf {m}_{i}|\mathbf {y}_{i},\varvec{\eta }\right) \bigg \} f\left( \mathbf {z}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right) d\mathbf {z}_{i}\\&\equiv {\text {E}}_{\mathbf {z}|\mathbf {x},\mathbf {m},\varvec{\lambda }^{(t)}}\left\{ l\left( \varvec{\lambda }; \mathbf {y}_{i},\mathbf {m}_{i}\right) \right\} , \end{aligned}$$

(19)

where $\mathbf {v}_{i}^{(t)} = \mathbf {B}^{(t)}\mathbf {W}^{(t)^{\top }}\left( \mathbf {y}_{i} - \varvec{\mu }^{(t)}\right) /\psi ^{(t)}$ and $\mathbf {B}^{(t)} = \left\{ \mathbf {W}^{(t)^{\top }} \mathbf {W}^{(t)}/\psi ^{(t)} + \mathbf {I}_{q}\right\} ^{-1}$. Note that by assumption $\mathbf {m}_{i}$ is independent from $\mathbf {u}_i$. The expectation above is now taken with respect to

$$\begin{aligned} f\left( \mathbf {z}_{i}|\mathbf {x}_{i},\mathbf {m}_{i},\varvec{\lambda }^{(t)}\right) \propto \exp \left\{ -\frac{1}{2}\left( \mathbf {y}_{i}-\varvec{\mu }^{(t)}\right) ^{\top }\mathbf {C}^{(t)^{-1}}\left( \mathbf {y}_{i}-\varvec{\mu }^{(t)}\right) \right\} f\left( \mathbf {m}_{i}|\mathbf {y}_{i},\varvec{\eta }^{(t)}\right) , \end{aligned}$$

$\mathbf {C}^{(t)} = \mathbf {W}^{(t)}\mathbf {W}^{(t)^{\top }} + \varvec{\varPsi }^{(t)}$.

Again, we obtain a sample $\tilde{\mathbf {z}}_{ik}$, $i = 1,\ldots ,n$, $k = 1,\ldots ,K$ and calculate the approximate E-step

$$\begin{aligned} Q(\varvec{\lambda }|\varvec{\lambda }^{(t)}) = \frac{1}{K} \sum _{i = 1}^{n}\sum _{k=1}^{K}l\left( \varvec{\lambda }; \tilde{\mathbf {z}}_{ik}, \mathbf {x}_{i},\mathbf {m}_{i}\right) . \end{aligned}$$

(20)

The MLE equations of the M-step which follow from maximizing the log-likelihood in (20) are similar to equations (27) and (28) in [42] and they do not require explicit computation of the covariance matrix. We omit them for the sake of brevity.

Finally, we note that, based on the linear predictions

$$\begin{aligned} \hat{\mathbf {u}}_{ik} = \left( \hat{\mathbf {W}}^{\top }\hat{\mathbf {W}} + \hat{\varvec{\varPsi }}\right) ^{-1}\hat{\mathbf {W}}^{\top }(\tilde{\mathbf {y}}_{ik} - \hat{\varvec{\mu }}), \end{aligned}$$

(21)

where $\tilde{\mathbf {y}}_{ik} = \left( \tilde{\mathbf {z}}_{ik},\mathbf {x}_{i}\right)$ is the complete data vector at convergence, we can calculate the element-wise variances of $\frac{1}{K}\sum _{k=1}^{K} \hat{\mathbf {u}}_{ik}$ over the individuals space as estimates of $\delta _{1},\ldots ,\delta _{q}$. The quantity $(p-q)\cdot \hat{\varvec{\psi }}$ provides the portion of the total variability associated with the ‘discarded’ components.