Imputation and low-rank estimation with Missing Not At Random data

Abstract

Missing values challenge data analysis because many supervised and unsupervised learning methods cannot be applied directly to incomplete data. Matrix completion based on low-rank assumptions are very powerful solution for dealing with missing values. However, existing methods do not consider the case of informative missing values which are widely encountered in practice. This paper proposes matrix completion methods to recover Missing Not At Random (MNAR) data. Our first contribution is to suggest a model-based estimation strategy by modelling the missing mechanism distribution. An EM algorithm is then implemented, involving a Fast Iterative Soft-Thresholding Algorithm (FISTA). Our second contribution is to suggest a computationally efficient surrogate estimation by implicitly taking into account the joint distribution of the data and the missing mechanism: the data matrix is concatenated with the mask coding for the missing values; a low-rank structure for exponential family is assumed on this new matrix, in order to encode links between variables and missing mechanisms. The methodology that has the great advantage of handling different missing value mechanisms is robust to model specification errors. The performances of our methods are assessed on the real data collected from a trauma registry (TraumaBase$^{\textregistered }$) containing clinical information about over twenty thousand severely traumatized patients in France. The aim is then to predict if the doctors should administrate tranexomic acid to patients with traumatic brain injury, that would limit excessive bleeding.

Appendices

The FISTA algorithm

We first present the proximal gradient method. The following optimisation problem is considered:

$$\begin{aligned} \hat{\varTheta } \in \text {argmin}_\varTheta \, h_1(\varTheta ) + h_2(\varTheta ), \end{aligned}$$

where $h_1$ is a convex function, $h_2$ a differentiable and convex function and L the gradient Lipschitz of $h_2$.

The main trick of the FISTA algorithm is to add a momentum term to the proximal gradient method, in order to yield smoother trajectory towards the convergence point. In addition, the proximal operator is performed on a specific linear combination of the previous two iterates, rather than on the previous iterate only.

In our specific model, to solve (2), $h_1(\varTheta )=\Vert \varTheta \Vert _\star$ and $h_2(\varTheta )=\Vert \varOmega \odot (\varTheta -Y) \Vert ^2_F$. Let us precise that:

$$\begin{aligned} \frac{\partial h_2(\varTheta )}{\partial \varTheta _{ij}}=\nabla _{\varTheta _{ij}} h_2(\varTheta )=\varOmega _{ij}\left( \varTheta _{ij} - Y_{ij} \right) . \end{aligned}$$

Therefore,

$$\begin{aligned} \nabla h_2(\varTheta )=\varOmega \odot (\varTheta -Y) \end{aligned}$$

and L is equal to 1.

SoftImpute

We start by describing softImpute.

The proximal operator of the nuclear norm of a matrix X consists in a soft-thresholding of its singular values: we perform the SVD of X and we obtain the matrices U, V and D. Then

$$\begin{aligned} \text {prox}_{\lambda \Vert \cdot \Vert _{\star }}(X)=UD_{\lambda } V. \end{aligned}$$

$D_{\lambda }$ is the diagonal matrix such that for all i,

$$\begin{aligned} D_{\lambda ,ii}=\max ((\sigma _i-\lambda ),0) \end{aligned}$$

, where the $(\sigma _{ii})$’s are the singular values of X.

1.1 Equivalence between softImpute and the proximal gradient method

By using the same functions $h_1$ and $h_2$ as above, one has:

$$\begin{aligned} \hat{\varTheta }^{(t+1)}&=\text {prox}_{\lambda (1/L)h_1}(\hat{\varTheta }^{(t)}-(1/L) \nabla h_2(\hat{\varTheta }^{(t)})) \\&=\text {prox}_{\lambda \Vert \cdot \Vert _\star }(\hat{\varTheta }^{(t)}-\varOmega \odot (\hat{\varTheta }^{(t)} -Y)) \\&=\text {prox}_{\lambda \Vert .\Vert _\star }(\varOmega \odot Y + (1-\varOmega ) \odot \hat{\varTheta }^{(t)}), \end{aligned}$$

so that softImpute and the proximal gradient method are similar.

1.2 Equivalence between the EM algorithm and iterative SVD in the MAR case

We prove here that in the MAR setting, softImpute is similar to the EM algorithm. Let us recall that in the MAR setting the model of the joint distribution is not needed but only the one of the data distribution, so that the E-step is written as follows:

$$\begin{aligned} Q(\varTheta |\hat{\varTheta }^{(t)})&=\mathbb {E}_{Y_\text {mis}}\left[ \log (p(\varTheta ;y))|Y_\text {obs};\varTheta =\hat{\varTheta }^{(t)}\right] \\&\propto -\sum _{i=1}^{n} \sum _{j=1}^{p} \mathbb {E} \left[ \left( \frac{y_{ij}-\varTheta _{ij}}{\sigma }\right) ^2 |\hat{\varTheta _{ij}}^{(t)}\right] , \end{aligned}$$

by using (3) and the independance of $Y_{ij}, \, \forall i,j$). Then, by splitting into the observed and the missing elements,

$$\begin{aligned}&Q(\varTheta |\hat{\varTheta }^{(t)})\propto -\sum _{i=1}^{n} \sum _{j, \, \varOmega _{ij}=0} \mathbb {E}\left[ \left( \frac{y_{ij} -\varTheta _{ij}}{\sigma }\right) ^2 |\hat{\varTheta _{ij}}^{(t)} \right] \\&\quad -\sum _{i=1}^{n}\sum _{j, \, \varOmega _{ij}=1} \left( \frac{y_{ij}-\varTheta _{ij}}{\sigma }\right) ^2 \end{aligned}$$

Therefore,

$$\begin{aligned}&Q(\varTheta |\hat{\varTheta }^{(t)})\propto -\sum _{i=1}^{n} \sum _{j, \, \varOmega _{ij}=0}\mathbb {E}[y_{ij} ^2 -2\varTheta _{ij}y_{ij} +\varTheta _{ij}^2| \hat{\varTheta _{ij}}^{(t)} ]^2 \\&\quad -\sum _{i=1}^{n}\sum _{j, \, \varOmega _{ij}=1} \left( \frac{y_{ij}-\varTheta _{ij}}{\sigma }\right) ^2\\&\quad Q(\varTheta |\hat{\varTheta }^{(t)})\propto -\sum _{i=1}^{n}\\&\quad \sum _{j, \, \varOmega _{ij}=0} \left( \sigma ^2 + \hat{\varTheta _{ij}}^{(t)2}-2\hat{\varTheta _{ij}}^{(t)}\varTheta _{ij}+\varTheta _{ij}^2\right) \\&\quad -\sum _{i=1}^{n}\sum _{j, \, \varOmega _{ij}=1} \left( \frac{y_{ij}-\varTheta _{ij}}{\sigma }\right) ^2 \end{aligned}$$

which implies $Q(\varTheta |\hat{\varTheta }^{(t)})\propto \Vert \varOmega \odot Y + (1-\varOmega ) \odot \hat{\varTheta }^{(t)}-\varTheta \Vert ^2_F$

The M-step is then written as follows:

$$\begin{aligned} \hat{\varTheta }^{(t+1)} \in \text {argmin}_{\varTheta } \Vert \varOmega \odot Y + (1-\varOmega ) \odot \hat{\varTheta }^{(t)}-\varTheta \Vert ^2_F + \lambda \Vert \varTheta \Vert _\star \end{aligned}$$

The proximal gradient method is applied with

$$\begin{aligned} h_1(\varTheta )=\lambda \Vert \varTheta \Vert _\star \text { and } h_2(\varTheta )=\Vert \varOmega \odot Y + (1-\varOmega ) \odot \hat{\varTheta }^{(t)}-\varTheta \Vert ^2_F. \end{aligned}$$

Therefore, the EM algorithm in the MAR case is the same one as softImpute.

The EM algorithm in the MNAR case

For the sake of clarity, we present below the EM algorithm in the MNAR and low dimension case.

We already have given details for the stopping criterium.

We clarify the maximization step given by (8) and (9).

$$\begin{aligned}&\hat{\varTheta } \in \underset{\varTheta }{\text {argmin}} \sum _{i,j} \left( \frac{1}{N_s}\sum _{k=1}^{N_s} \frac{1}{2\sigma ^2}(v_{ij}^{(k)}-\varTheta _{ij})^2 \right) + \lambda \Vert \varTheta \Vert _\star \nonumber \\&\quad \in \underset{\varTheta }{\text {argmin}}\sum _{i,j}\left( \frac{1}{N_s\sigma ^2}\sum _{k=1}^{N_s}-v_{ij}^{(k)}\varTheta _{ij}+\frac{1}{2}\varTheta _{ij}^2 \right) +\lambda \Vert \varTheta \Vert _\star \nonumber \\&\quad \in \underset{\varTheta }{\text {argmin}} \Vert V-\varTheta \Vert _F^2 + \lambda \Vert \varTheta \Vert _\star , \text { where:} \nonumber \\&V=\begin{pmatrix} \frac{1}{N_s}\sum _{k=1}^{Ns}v_{11}^{(k)} \ldots \frac{1}{N_s}\sum _{k=1}^{Ns}v_{1p}^{(k)} \\ \vdots \ddots \vdots \\ \frac{1}{N_s}\sum _{k=1}^{Ns}v_{n1}^{(k)} \ldots \frac{1}{N_s}\sum _{k=1}^{Ns}v_{np}^{(k)} \end{pmatrix} \nonumber \\&\hat{\phi }^{(t+1)} \in \underset{\phi }{\text {argmin}} \sum _{i,j}\frac{1}{N_s} \sum _{k=1}^{N_s} (1-\varOmega _{ij})C_3 - \varOmega _{ij}C_4 \nonumber \\&\quad \in \underset{\phi }{\text {argmin}}\sum _{i,j} \frac{1}{N_s} \sum _{k=1}^{N_s} C_3 + \varOmega _{ij} \phi _{1j} (v_{ij}^{k}-\phi _{2j}) \end{aligned}$$

(17)

with:

$$\begin{aligned} C_3&=\log (1+e^{-\phi _{1j}(v_{ij}^{k}-\phi _{2j})}) \\ C_4&=\log (1-(1+e^{-\phi _{1j}(v_{ij}^{k}-\phi _{2j})})^{-1}) \end{aligned}$$

For all $j \in \{1,\ldots ,p\}$ such that $\exists i \in \{1,\ldots ,n\}, \, \varOmega _{ij}=0$, estimating the coefficients $\phi _{1j}$ and $\phi _{2j}$ remains to fit a generalized linear model with the binomial link function for the matrix $J^{(j)}$:

$$\begin{aligned} J^{(j)}=\begin{pmatrix} \varOmega _{1j} &{} v_{1j}^{(1)} \\ \vdots &{} \vdots \\ \varOmega _{nj} &{} v_{nj}^{(1)} \\ \vdots &{} \vdots \\ \varOmega _{1j} &{} v_{1j}^{(N_s)} \\ \vdots &{} \vdots \\ \varOmega _{nj} &{} v_{nj}^{(N_s)} \\ \end{pmatrix} \end{aligned}$$

(18)

1.1 SIR

In the Monte Carlo approximation, the distribution of interest is $\left[ y_{ij}|\varOmega _{ij};\hat{\phi }_j^{(t)},\hat{\varTheta }_{ij}^{(t)}\right]$. By using the Bayes rules:

$$\begin{aligned}&p \left( y_{ij}|\varOmega _{ij};\hat{\phi }_j^{(t)},\hat{\varTheta }_{ij}^{(t)} \right) =: f(y_{ij}) \\&\quad \propto p\left( y_{ij};\hat{\varTheta }_{ij}^{(t)}\right) p\left( \varOmega _{ij}|y_{ij};\hat{\phi }_j^{(t)}\right) =: g(y_{ij}) \end{aligned}$$

Denoting the Gaussian density function of mean $\varTheta ^{(t)}_{ij}$ and variance $\sigma ^2$ by $\varphi _{\varTheta ^{(t)}_{ij},\sigma ^2}$, if $\sigma >(2\pi )^{-1/2}$, the following condition holds:

$$\begin{aligned}f(y_{ij})=cg(y_{ij})\le \varphi _{\varTheta ^{(t)}_{ij},\sigma ^2}(x).\end{aligned}$$

For M large, the SIR algorithm to simulate

$$\begin{aligned} z \sim \left[ y_{ij}|\varOmega _{ij};\hat{\phi }_j^{(t)},\hat{\varTheta }_{ij}^{(t)}\right] \end{aligned}$$

is described as follows.

Details on the variables in TraumaBase$^{\textregistered }$

A description of the variables which are used in Sect. 5 is given. The indications given in parentheses ph (pre-hospital) and h (hospital) mean that the measures have been taken before the arrival at the hospital and at the hospital.

• SBP.ph, DBP.ph, HR.ph: systolic and diastolic arterial pressure and heart rate during pre-hospital phase. (ph)

• HemoCue.init: prehospital capillary hemoglobin concentration. (ph)

• SpO2.min: peripheral oxygen saturation, measured by pulse oxymetry, to estimate oxygen content in the blood. (ph)

• Cristalloid.volume: total amount of prehospital administered cristalloid fluid resuscitation (volume expansion). (ph)

• Shock.index.ph: ratio of heart rate and systolic arterial pressure during pre-hospital phase. (ph)

• Delta.shock.index: Difference of shock index between arrival at the hospital and arrival on the scene. (h)

• Delta.hemoCue: Difference of hemoglobin level between arrival at the hospital and arrival on the scene. (h)