Semi-parametric regression when some (expensive) covariates are missing by design

Abstract

The paper deals with the scenario where some covariates are observed by design for a subset of the observations only. In the example treated in the paper this occurs with a two phase sampling scheme where in the first phase a relatively large sample is drawn to record a response variable Y and a set of (cheap) covariates x. In a second phase a smaller sample is drawn from the first phase sample where additional (usually expensive) covariates z are also recorded. The second phase can be drawn with unequal probability sampling, where the sampling weights depend on the observed Y and x. The overall intention is to fit a regression model of Y on both, x and z. Due to the design of the data collection we are faced with missing values for z for a majority of observations. We propose an approximate estimation approach using semi-parametric mean and variance regression of Y on x only and augment this fit with a full regression model of Y on x and z. The idea extends the approach of Little (1992) towards non-normal data and non-linear models. The proposed estimation is numerically rather simple and performs convincingly well in simulation studies compared to alternatives such as complete-case and multiple imputation analysis.

Appendices

Appendix A: Penalized spline smoothing

Penalized spline smoothing is a very general and numerically stable routine for fitting smooth functions. We refer to Ruppert et al. (2003, 2009) for an excessive discussion of the field. Subsequently we sketch the basic ideas. The main principle is to replace the smooth function m(x) in model (7) and the smooth functions $m_1(x)$ and $\sigma ^2_1(x)$ in model (8) by spline bases representation. That is we make $m(x) = B(x)u$ and $m_1(x) = B(x)u_1$, $\sigma ^2_i (x) = \exp (B_\sigma (x) u_\sigma$), where B(x) is spline basis and so is $B_\sigma (x)$ and in principle we can set $B(x) = B_\sigma (x)$. A convenient setting is to use a B-spline basis (see Boor 1972), which is constructed from piece-wise polynomial functions, tied together in a continuous (and where necessary differentable) way. This makes the whole model parametric where the spline coefficients u in model (7) and the coefficients $u_1$ and $u_\sigma$ are the parameters which need to be estimated. Given that B(x) is chosen as high dimensional basis we find the coefficient vectors to be high dimensional as well. Estimation will induce large estimation variability which is why Eilers and Marx (1996) proposed to impose a penalization on u, e.g. neighboring coefficients should not differ very much. Such penalization can be written as quadratic form $\lambda u^t D u$ for an appropriately chosen penalty matrix D. This leads to the penalized likelihood

$$\begin{aligned} l(\theta ) - \frac{1}{2}\lambda u^t D u \end{aligned}$$

(A1)

where $\theta$ is the parameter vector of the model that does also contain the coefficient vector u. Parameter $\lambda$ plays the role of the smoothing parameter and increasing $\lambda$ will lead to a more penalized fit. Comprehending the latter component in (A1) as log prior leads to a Bayesian framework so that

$$\begin{aligned}&u \sim N(0, \lambda ^{-1} D^-) \\&y|u \sim \exp (l(\theta )) \end{aligned}$$

where $D^-$ stands for the (generalized) inverse of D. Now $\lambda$ plays the role of a hyper parameter which can be estimated using empirical Bayes ideas. We refer to Wand (2003) for details in this direction.

Appendix B: Multivariate metrical variables

We repeat the simulation for bivariate x and simulate data from the model

$$\begin{aligned} Y = \beta _0 + m(x_1) + v(x_2) + z\beta _z + \varepsilon \end{aligned}$$

(B1)

where $\varepsilon \sim N(0, \sigma )$ and $z=(z_1,z_2,z_3)$ is a vector of binary covariates which are correlated with $x=(x_1, x_2)$. For the functional forms $m(x_1)$ and $v(x_2)$ we use the same response functions as shown in Fig. 2 for different values of $z_1, z_2$ and $z_3$ for univariate x. The population size, the first and second phase sample size and the true $\beta _z$ values for covariates z remain unchanged as for model (14). We consider two cases here. In the first case, the response variable Y and the covariates $x_1$, $x_2$ are observed in first phase $s_1$ while covariates z are missing and observed in $s_2$ only. In the second case, we observe the values of a response variable Y and the covariate $x_1$ while $x_2$ and z are missing in first phase and observed in second phase sample only. We use $\varepsilon \sim N(0, 1)$ and $\varepsilon \sim N(0, 1.5)$ for model (B1) for the case 1 and 2, respectively. The covariates $x_1$ and $x_2$ are generated independently from a uniform distribution with parameters $x_1 \sim (20, 160)$ and $x_2 \sim (25, 100)$ for case 1 and $x_1 \sim (20, 160)$ and $x_2 \sim (5, 20)$ for case 2. The covariates z are generated from a Bernoulli distribution using both variables $x_1$ and $x_2$ in response functions similar as in the univariate case. The ratio of the prediction errors are shown in Figs. 11 and 12 for case 1, and 13 and 14 for case 2 and the median values of mean squared prediction error for both cases are given in Table 5. The overall interpretation remains unchanged. The bias and the estimated variance of the regression coefficient $\hat{\beta _2}$ are given in Table 6. The results are similar to those for univariate x as discussed in Sect. 3.1.

Fig. 11

$n_2=200$: ratio of mean squared prediction error for simulated data. Second phase sample selection with equal, Tille covariate and Tille residual dependent probability sampling with case 1

Fig. 12

$n_2=400$: ratio of mean squared prediction error for simulated data. Second phase sample selection with equal, Tille covariate and Tille residual dependent probability sampling with case 1

Fig. 13

$n_2=200$: ratio of mean squared prediction error for simulated data. Second phase sample selection with equal, Tille covariate and Tille residual dependent probability sampling with case 2

Fig. 14

$n_2=400$: ratio of mean squared prediction error for simulated data. Second phase sample selection with equal, Tille covariate and Tille residual dependent probability sampling with case 2

Table 5 Median of mean squared prediction error for simulated data (for bivariate x)

Table 6 Bias and estimated variance for regression coefficient ${\hat{\beta }}_2$ for simulated data (for bivariate x)