Information-based optimal subdata selection for non-linear models

Abstract

Subdata selection methods provide flexible tradeoffs between computational complexity and statistical efficiency in analyzing big data. In this work, we investigate a new algorithm for selecting informative subdata from massive data for a broad class of models, including generalized linear models as special cases. A connection between the proposed method and many widely used optimal design criteria such as A-, D-, and E-optimality criteria is established to provide a comprehensive understanding of the selected subdata. Theoretical justifications are provided for the proposed method, and numerical simulations are conducted to illustrate the superior performance of the selected subdata.

Appendix A Proofs

1.1 A.1 Proof of Theorem 1

Proof

Note that

$$\begin{aligned} \varvec{\delta }^{opt}_{T}= \arg \max _{\varvec{\delta }}\textrm{tr}\left( {\mathcal {I}}(\varvec{\delta })\right) =\arg \max _{\varvec{\delta }}\delta _i\Vert \varvec{x}_i\Vert ^2. \end{aligned}$$

Thus, the T-optimal subdata is the subdata with the k largest values of $\Vert \varvec{x}_i\Vert ^2$. $\square$

1.2 A.2 Proof of Theorem 2

Before proving Theorem 2, we need the following lemma.

Lemma A.1

Let $\lambda _{1},\ldots ,\lambda _{d}$ be the d eigenvalues of ${\mathcal {I}}(\varvec{\delta })$ with $\lambda _{\min }=\lambda _{1}\le \ldots \le \lambda _{d}=\lambda _{\max }$. Assume that $\textrm{var}(\varvec{Z}_{1}^*)\le \ldots \le \textrm{var}(\varvec{Z}_{d}^*)$ with $\textrm{var}(\varvec{Z}_j^*)$ being the sample variance for the jth column of $\varvec{Z}^*$. For any subdata of size n, represented by $\varvec{\delta }$, it holds that

$$\begin{aligned} (n-1)\lambda _{\min }(\varvec{R}^*)\textrm{var}(\varvec{Z}_{j}^*)\le \lambda _j({\mathcal {I}}(\varvec{\delta })) \le (n-1)\lambda _{\max }(\varvec{R}^*)\textrm{var}(\varvec{Z}_{j}^*), \end{aligned}$$

(21)

where $\varvec{R}^*$ is the sample correlation matrix of $\varvec{Z}^*$.

Proof

Recall $\varvec{Z}^*=(\varvec{z}_{1}^*,\ldots , \varvec{z}_{n}^*)^\textrm{T}$. Let ${\textbf{1}}$ be a $n\times 1$ vector of ones, and $\textrm{var}(\varvec{Z}_j^*)$ be the sample variance for the jth column of $\varvec{Z}^*$, $j=1,\ldots , d$. It follows that

$$\begin{aligned} {\mathcal {I}}(\varvec{\delta },\varvec{\theta })=&\varvec{Z}^{*\mathrm T}\varvec{Z}^*\\ \ge&\varvec{Z}^{*\textrm{T}}\Big (\varvec{I}-\frac{1}{n}{\textbf{1}}{\textbf{1}}^\textrm{T}\Big )\varvec{Z}^*\\ =&(n-1) \begin{bmatrix} \sqrt{\textrm{var}(\varvec{Z}_{1}^*)} &{} &{} \\ {} &{} \ddots &{} \\ &{} &{} \sqrt{\textrm{var}(\varvec{Z}_{d}^*)} \end{bmatrix} {\textbf{R}}^* \begin{bmatrix} \sqrt{\textrm{var}(\varvec{Z}_{1}^*)} &{} &{} \\ &{} \ddots &{} \\ &{} &{} \sqrt{\textrm{var}(\varvec{Z}_{d}^*)} \end{bmatrix}. \end{aligned}$$

Note the fact that for any matrices $A\ge 0$ and $B\ge 0$, it holds that $\lambda _{\min }(B)\lambda _j(A^2)\le \lambda _j(ABA)\le \lambda _{\max }(B)\lambda _j(A^2)$. The desired result follows immediately by letting $B=\varvec{R}^*$ and $A=\textrm{diag}(\sqrt{\textrm{var}(\varvec{Z}_{1}^*)},\ldots ,\sqrt{\textrm{var}(\varvec{Z}_{d}^*)})$. $\square$

Proof of Theorem 2

Recall that $\lambda _{1},\ldots ,\lambda _{d}$ are d eigenvalues of ${\mathcal {I}}(\varvec{\delta })$ with $0<\lambda _{\min }=\lambda _{1}\le \ldots \le \lambda _{d}=\lambda _{\max }$. We have

$$\begin{aligned} \{\textrm{tr}(d^{-1}{\mathcal {I}}^{-1}(\varvec{\delta }))\}^{-1}&=\left( d^{-1}\sum _{j=1}^d\lambda _j^{-1}\right) ^{-1},\end{aligned}$$

(22)

$$\begin{aligned} \{\det ({\mathcal {I}}(\varvec{\delta }))\}^{1/d}&=\left( \prod _{j=1}^d\lambda _j\right) ^{1/d}. \end{aligned}$$

(23)

Thus (13)–(15) can be easily obtained by Lemma A.1. $\square$

1.3 A.3 Proof of Theorem 3

Proof

For each sample variance,

$$\begin{aligned} \textrm{var}(\tilde{\varvec{Z}}_{j}^*) =&\frac{1}{n-1}\sum _{i=1}^{n}({\tilde{{z}}}_{ij}^*-\bar{{\tilde{z}}}_j^*)^2\nonumber \\ =&\frac{({\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j})^2}{k-1}\sum _{i=1}^{n}\left( \frac{{\tilde{z}}_{ij}^*-\bar{{\tilde{z}}}_j^*}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2\nonumber \\ \ge&\frac{({\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j})^2}{n-1} \left( \sum _{i=1}^{r}+\sum _{i=N-r+1}^N\right) \left( \frac{{\tilde{z}}_{(i)j}-\bar{{\tilde{z}}}_j^*}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2. \end{aligned}$$

(A.1)

For the first summation in (A.4),

$$\begin{aligned}&\sum _{i=1}^{r}\left( \frac{{\tilde{z}}_{(i)j}-\bar{{\tilde{z}}}_j^*}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2 =\frac{1}{n^2}\sum _{i=1}^{r}\left( \frac{\sum _{s=1}^n{{\tilde{z}}}_{sj}^*-n{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2. \end{aligned}$$

(A.2)

Each term in the summation of (A.5) can be written as

$$\begin{aligned}&\frac{\sum _{s=1}^n{{\tilde{z}}}_{sj}^*-n{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\nonumber \\&\quad =\sum _{s=N-r+1}^N\frac{{\tilde{z}}_{(s)j}-{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}} +\sum _{s=1}^r\frac{{\tilde{z}}_{(s)j}-{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\nonumber \\&\qquad +\sum _{l\ne j}\left( \sum _{s=1}^r +\sum _{s=N-r+1}^N\right) \frac{{\tilde{z}}_j^{(s)l}-{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}, \end{aligned}$$

(A.3)

where ${\tilde{z}}_j^{(s)l}$ is the jth dimension of the subdata point selected according to $\{\tilde{{z}}_{il},i=1,\ldots ,N\}$ in the second step of Algorithm 1. From the Assumption 1, we have that for $s,i\le r$, ${({\tilde{z}}_{(s)j}-{\tilde{z}}_{(i)j})}/{({\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j})}=o_{P}(1)$ and ${({\tilde{z}}_j^{(s)l}-{\tilde{z}}_{(i)j})}/{({\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j})}$ is either positive or $o_{P}(1)$. Thus (A.6) implies

$$\begin{aligned}&\frac{\sum _{s=1}^n{{\tilde{z}}}_{sj}^*-n{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}} \ge \sum _{s=N-r+1}^N\frac{{\tilde{z}}_{(s)j}-{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}. \end{aligned}$$

(A.4)

From assumptions 1 and 2, for $s\ge N-r+1$ and $i\le r$, as $N\rightarrow \infty$,

$$\begin{aligned} \frac{\tilde{{z}}_{(s)j}-\tilde{{z}}_{(i)j}}{\tilde{{z}}_{(N)j}-\tilde{{z}}_{(1)j}} =\frac{\tilde{{z}}_{(s)j}-\tilde{{z}}_{(N)j}}{\tilde{{z}}_{(N)j}-\tilde{{z}}_{(1)j}} +\frac{\tilde{{z}}_{(N)j}-\tilde{{z}}_{(1)j}}{\tilde{{z}}_{(N)j}-\tilde{{z}}_{(1)j}} +\frac{\tilde{{z}}_{(1)j}-\tilde{{z}}_{(i)j}}{\tilde{{z}}_{(N)j}-\tilde{{z}}_{(1)j}} =1+o_{P}(1) \end{aligned}$$

(A.5)

From (A.5), (A.7) and (A.8),

$$\begin{aligned} \sum _{i=1}^{r}\left( \frac{{\tilde{z}}_{(i)j}-\bar{{\tilde{z}}}_j^*}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2 \ge&\frac{1}{n^2}\sum _{i=1}^{r}\left( \sum _{s=N-r+1}^N \frac{{\tilde{z}}_{(s)j}-{\tilde{z}}_{(i)j}}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}+o_{P}(1)\right) ^2 =\frac{r^3}{n^2}+o_{P}(1). \end{aligned}$$

(A.6)

Similarly,

$$\begin{aligned} \sum _{i=N-r+1}^N\left( \frac{{\tilde{z}}_{(i)j}-\bar{{\tilde{z}}}_j^*}{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}\right) ^2 \ge \frac{r^3}{n^2}+o_{P}(1). \end{aligned}$$

(A.7)

Combining (A.4), (A.9) and (A.10),

$$\begin{aligned} \textrm{var}(\tilde{\varvec{Z}}_{j}^*) \ge \frac{2r^3({\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j})^2}{n^2(n-1)}(1+o_{P}(1)). \end{aligned}$$

(A.8)

If ${\tilde{z}}_{(1)j}/{\tilde{z}}_{(N)j}\overset{P}{\rightarrow }0$ or $\pm \infty$, then

$$\begin{aligned} \frac{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}{|{\tilde{z}}|_{(N)j}}=1+o_{P}(1). \end{aligned}$$

(A.9)

Combining (A.11) and (A.12) shows that

$$\begin{aligned} \textrm{var}(\tilde{\varvec{Z}}_{j}^*) \ge \frac{2r^3|{\tilde{z}}|_{(N)j}^2}{n^2(n-1)}(1+o_{P}(1)). \end{aligned}$$

(A.10)

Thus, the desired results come from Theorem 2 and Slutsky’s theorem.

If ${\tilde{z}}_{(N)j}\rightarrow \infty$ and ${\tilde{z}}_{(1)j}$ is bounded below, or ${\tilde{z}}_{(1)j}\rightarrow -\infty$ and ${\tilde{z}}_{(N)j}$ is bounded above, then

$$\begin{aligned} \frac{{\tilde{z}}_{(N)j}-{\tilde{z}}_{(1)j}}{|{\tilde{z}}|_{(N)j}}=1+o_{P}(1). \end{aligned}$$

(A.11)

From (A.11) and (A.14), it follows that

$$\begin{aligned} \textrm{var}(\tilde{\varvec{Z}}_{j}^*) \ge \frac{2r^3|{\tilde{z}}|_{(N)j}^2}{n^2(n-1)}(1+o_{P}(1)). \end{aligned}$$

(A.12)

Thus, the desired results come from Theorem 2 and Slutsky’s theorem.

$\square$

1.4 A.4 Some details of Example 3 and Proposition 4

Lemma A.2

Suppose $\varvec{x}\sim N(\varvec{\mu },\Sigma )$ with $\Sigma >0$ and $\varvec{\theta }$ has more than one nonzero elements. Then it holds that $(x_j,\varvec{\theta }^\textrm{T}\varvec{x})^\textrm{T}$ is still a nondegenerate normal distribution for all j.

Lemma A.3

Suppose $\varvec{x}\sim N(\varvec{\mu },\Sigma )$ with $\Sigma >0$, and $\varvec{\theta }$ lies in a compact ball with more than one nonzero element. Then for any given M, and $C>0$, it holds that

$$\begin{aligned}&\Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}> M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )>0,\quad \\&\Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}< -M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )>0, \end{aligned}$$

for all j.

Proof of Proposition 4

Note that the jth dimension of $\tilde{\varvec{z}}$ can be written as

$$\begin{aligned}\tilde{{z}}_j=\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}{\tilde{\varvec{\theta }}}/2}+e^{\varvec{x}^\textrm{T}{\tilde{\varvec{\theta }}}/2}}.\end{aligned}$$

For any j, one can see that

$$\begin{aligned} \Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}> M\Big )\ge \Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}> M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )>0, \end{aligned}$$

where M and C are some constant independent of $\varvec{x}$.

Similarly, one can show that

$$\begin{aligned} \Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}< -M\Big )\ge \Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}< -M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )>0. \end{aligned}$$

From Lemma A.3, the desired result follows. $\square$

Details of Example 3

Grounded on the two lemmas, we can see that for any $M>0$ and $j=1,\ldots ,d$,

$$\begin{aligned} \Pr (\tilde{{z}}_{(N)j}M)&=1-\Pr (\tilde{{z}}_{j,(N)}\le M)\end{aligned}$$

(A.13)

$$\begin{aligned}&=1-{\Pr }^N(\tilde{{z}}_{j}\le M)\end{aligned}$$

(A.14)

$$\begin{aligned}&=1-{\Pr }^N\left( \frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}\le M\right) . \end{aligned}$$

(A.15)

From Lemma A.3, it is clear to see that

$$\begin{aligned}&{\Pr }^N\Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}\le M\Big ) \rightarrow 0. \end{aligned}$$

(A.16)

Thus the result follows. $\square$

Proof of Lemma A.2

Without loss of generality, we only consider the case $j=1$ here. Note that $\varvec{x}$ is a multivariate normal distribution. Let ${\varvec{e}_j}$ be a unit vector with jth element being one and $C=(\varvec{e}_j,\varvec{\theta })$. Note that $\varvec{\theta }$ has more than one nonzero element by assumption. It is clear to see the rank of C equals two. Since $\Sigma >0$, the rank of $C^\textrm{T}\Sigma C$ is also two. The $\check{\varvec{x}}=C^\textrm{T}\varvec{x}=(x_j,\varvec{\theta }^\textrm{T}\varvec{x})^\textrm{T}$ is also a non degenerate normal distribution with mean $C^\textrm{T}\varvec{\mu }=(\mu _{j},\varvec{\mu }^\textrm{T}\varvec{\theta })^\textrm{T}$, and variance

$$\begin{aligned} C^\textrm{T}\Sigma C={\check{\Sigma }}=\left( \begin{array}{ll} \sigma _{jj} &{} {{\check{\sigma }}}_{12}\\ {{\check{\sigma }}}_{12}^\textrm{T}&{} {{\check{\sigma }}}_{22} \end{array} \right) , \end{aligned}$$

where $\mu _{j}$ is the jth elements in $\varvec{\mu }$, $\sigma _{jj}$ is the (j, j)th element of $\Sigma$, ${{\check{\sigma }}}_{12}$ equals to $(\Sigma _{\cdot j}^\textrm{T}\varvec{\theta })$ with $\Sigma _{\cdot j}$ being the jth column of $\Sigma$, and ${{\check{\sigma }}}_{22}=\varvec{\theta }^\textrm{T}\Sigma \varvec{\theta }$.

$\square$

Proof of Lemma A.3

For the first result, simple calculation yields that

$$\begin{aligned}&\Pr \Big (\frac{x_{j}}{e^{-\varvec{x}^\textrm{T}\varvec{\theta }/2}+e^{\varvec{x}^\textrm{T}\varvec{\theta }/2}}> M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )\end{aligned}$$

(A.17)

$$\begin{aligned}&\ge \Pr \Big (x_{j}> 2e^{C}M,\ |\varvec{x}^\textrm{T}\varvec{\theta }|\le 2C\Big )>0, \end{aligned}$$

(A.18)

since $(x_j,\varvec{x}^\textrm{T}\varvec{\theta })^\textrm{T}$ are non degenerate normal distribution by the fact proved in Lemma A.2. The second result is quite similar. Thus we omit it.

$\square$