A review on design inspired subsampling for big data

Abstract

Subsampling focuses on selecting a subsample that can efficiently sketch the information of the original data in terms of statistical inference. It provides a powerful tool in big data analysis and gains the attention of data scientists in recent years. In this review, some state-of-the-art subsampling methods inspired by statistical design are summarized. Three types of designs, namely optimal design, orthogonal design, and space filling design, have shown their great potential in subsampling for different objectives. The relationships between experimental designs and the related subsampling approaches are discussed. Specifically, two major families of design inspired subsampling techniques are presented. The first aims to select a subsample in accordance with some optimal design criteria. The second tries to find a subsample that meets some design requirements, including balancing, orthogonality, and uniformity. Simulated and real data examples are provided to compare these methods empirically.

Appendix

1.1 A summary of subsampling methods

See Table 8.

Table 8 A summary of subsampling methods

1.2 Technical details

Proof of Theorem 1

The estimator $\tilde{{\varvec{\beta }}}$ is the maximizer of $L^*({\varvec{\beta }})$ in (1), so conditional on ${\mathcal {F}}_n$, $\sqrt{r}(\tilde{{\varvec{\beta }}}-\hat{{\varvec{\beta }}})$ is the maximizer of $\Lambda ({\varvec{\eta }})=n^{-1}rL^*(\hat{{\varvec{\beta }}}+{{\varvec{\eta }}}/\sqrt{r})-n^{-1}rL^*(\hat{{\varvec{\beta }}})$. For natation simplicity, let ${\dot{f}}({\varvec{\beta }})$ and $\ddot{f}({\varvec{\beta }})$ be the first and second order derivatives of a given function $f({\varvec{\beta }})$ with respect to ${\varvec{\beta }}$. Preforming Taylor’s expansion on $\Lambda ({\varvec{\eta }})$,

$$\begin{aligned} \Lambda ({\varvec{\eta }})=\frac{\sqrt{r}}{n}{{\varvec{\eta }}}^T{\dot{L}}^*(\hat{{\varvec{\beta }}})+\frac{1}{2n}{{\varvec{\eta }}}^T{\ddot{L}}^*(\hat{{\varvec{\beta }}}){{\varvec{\eta }}}+R, \end{aligned}$$

(A.1)

where R is the remainder term. Under Assumption 3, one can see that R goes to zero in probability conditional on ${\mathcal {F}}_n$, i.e., $R=o_{P|{\mathcal {F}}_n}(1)$. Precisely,

$$\begin{aligned} |R|\le \frac{p^3\Vert {\varvec{\eta }}\Vert ^3}{3\sqrt{r}}\times \frac{1}{n}\sum _{i=1}^n H({\varvec{x}}_i,y_i)=o_{P}(1), \end{aligned}$$

(A.2)

thus the result holds by Theorem 3.3 in Xiong and Li (2008).

Recall that ${\dot{\ell }}(y_i|{\varvec{x}}_i, {{\varvec{\beta }}})=n^{-1}(\partial \log f(y_i|{\varvec{x}}_i,{\varvec{\beta }})/\partial {\varvec{\beta }})$, and the corresponding subsampling version ${\dot{\ell }}^*(y_i|{\varvec{x}}_i, {{\varvec{\beta }}})=(n\pi _i)^{-1}\delta _i(\partial \log f(y_i|{\varvec{x}}_i,{\varvec{\beta }})/\partial {\varvec{\beta }})$. One can see that ${\dot{\ell }}^*(y_i|{\varvec{x}}_i, {{\varvec{\beta }}})$ are independent random vectors conditional on ${\mathcal {F}}_n$. Direct calculation yields

$$\begin{aligned} E({\dot{L}}^*(\hat{{\varvec{\beta }}})|{\mathcal {F}}_n)&={\varvec{0}}, \end{aligned}$$

(A.3)

$$\begin{aligned} \textrm{var}({\dot{L}}^*(\hat{{\varvec{\beta }}})|{\mathcal {F}}_n)&=\sum _{i=1}^n\frac{1-\pi _i}{\pi _i}{\dot{\ell }}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}}){\dot{\ell }}^{T}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}})\nonumber \\&=\sum _{i=1}^n\frac{1}{\pi _i}{\dot{\ell }}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}}){\dot{\ell }}^{T}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}})+o_{P|{\mathcal {F}}_n}(1)\nonumber \\&=V_c+o_{P|{\mathcal {F}}_n}(1), \end{aligned}$$

(A.4)

where the second last equation comes from the facts

$$\begin{aligned}&\sum _{i=1}^n{\dot{\ell }}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}}){\dot{\ell }}^{T}(y_i|{\varvec{x}}_i, \hat{{\varvec{\beta }}})\\ \le&\sup _{{\varvec{\beta }}\in \Theta }\frac{1}{n^2}\sum _{i=1}^n(\partial \log f(y_i|{\varvec{x}}_i,{\varvec{\beta }})/\partial {\varvec{\beta }})(\partial \log f(y_i|{\varvec{x}}_i,{\varvec{\beta }})/\partial {\varvec{\beta }})^T\\ \le&\frac{1}{n^2}\sup _{{\varvec{\beta }}\in \Theta }\sum _{i=1}^n \Vert \partial \log f(y_i|{\varvec{x}}_i,{\varvec{\beta }})/\partial {\varvec{\beta }}\Vert ^2 I_p =o_{P|{\mathcal {F}}_n}(1), \end{aligned}$$

under Assumption 2 and $r/n\rightarrow 0$.

Under Assumption 5, the Lindeberg-Feller condition holds under the condition distribution. Thus from the central limit theorem,

$$\begin{aligned} \frac{\sqrt{r}}{n}{\dot{L}}^*(\hat{{\varvec{\beta }}})\rightarrow N({\varvec{0}},V_c). \end{aligned}$$

(A.5)

Under Assumption 2, by the law of large number, we know that

$$\begin{aligned} \frac{1}{n}{\ddot{L}}^*(\hat{{\varvec{\beta }}})=-{\mathcal {J}}_{\hat{{\varvec{\beta }}}}+o_{P|{\mathcal {F}}_n}(1). \end{aligned}$$

(A.6)

Thus the desired result follows by the Basic Corollary in Lid Hjort and Pollard (2011) and Theorem 3.3 in Xiong and Li (2008). $\square$

Proof of Theorem 2

Note that ${\varvec{z}}=\Sigma ^{-1/2}{\varvec{x}}$ is standard normal and

$$\begin{aligned}E(X_s^{ \textrm{T} }X_s)=\Sigma ^{1/2}\left( \int {\mathbb {I}}({\varvec{z}}^{ \textrm{T} }{\varvec{z}}>c){\varvec{z}}{\varvec{z}}^{ \textrm{T} }\phi _I({\varvec{z}})d{\varvec{z}}\right) \Sigma ^{1/2},\end{aligned}$$

where $\phi _I({\varvec{z}})$ is the density function of the p-dimensional standard normal distribution.

By the inequality of arithmetic and geometric means for the eigenvalues, one can see that

$$\begin{aligned} p\left( \det (\int {\mathbb {I}}({\varvec{z}}^{ \textrm{T} }{\varvec{z}}>c){\varvec{z}}{\varvec{z}}^{ \textrm{T} }\phi _I({\varvec{z}})d{\varvec{z}})\right) ^{1/p}&\le \textrm{tr}\left( \int {\mathbb {I}}({\varvec{z}}^{ \textrm{T} }{\varvec{z}}>c){\varvec{z}}{\varvec{z}}^{ \textrm{T} }\phi _I({\varvec{z}})d{\varvec{z}}\right) \\&=\int _{{\varvec{z}}^{ \textrm{T} }{\varvec{z}}>c}\Vert {\varvec{z}}\Vert ^2\phi _I({\varvec{z}})d{\varvec{z}}, \end{aligned}$$

the equality holds by the fact that $\int _{{\varvec{z}}^{ \textrm{T} }{\varvec{z}}>c}\Vert {\varvec{z}}\Vert ^2\phi _I({\varvec{z}})d{\varvec{z}}$ is a nonzero multiple of the identity matrix.

Note that for any selection rule $\textrm{tr}\left( \int {\mathbb {I}}(g(\Sigma ^{1/2}{\varvec{z}})){\varvec{z}}{\varvec{z}}^{ \textrm{T} }\phi _I({\varvec{z}})d{\varvec{z}}\right) \!=\!\int _{g(\Sigma ^{1/2}{\varvec{z}})} \Vert {\varvec{z}}\Vert ^2\phi _I({\varvec{z}})d{\varvec{z}}$. Clearly, such kind of selection also achieve the upper bound of $\textrm{tr}\big (\int {\mathbb {I}}(g(\Sigma ^{1/2}{\varvec{z}})){\varvec{z}}{\varvec{z}}^{ \textrm{T} }\phi _I({\varvec{z}})d{\varvec{z}}\big )$ for any $g(\cdot )$ satisfies the constraint. Thus the result follows by the fact $\det (AB)=\det (A)\det (B)$ for any $A,B>0$. $\square$

1.3 Brief introduction of the optimal transport map

To ease the conversation, we only present a specific optimal transport map from a general distribution F on $\Omega \subseteq {\mathbb {R}}^{p+1}$ to an uniform distribution $F_\textrm{unif}$ on $[0,1]^{p+1}$. For the general definition of the optimal transport map, we refer to Villani (2008) for more details. Let $\#$ be the push-forward operator, such that for all measurable $B\subseteq \Omega$, we have $\phi _\#(F)(B)=F(\phi ^{-1}(B))$. The optimal transport map $\phi ^*$ of our interest is the one that minimizes the $l_2$ cost, $\int _\Omega \Vert {\varvec{z}}- \phi ({\varvec{z}}) \Vert ^2 \text{ d }F$ with respect to all the measures preserving maps $\phi :\Omega \rightarrow [0,1]^{p+1}$ such that $\phi _\#(F) = F_{\textrm{unif}}$ and $\phi ^{-1}_\#(F_\textrm{unif}) = F$. It has been shown that when $\Omega ={\mathbb {R}}$, $\phi ^*$ is equivalent to the cumulative distribution function F (Villani 2008).

For the given observations $\{{\varvec{z}}_i\}_{i=1}^n$ comes from F and $\{{\varvec{s}}_i\}_{i=1}^n$ from $F_{\textrm{unif}}$. The empirical optimal transport map can be estimated by the auction algorithm or the refined auction algorithm (Bertsekas 1992). To further alleviate the computational burden, in practice, some projection-based methods can be used to approximate the optimal transport map $\phi ^*$. Readers may refer Zhang et al. (2022) for detail discussions on computational issue.

1.4 Additional simulation results for $n=1,000,000$

In this subsection, we will provide the log EMSE for the subsampling methods with large n. More precisely, we consider $n=1,000,000$ with the three cases given in Sect. 6. Due to the limitation of the computational resources, we opt to report the results for the methods whose computing time is no more than $O(np^2)$. The results for the linear regression model and Logistic regression model are reported in Tables 9 and 10, respectively.

Table 9 The log EMSE of the different subsampling methods for the linear models with different r

Table 10 The log EMSE of the different subsampling methods for the Logistic regression models with different r