Simultaneous Learning the Dimension and Parameter of a Statistical Model with Big Data

Abstract

Estimating the dimension of a model along with its parameters is fundamental to many statistical learning problems. Traditional model selection methods often approach this task by a two-step procedure: first estimate model parameters under every candidate model dimension, then select the best model dimension based on certain information criterion. When the number of candidate models is large, however, this two-step procedure is highly inefficient and not scalable. We develop a novel automated and scalable approach with theoretical guarantees, called mixed-binary simultaneous perturbation stochastic approximation (MB-SPSA), to simultaneously estimate the dimension and parameters of a statistical model. To demonstrate the broad practicability of the MB-SPSA algorithm, we apply the MB-SPSA to various classic statistical models including K-means clustering, Gaussian mixture models with an unknown number of components, sparse linear regression, and latent factor models with an unknown number of factors. We evaluate the performance of the MB-SPSA through simulation studies and an application to a single-cell sequencing dataset in terms of accuracy, running time, and scalability. The code implementing the MB-SPSA is available at http://github.com/wanglong24/MB-SPSA.

Appendix: Proof of Theorem 1

Proof

Denote $L({\hat{{\varvec{\theta }}}}_k^{(+)}) = L_k^{(+)}$ and $L_k^{(-)} = L({\hat{{\varvec{\theta }}}}_k^{(-)})$. Before starting the main proof, we first define some useful notations below

$$\begin{aligned} {\bar{\varvec{g}}}_k&= {\mathbb {E}}\left[ {\hat{\varvec{g}}}_k \mid {\hat{{\varvec{\theta }}}}_k\right] , \end{aligned}$$

(12)

$$\begin{aligned} {\hat{{\varvec{b}}}}_k&= \frac{1}{2}(\varvec{C}_k \circ {\varvec{\Delta }}_k)\left[ L_k^{(+)} - L_k^{(-)}\right] - {\bar{\varvec{g}}}_k, \end{aligned}$$

(13)

$$\begin{aligned} {\hat{{\varvec{e}}}}_k&= {\hat{\varvec{g}}}_k - \frac{1}{2}(\varvec{C}_k \circ {\varvec{\Delta }}_k)\left[ L_k^{(+)} - L_k^{(-)}\right] = \frac{1}{2}(\varvec{C}_k \circ {\varvec{\Delta }}_k)\left[ \epsilon _k^{(+)} - \epsilon _k^{(-)}\right] , \end{aligned}$$

(14)

where the expectation in (12) is taken over both the perturbation vector ${\varvec{\Delta }}_k$ and the noise term $\epsilon _k$. Using (12), (13) and (14), we can write the updating equation as

$$\begin{aligned} {\hat{{\varvec{\theta }}}}_{k+1}&= {\hat{{\varvec{\theta }}}}_k - a_k {\hat{\varvec{g}}}_k \nonumber \\&= {\hat{{\varvec{\theta }}}}_k - a_k \left( \frac{1}{2}(\varvec{C}_k \circ {\varvec{\Delta }}_k)\left[ L_k^{(+)} - L_k^{(-)}\right] +\frac{1}{2}(\varvec{C}_k \circ {\varvec{\Delta }}_k)\left[ \epsilon _k^{(+)} - \epsilon _k^{(-)}\right] \right) \nonumber \\&={\hat{{\varvec{\theta }}}}_k - a_k \left( {\bar{\varvec{g}}}_k + {\hat{{\varvec{b}}}}_k + {\bar{\varvec{g}}}_k\right) . \end{aligned}$$

(15)

For any $\omega \in \Omega _0$ such that $P(\Omega _0) = 1$, since $\{{\hat{{\varvec{\theta }}}}_k(\omega )\}$ is a bounded sequence by Assumption 2, the Bolzano–Weierstrass Theorem implies that there exists $\Omega _1 \subset \Omega$ such that $P(\Omega _1) = 1$ and for any $\omega \in \Omega _1$ there exists a convergent subsequence $\{{\hat{{\varvec{\theta }}}}_{k_s}(\omega )\}$. Denote the limiting point of the convergent subsequence as ${\varvec{\theta }}'(\omega )$. For simplicity, the notation $\omega$ is suppressed below.

According to (15), we can write

$$\begin{aligned} {\varvec{\theta }}' - {\hat{{\varvec{\theta }}}}_{k_s} = \lim _{s\rightarrow \infty } \sum _{i=s}^{n} ({\hat{{\varvec{\theta }}}}_{k_{i+1}} - {\hat{{\varvec{\theta }}}}_{k_i}) = -\lim _{s\rightarrow \infty } \sum _{i=s}^{n} a_{k_i}\left( {\bar{\varvec{g}}}_{k_i} + {\hat{{\varvec{b}}}}_{k_i} + {\bar{\varvec{g}}}_{k_i}\right) , \end{aligned}$$

(16)

Since ${\varvec{\theta }}' - {\hat{{\varvec{\theta }}}}_{k_s} \rightarrow \varvec{0}$ as $s \rightarrow \infty$, we will show below that all the three terms of the right-hand side of (16) must also converge to $\varvec{0}$.

First note that by Assumption 3 and (13), we have

$$\begin{aligned} {\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_{k+1} \mid {\mathcal {F}}_k \right] = \varvec{0}, \end{aligned}$$

which implies that $\{\sum _{i=k}^{m} a_i{\varvec{b}}_i\}_{m\ge k}$ is a martingale sequence as

$$\begin{aligned} {\mathbb {E}}\left[ \sum _{i=k}^{m+1} a_i{\hat{{\varvec{b}}}}_i \mid {\mathcal {F}}_m\right] = \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i + a_{m+1}{\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_{m+1} \mid {\mathcal {F}}_m\right] = \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i. \end{aligned}$$

Given that $\{\sum _{i=k}^{m} a_i{\varvec{b}}_i\}_{m\ge k}$ is a martingale sequence, the Doob’s martingale inequality implies that for any $\eta > 0$

$$\begin{aligned} P\left( \sup _{m\ge k} \left\| \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right) \le \eta ^{-2} {\mathbb {E}}\left[ \left\| \sum _{i=k}^{\infty } a_i{\hat{{\varvec{b}}}}_i \right\| ^2 \right] = \eta ^{-2} \sum _{i=k}^{\infty }a_i^2{\mathbb {E}}\left[ \left\| {\hat{{\varvec{b}}}}_i \right\| ^2 \right] , \end{aligned}$$

(17)

where the last equality is due to Assumption 3, since

$$\begin{aligned} {\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_i^T {\hat{{\varvec{b}}}}_i \right] = {\mathbb {E}}\left[ {\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_i^T {\hat{{\varvec{b}}}}_i \mid {\mathcal {F}}_j, {\mathcal {G}}_{j-1}\right] \right] = {\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_i^T {\mathbb {E}}\left[ {\hat{{\varvec{b}}}}_i \mid {\mathcal {F}}_j\right] \right] = 0. \end{aligned}$$

By Assumption 1, we have $b_k > 0$ and $c_k \le c_0$. Hence, there exist a constant ${\bar{c}}$ such that we can write (17) as

$$\begin{aligned} \sum _{i=k}^{\infty }a_i^2{\mathbb {E}}\left[ \left\| {\hat{{\varvec{b}}}}_i \right\| ^2 \right]&\le \sum _{i=k}^{\infty }a_i^2 {\mathbb {E}}\left[ \left( L_k^{(+)} - L_k^{(-)}\right) ^2(2\varvec{C}_i\circ {\varvec{\Delta }}_i)^{-T}(2\varvec{C}_i\circ {\varvec{\Delta }}_i)^{-1}\right] \\&\le \sum _{i=k}^{\infty }{\bar{c}}^2 \frac{a_i^2}{c_i^2} {\mathbb {E}}\left[ \left( L_k^{(+)} - L_k^{(-)}\right) ^2{\varvec{\Delta }}_i^{-T}{\varvec{\Delta }}_i^{-1}\right] < \infty , \end{aligned}$$

which further implies that

$$\begin{aligned} \lim _{k\rightarrow \infty }a_i^2{\mathbb {E}}\left[ \left\| {\hat{{\varvec{b}}}}_i \right\| ^2 \right] = 0. \end{aligned}$$

For any $\eta > 0$ and all $k \ge n$, since

$$\begin{aligned} \left\{ \sup _k \left\| \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right\} \subset \left\{ \sup _{m\ge k} \left\| \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right\} , \end{aligned}$$

we can use (17) to get

$$\begin{aligned} P\left( \sup _k \left\| \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right) \le P\left( \sup _{m\ge k} \left\| \sum _{i=k}^{m} a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right) \le \eta ^{-2} \sum _{i=k}^{\infty }a_i^2{\mathbb {E}}\left[ \left\| {\hat{{\varvec{b}}}}_i \right\| ^2 \right] . \end{aligned}$$

As $n \rightarrow \infty$, for all $k \ge n$,

$$\begin{aligned} \lim _{k\rightarrow \infty }\eta ^{-2} \sum _{i=k}^{\infty }a_i^2{\mathbb {E}}\left[ \left\| {\hat{{\varvec{b}}}}_i \right\| ^2 \right] = 0, \end{aligned}$$

and

$$\begin{aligned} \lim _{n\rightarrow \infty } P\left( \sup _{k\ge n} \left\| \sum _{i=k}^{\infty } a_i{\hat{{\varvec{b}}}}_i \right\| \ge \eta \right) = 0. \end{aligned}$$

Therefore, we conclude

$$\begin{aligned} \lim _{k\rightarrow \infty } \sum _{i=k}^{\infty } a_i{\hat{{\varvec{b}}}}_i = \varvec{0}, \end{aligned}$$

and

$$\begin{aligned} \lim _{s\rightarrow \infty } \sum _{i=s}^{\infty } a_{k_i}{\hat{{\varvec{b}}}}_{k_i} = \varvec{0}. \end{aligned}$$

(18)

Similarly, we can also show that

$$\begin{aligned} \lim _{s\rightarrow \infty } \sum _{i=s}^{\infty } a_{k_i}{\hat{{\varvec{e}}}}_{k_i} = \varvec{0}. \end{aligned}$$

(19)

Combining (16) with results in (18) and (19), we have

$$\begin{aligned} \lim _{s\rightarrow \infty } \sum _{i=s}^{n} a_{k_i} {\bar{\varvec{g}}}_{k_i} = \varvec{0}. \end{aligned}$$

(20)

Suppose ${\varvec{\theta }}' \ne {\varvec{\theta }}^*$. Given $\lim _{s\rightarrow \infty } {\hat{{\varvec{\theta }}}}_{k_s} = {\varvec{\theta }}'$, for any $\delta > 0$, there exists a S such that for any $s > S$, $\Vert {\hat{{\varvec{\theta }}}}_{k_s} - {\varvec{\theta }}' \Vert \le \delta$. Let $\delta$ be sufficiently small, we have ${\hat{{\varvec{\theta }}}}_{k_s} \in B_r({\varvec{\theta }}')$. By Assumption 1 and 6, we must have $\sum _{i=s}^\infty a_{k_i} = \infty$ implies

$$\begin{aligned} \lim _{s\rightarrow \infty } \sum _{i=s}^{n} a_{k_i} {\bar{\varvec{g}}}_{k_i}^T({\varvec{\theta }}' - {\varvec{\theta }}^*) = \infty , \end{aligned}$$

which contradicts with (20). Hence, we conclude that ${\varvec{\theta }}' = {\varvec{\theta }}^*$. Since ${\varvec{\theta }}'$ is chosen to be the limiting point of any convergent subsequence, we have all the convergent subsequence converges to the same liming points and consequently ${\hat{{\varvec{\theta }}}}_k \rightarrow {\varvec{\theta }}^*$ a.s. when $k \rightarrow \infty$. $\square$