A Blockwise Consistency Method for Parameter Estimation of Complex Models

Abstract

The drastic improvement in data collection and acquisition technologies has enabled scientists to collect a great amount of data. With the growing dataset size, typically comes a growing complexity of data structures and of complex models to account for the data structures. How to estimate the parameters of complex models has put a great challenge on current statistical methods. This paper proposes a blockwise consistency approach as a potential solution to the problem, which works by iteratively finding consistent estimates for each block of parameters conditional on the current estimates of the parameters in other blocks. The blockwise consistency approach decomposes the high-dimensional parameter estimation problem into a series of lower-dimensional parameter estimation problems, which often have much simpler structures than the original problem and thus can be easily solved. Moreover, under the framework provided by the blockwise consistency approach, a variety of methods, such as Bayesian and frequentist methods, can be jointly used to achieve a consistent estimator for the original high-dimensional complex model. The blockwise consistency approach is illustrated using high-dimensional linear regression with both univariate and multivariate responses. The results of both problems show that the blockwise consistency approach can provide drastic improvements over the existing methods. Extension of the blockwise consistency approach to many other complex models is straightforward.

Appendix

Proof of Theorem 2.1

We follow the proof of Theorem 2.4.3 of van der Vaart and Wellner (1996). By the symmetrization Lemma 2.3.1 of van der Vaart and Wellner (1996), measurability of the class $\mathcal {F}_{n}$, and Fubini’s theorem,

$$\begin{array}{@{}rcl@{}} &&E^{*}\sup\limits_{\theta^{(s)} \in {\Theta}_{n}^{(s)},\hat{\boldsymbol{\theta}}_{t-1}^{(-s)}\in {\Theta}_{n,T}^{(-s)}} \left| \widehat{G}_{n}(\theta^{(s)}|\hat{\boldsymbol{\theta}}_{t-1}^{(-s)}) - G_{n}(\theta^{(s)}|\hat{\boldsymbol{\theta}}_{t-1}^{(-s)}) \right|\\ &\leq& 2 E_{x} E_{\epsilon} \sup\limits_{q\in \mathcal{F}} \left\|\frac{1}{n}\sum\limits_{i = 1}^{n} \epsilon_{i} q(x_{i}) \right\| \leq 2 E_{x}E_{\epsilon} \sup\limits_{q\in \mathcal{G}_{n,M}} \left\|\frac{1}{n}\sum\limits_{i = 1}^{n} \epsilon_{i} q(x_{i}) \right\|\\ &&+ 2 E^{*}[m_{n}(x)1(m_{n}(x)>M)], \\ \end{array} $$

where 𝜖_i are i.i.d. Rademacher random variables with P(𝜖_i = + 1) = P(𝜖_i = − 1) = 1/2, and E^∗ denotes the outer expectation.

By condition (B2)-(a), 2E^∗[m_n(x)1(m_n(x) > M)] → 0 for sufficiently large M. To prove convergence in mean, it suffices to show that the first term converges to zero for fixed M. Fix x₁,...,x_n, and let $\mathcal {H}$ be a 𝜖-net in $L_{1}(\mathbb {P}_{n})$ over $\mathcal {G}_{M}$, then

$$E_{\epsilon} \sup\limits_{q\in \mathcal{G}_{n,M}}\left\|\frac{1}{n}\sum\limits_{i = 1}^{n} \epsilon_{i} q(x_{i}) \right\|\leq E_{\epsilon} \sup\limits_{q\in \mathcal{H}}\left\|\frac{1}{n}\sum\limits_{i = 1}^{n} \epsilon_{i} q(x_{i}) \right\|+\epsilon, $$

where the cardinality of $\mathcal {H}$ can be chosen equal to $N(\epsilon ,\mathcal {G}_{n,M},L_{1}(\mathbb {P}_{n})$. Bound the l₁-norm on the right by the Orlicz-norm ψ₂ and use the maximal inequality (Lemma 2.2.2 of van der Vaart and Wellner (1996)) and Hoeffding’s inequality, it can be shown that

$$ E_{\epsilon} \sup\limits_{q\in \mathcal{G}_{n,M}}\left\|\frac{1}{n}\sum\limits_{i = 1}^{n} \epsilon_{i} q(x)\right\| \leq K \sqrt{1+\log N(\epsilon,\mathcal{G}_{n,M},L_{1}(\mathbb{P}_{n}))}\sqrt{\frac{6}{n}}M +\epsilon \rightarrow_{P^{*}} \epsilon, $$

(A.1)

where K is a constant, and P^∗ denotes outer probability. It has been shown that the left side of Eq. A.1 converges to zero in probability. Since it is bounded by M, its expectation with respect to x₁,…,x_n converges to zero by the dominated convergence theorem.

This concludes the proof that $\sup _{\theta ^{(s)} \in {\Theta }_{n}^{(s)},\hat {\boldsymbol {\theta }}_{t-1}^{(-s)}\in {\Theta }_{n,T}^{(-s)}} \left | \widehat {G}_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1}^{(-s)}) - G_{n}\right .$$\left .(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1}^{(-s)}) \right | \rightarrow _{p} 0$ in mean. Further, by Markov inequality, we conclude that Eq. 2.6 holds.

Proof of Theorem 2.2

Since both $\widehat {G}_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1}^{(-s)})$ and $G_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1}^{(-s)})$ are continuous in 𝜃^(s) as implied by the continuity of log π(x|𝜃) in 𝜃, the remaining part of the proof follows from Lemma A.1.

Lemma A.1.

Consider a sequence of functionsQ_t(𝜃, X_n) fort = 1,2,…,T.Suppose that the following conditions are satisfied: (C1) For eacht,Q_t(𝜃, X_n) is continuous in𝜃and there exists a function$Q_{t}^{*}(\boldsymbol {\theta })$,which is continuous in𝜃and uniquely maximized at$\boldsymbol {\theta }_{*}^{(t)}$.(C2) For any𝜖 > 0,$\sup _{\boldsymbol {\theta } \in {\Theta }_{n}\setminus B_{t}(\epsilon )} Q_{t}^{*}(\boldsymbol {\theta })$exists,where$B_{t}(\epsilon )=\{\boldsymbol {\theta }: \|\boldsymbol {\theta }-\boldsymbol {\theta }_{*}^{(t)}\| < \epsilon \}$;Let$\delta _{t}=Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)})- \sup _{\boldsymbol {\theta } \in {\Theta }_{n}\setminus B_{t}(\epsilon )} Q_{t}^{*}$ (𝜃),δ = min_{t∈{1,2,…,T}}δ_t > 0.(C3)$\sup _{t\in \{1,2,\ldots ,T\}} \sup _{\boldsymbol {\theta } \in {\Theta }_{n}} |Q_{t}(\boldsymbol {\theta }, \boldsymbol {X}_{n})-Q_{t}^{*}(\boldsymbol {\theta })| \to _{p} 0$asn →∞.(C4) The penalty function$P_{\lambda _{n}}(\boldsymbol {\theta })$isnon-negative and converges to 0 uniformly over the set$\{\boldsymbol {\theta }_{*}^{(t)}: t = 1,2,\ldots ,T\}$asn →∞,whereλ_nis a regularization parameter and its value can depend on the sample size n.Let$\boldsymbol {\theta }_{n}^{(t)}=\arg \max _{\boldsymbol {\theta }\in {\Theta }_{n}} \{ Q_{t}(\boldsymbol {\theta }, \boldsymbol {X}_{n})-P_{\lambda _{n}}(\boldsymbol {\theta })\}$.Then the uniform convergence holds, i.e.,$\sup _{t \in \{1,2,\ldots ,T\}} \|\boldsymbol {\theta }_{n}^{(t)}- \boldsymbol {\theta }_{*}^{(t)}\|\to _{p} 0$.

Proof.

Consider two events (i)$\sup _{t \in \{1,2,\ldots ,T\}} \sup _{\boldsymbol {\theta } \in {\Theta }_{n}\setminus B_{t}(\epsilon )} |Q_{t}(\boldsymbol {\theta },\boldsymbol {X}_{n})-Q_{t}^{*}(\boldsymbol {\theta })| < \delta /2$,and (ii) sup_{t∈{1,2,…,T}}$\sup _{\boldsymbol {\theta } \in B_{t}(\epsilon )} |Q_{t}(\boldsymbol {\theta },\boldsymbol {X}_{n})-Q_{t}^{*}(\boldsymbol {\theta })| < \delta /2$.From event (i), we can deduce that for anyt ∈{1,2,…,T} and any𝜃 ∈Θ_n ∖ B_t(𝜖),$Q_{t}(\boldsymbol {\theta }, \boldsymbol {X}_{n}) < Q_{t}^{*}(\boldsymbol {\theta })+\delta /2 \leq Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)}) -\delta _{t} +\delta /2 \leq Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)}) -\delta /2$.Therefore,$Q_{t}(\boldsymbol {\theta }, \boldsymbol {X}_{n}) -P_{\lambda _{n}}(\boldsymbol {\theta }) < Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)}) -\delta /2 -o(1)$bycondition (C4).

From event (ii), we can deduce that for anyt ∈{1,2,…,T} and any𝜃 ∈ B_t(𝜖),$Q_{t}(\boldsymbol {\theta }, \boldsymbol {X}_{n})> Q_{t}^{*}(\boldsymbol {\theta }) -\delta /2$and$Q_{t}(\boldsymbol {\theta }_{*}^{(t)}, \boldsymbol {X}_{n})> Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)}) -\delta /2$.Therefore,$Q_{t}(\boldsymbol {\theta }_{*}^{(t)}, \boldsymbol {X}_{n})-P_{\lambda _{n}}(\boldsymbol {\theta }_{*}^{(t)}) > Q_{t}^{*}(\boldsymbol {\theta }_{*}^{(t)}) -\delta /2- o(1)$bycondition (C4).

If both events hold simultaneously, then we musthave${\boldsymbol {\theta }}_{n}^{(t)} \in B_{t}(\epsilon )$forallt ∈{1,2,…,T} asn →∞.By condition (C3), the probability that both events hold tends to 1.Therefore,

$$P(\boldsymbol{\theta}_{n}^{(t)} \in B_{t}(\epsilon) \text{ for all } t = 1,2,\ldots,T) \to 1, $$

which concludes the lemma. □

Proof of Theorem 2.3

Applying Taylor expansion to $G_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)})$ at $\theta _{t*}^{(s)}$, we get $G_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)}) - G_{n}(\theta _{t*}^{(s)}|\hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)}) =O_{p}(1/n)$, following from the condition (B5) and the condition (B3) that $G_{n}(\theta ^{(s)}|\hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)})$ is maximized at $\boldsymbol {\theta }_{t*}^{(s)}$. Therefore,

$$\begin{array}{@{}rcl@{}} n[\widehat{G}_{n}(\theta_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})] &=&Z_{t,1}+\cdots+Z_{t,n}+n[G_{n}(\theta_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})\\&&- G_{n}(\boldsymbol{\theta}_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})]= Z_{t,1}+\cdots+Z_{t,n}+ \epsilon_{n}, \end{array} $$

where 𝜖_n = O_p(1), and

$$ P(n|\widehat{G}_{n}(\boldsymbol{\theta}_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})|>nz) \leq\! P(|Z_{t,1}+\cdots+Z_{t,n}|>nz-|\epsilon_{n}|). $$

(A.2)

By Bernstein’s inequality,

$$ P(|Z_{t,1}+\cdots+Z_{t,n}|>nz-|\epsilon_{n}|) \leq 2 \exp\left\{-\frac{1}{2} \frac{(z-|\epsilon_{n}|/n)^{2}}{ \tilde{v}^{\prime}+\tilde{M}_{b}^{\prime}(z-|\epsilon_{n}|/n)} \right\}, $$

(A.3)

for $\tilde {v}^{\prime } \geq (\tilde {v}_{1}+\cdots +\tilde {v}_{n})/n^{2}$ and $\tilde {M}_{b}^{\prime }=\tilde {M}_{b}/n$. Applying Taylor expansion to the right of Eq. A.3 at z and combining with Eq. A.2 leads to

$$ P(|\widehat{G}_{n}(\theta_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})| >z) \leq K \exp\left\{-\frac{1}{2} \frac{z^{2}}{\tilde{v}^{\prime}+\tilde{M}_{b}^{\prime} z} \right\}, $$

(A.4)

where $K = 2+\frac {3}{\tilde {M}_{b}^{\prime }}O_{p}(1/n)= 2+\frac {3}{\tilde {M}_{b}}O_{p}(1)$, since the derivative $|d[z^{2}/(\tilde {v}^{\prime }+\tilde {M}_{b}^{\prime } z)]/dz| \leq 3/\tilde {M}_{b}^{\prime }$.

By applying Lemma 2.2.10 of van der Vaart and Wellner (1996), for Orlicz norm ψ₁, we have

$$\begin{array}{@{}rcl@{}} &&\left\| \sup\limits_{\theta_{t,g}^{(s)} \in {\Theta}_{n}^{(s)}, t = 1,2,\ldots,T} \left |\widehat{G}_{n}(\theta_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right| \right\|_{\psi_{1}}\\ &&\leq \epsilon + K^{\prime}\left( \tilde{M}_{b}^{\prime} \log (1+TN(\epsilon,\mathcal{G}_{n,M},L_{1}(\mathbb{P}_{n}))) +\sqrt{\tilde{v}^{\prime}}\sqrt{\log(1+TN(\epsilon,\mathcal{G}_{n,M},L_{1}(\mathbb{P}_{n})))} \right),\\ \end{array} $$

(A.5)

for a constant K^′ and any 𝜖 > 0. Since $\tilde {v}^{\prime }=O(1/n)$, $\tilde {M}_{b}^{\prime }=O(1/n)$, log(T) = o(n), and $\log N(\epsilon ,\mathcal {G}_{n,M},L_{1}(\mathbb {P}_{n}))=o(n)$, we have

$$\left\| \sup\limits_{\theta_{t,g}^{(s)} \in {\Theta}_{n}^{(s)}, t = 1,2,\ldots,T} \left|\widehat{G}_{n}(\theta_{t,g}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)}|\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right| \right\|_{\psi_{1}} \rightarrow_{P^{*}} \epsilon. $$

Therefore,

$$ \sup\limits_{\theta_{t,g}^{(s)} \in {\Theta}_{n}^{(s)}, t \in \{1,2,\ldots,T\}} \left|\widehat{G}_{n}(\theta_{t,g}^{(s)}| \hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})- G_{n}(\theta_{t*}^{(s)} |\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right| \to_{p} 0. $$

(A.6)

Note that, as implied by the proof of Lemma 2.2.10 of van der Vaart and Wellner (1996), Eq. A.5 holds for a general constant K in Eq. A.4. Then, by condition (B3), we must have the uniform convergence that $\theta _{t,g}^{(s)} \in B_{t}(\epsilon )$ for all t as n →∞, where B_t(𝜖) is as defined in (B3). This statement can be proved by contradiction as follows:

Assume $\theta _{t,g}^{(s)} \notin B_{i}(\epsilon )$ for some i ∈{1,2,…,T}. By the uniform convergence established in Theorem 2.1, $\left |\widehat {G}_{n}(\theta _{t,g}^{(s)}| \hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)})- G_{n}(\theta _{t,g}^{(s)} |\hat {\boldsymbol {\theta }}_{t-1,g}^{(-s)}) \right | =o_{p}(1)$. Further, by condition (B3) and the assumption $\theta _{t,g}^{(s)} \notin B_{i}(\epsilon )$,

$$\begin{array}{@{}rcl@{}} \left| \widehat{G}_{n}(\theta_{t,g}^{(s)}| \hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)})- G_{n}(\theta_{t*}^{(s)} |\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right| & \geq& \left|G_{n}(\theta_{t,g}^{(s)} |\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t*}^{(s)} |\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right|\\ &&- \left|\widehat{G}_{n}(\theta_{t,g}^{(s)}| \hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) - G_{n}(\theta_{t,g}^{(s)} |\hat{\boldsymbol{\theta}}_{t-1,g}^{(-s)}) \right| \\ & \geq& \delta-o_{p}(1), \end{array} $$

which contradicts with the uniform convergence established in Eq. A.6. This concludes the proof.

Proof of Theorem 2.4

Define $d_{t}^{(n)}:=\|\hat {\boldsymbol {\theta }}_{t}-\widetilde {\boldsymbol {\theta }}_{t}\|$, where n indicates the implicit dependence of $\hat {\boldsymbol {\theta }}_{t}$ on n. Then

$$ d_{t}^{(n)}:=\|\hat{\boldsymbol{\theta}}_{t}-\widetilde{\boldsymbol{\theta}}_{t}\| \leq \|\hat{\boldsymbol{\theta}}_{t}-M_{s}(\hat{\boldsymbol{\theta}}_{t-1})\| + \|M_{s}(\hat{\boldsymbol{\theta}}_{t-1})-\widetilde{\boldsymbol{\theta}}_{t}\|. $$

(A.7)

For the first component of the inequality (A.7), we define

$$g_{n}:=\sup\limits_{t \in \Bbb{N},\hat{\boldsymbol{\theta}}_{t-1} \in {\Theta}_{n}} \|\hat{\boldsymbol{\theta}}_{t}-M_{s}(\hat{\boldsymbol{\theta}}_{t-1})\|, $$

which converges to zero in probability as n →∞, following from Theorems 2.2 and 2.3 for both types of consistent estimation procedures considered in the paper. For the second component of the inequality (A.7), we have

$$\|M_{s}(\hat{\boldsymbol{\theta}}_{t-1})-\widetilde{\boldsymbol{\theta}}_{t}\|=\|M_{s}(\hat{\boldsymbol{\theta}}_{t-1})-M_{s}(\widetilde{\boldsymbol{\theta}}_{t-1})\| \leq \rho^{*} \|\hat{\boldsymbol{\theta}}_{t-1}-\widetilde{\boldsymbol{\theta}}_{t-1} \|=\rho^{*} d_{t-1}^{(n)}, $$

following from condition (B6).

Combining with the fact that d₀ = 0, i.e., the two paths $\{\hat \theta _{t}\}$ and $\{\widetilde \theta _{t}\}$ started from the same point, we have

$$ d_{t}^{(n)}\leq \sum\limits_{l = 0}^{t-1} g_{n} (\rho^{*})^{l} \leq \frac{g_{n}}{1-\rho^{*}} \stackrel{p}{\to} 0, $$

(A.8)

where the convergence is uniform over t as $g_{n} \stackrel {p}{\to } 0$. Moreover, since $\widetilde {\boldsymbol {\theta }}_{t}$ converges to a coordinatewise maximum point of $E_{\boldsymbol {\theta }_{*}} \log \pi (X|\boldsymbol {\theta })$ under conditions (A1) and (A2), $\hat {\boldsymbol {\theta }}_{t}$ will converge to the same point in probability. That is, $\hat {\boldsymbol {\theta }}_{\infty }:=\lim _{t\rightarrow \infty }\hat {\boldsymbol {\theta }}_{t} \stackrel {p}{\to } \widetilde {\boldsymbol {\theta }}_{\infty }: =\lim _{t\rightarrow \infty } \widetilde {\boldsymbol {\theta }}_{t}$.