Robust estimation for nonrandomly distributed data

Abstract

In recent years, many methodologies for distributed data have been developed. However, there are two problems. First, most of these methods require the data to be randomly and uniformly distributed across different machines. Second, the methods are mainly not robust. To solve these problems, we propose a distributed pilot modal regression estimator, which achieves robustness and can adapt when the data are stored nonrandomly. First, we collect a random pilot sample from different machines; then, we approximate the global MR objective function by a communication-efficient surrogate that can be efficiently evaluated by the pilot sample and the local gradients. The final estimator is obtained by minimizing the surrogate function in the master machine, while the other machines only need to calculate their gradients. Theoretical results show the new estimator is asymptotically efficient as the global MR estimator. Simulation studies illustrate the utility of the proposed approach.

Appendix

Proof of Theorem 1

First, we compute the order of $\Vert \nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0})-\nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})\Vert _{\infty }$. Let ${\varvec{\Sigma }}=E({\varvec{X}}{\varvec{X}}^{T})$, we have that

$$\begin{aligned}&~~~~\Vert \nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0})-\nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})\Vert _{\infty } \\&=O_{p}\left( \left\| {\varvec{\Sigma }}-\frac{1}{N}\widetilde{{\varvec{X}}} \widetilde{{\varvec{X}}}^{T} \right\| _{\infty }\right) +O_{p}\left( \left\| \frac{1}{n}\widetilde{{\varvec{X}}}_{{\mathcal {P}}}\widetilde{{\varvec{X}}}_{{\mathcal {P}}}^{T}-{\varvec{\Sigma }} \right\| _{\infty }\right) +O_{p}(N^{-1/2}), \end{aligned}$$

where $\widetilde{{\varvec{X}}}=({\varvec{X}}_{1},\dots ,{\varvec{X}}_{N})^{\text {T}}$ and $\widetilde{{\varvec{X}}}_{{\mathcal {P}}}=({\varvec{X}}_{i},i\in {{\mathcal {P}}})^{\text {T}}$. It is easy to obtain

$$\begin{aligned} P\left( \left| \frac{1}{N}\sum \nolimits _{i=1}^{N}X_{ij}X_{ik}-{\varvec{\Sigma }}_{jk}\right| >t\right) \leqslant \exp (-c_{1}\min (t^{2},t)N), \end{aligned}$$

where $c_{1}$ is a constant that depends on ${\varvec{\Sigma }}$. By a union bound over all (j, k) pairs,

$$\begin{aligned} P\left( \left| \frac{1}{N}\sum \nolimits _{i=1}^{N}\widetilde{{\varvec{X}}}\widetilde{{\varvec{X}}}^{T}-{\varvec{\Sigma }}\right| >t\right) \leqslant \exp (2\log p-c_{1}\min (t^{2},t)N). \end{aligned}$$

Thus, letting $t=C\sqrt{\frac{\log p}{N}}$, we have $O_{p}(\Vert {\varvec{\Sigma }}-\frac{1}{N}\widetilde{{\varvec{X}}}\widetilde{{\varvec{X}}}^{T}\Vert _{\infty })=O_{p}(N^{-1/2})$. By a similar argument, $O_{p}(\Vert \frac{1}{n}\widetilde{{\varvec{X}}}_{{\mathcal {P}}}\widetilde{{\varvec{X}}}_{{\mathcal {P}}}^{T}-{\varvec{\Sigma }}\Vert _{\infty })=O_{p}(n^{-1/2})$. Then, we can get that

$$\begin{aligned} \left\| \nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0})-\nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})\right\| _{\infty }=O_{p}\left( n^{-1/2}\right) . \end{aligned}$$

By applying Lemma 6 in Zhang et al. (2013) with $F_{1}=\widetilde{{L}}_\text {N}^h({\varvec{\beta }})$ in the notation therein, we can also obtain

$$\begin{aligned} \left\| \widetilde{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_\text {N} \right\| =O_{p}\left( \left\| \nabla \widetilde{{L}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})\right\| \right) . \end{aligned}$$

A simple calculation yields

$$\begin{aligned} \nabla \widetilde{{L}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})=\nabla {{Q}}_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_\text {N})- \nabla {{Q}}_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}})+\nabla {{Q}}_\text {N}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}}), \end{aligned}$$

and note that $\nabla {{Q}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})={\varvec{0}}$, we obtain

$$\begin{aligned} \nabla \widetilde{{L}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})= \left( \nabla {{Q}}_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_\text {N})-\nabla {{Q}}_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}})\right) - \left( \nabla {{Q}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})-\nabla {{Q}}_\text {N}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}})\right) . \end{aligned}$$

By the integral form of Taylor’s expansion, we have

$$\begin{aligned} \nabla Q_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_\text {N})-\nabla Q_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}})={\varvec{H}}_{{\mathcal {P}}}(\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}) \text{ and } \nabla Q_{N}^h(\hat{{\varvec{\beta }}}_\text {N})-\nabla Q_{N}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}})={\varvec{H}}_{N}(\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}), \end{aligned}$$

where ${\varvec{H}}_{{\mathcal {P}}}=\int _{0}^{1}\nabla ^{2}Q_{{\mathcal {P}}}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}}+t(\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}))dt$ and ${\varvec{H}}_{N}=\int _{0}^{1}\nabla ^{2}Q_{N}^h(\hat{{\varvec{\beta }}}_{{\mathcal {P}}}+t(\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}))dt$ satisfy $\Vert {\varvec{H}}_{{\mathcal {P}}}-\nabla ^{2}Q_{{\mathcal {P}}}^h({{\varvec{\beta }}}_0)\Vert =O_{p}(\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert +\Vert \hat{{\varvec{\beta }}}_\text {N}-{{\varvec{\beta }}}_{0}\Vert )$ and $\Vert {\varvec{H}}_{N}-\nabla ^{2}Q_{N}^h({{\varvec{\beta }}}_0)\Vert =O_{p}(\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert +\Vert \hat{{\varvec{\beta }}}_\text {N}-{{\varvec{\beta }}}_{0}\Vert )$, respectively. Thus, we have

$$\begin{aligned}&~~~~\left\| \nabla \widetilde{{L}}_\text {N}^h(\hat{{\varvec{\beta }}}_\text {N})\right\| \\&=\left\| ({\varvec{H}}_{{\mathcal {P}}}-\nabla ^{2}{{Q}}_{{\mathcal {P}}}^h({{\varvec{\beta }}}_0)) (\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}})-({\varvec{H}}_{N}-\nabla ^{2}{{Q}}_\text {N}^h({{\varvec{\beta }}}_0)) (\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}})\right. \\&~~~~~~\left. +(\nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})-\nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0}))(\hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}) \right\| \\&\leqslant \Vert {\varvec{H}}_{{\mathcal {P}}}-\nabla ^{2}{{Q}}_{{\mathcal {P}}}^h({{\varvec{\beta }}}_0)\Vert \Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert +\Vert {\varvec{H}}_{N}-\nabla ^{2}{{Q}}_\text {N}^h({{\varvec{\beta }}}_0)\Vert \Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert \\&~~~~~~+\Vert \nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})-\nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0})\Vert \Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert \\&=(O_{p}(\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert )+O_{p}(\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert )+\Vert \nabla ^2{Q}_{{\mathcal {P}}}^h({\varvec{\beta }}_{0})- \nabla ^2{Q}_\text {N}^h({\varvec{\beta }}_{0})\Vert )\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert \\&=O_{p}\left( \frac{1}{\sqrt{n}}\right) \Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert . \end{aligned}$$

Now, we complete the proof of (a).

Together with

$$\begin{aligned} \hat{{\varvec{\beta }}}_\text {N}-{\varvec{\beta }}_{0}=-\frac{1}{E(\ddot{\phi }_{h}(\epsilon ))}{\varvec{\Sigma }}^{-1}\nabla Q_\text {N}^h({{\varvec{\beta }}}_{0})+O_{p}\left( \frac{1}{N}\right) , \end{aligned}$$

we can get that

$$\begin{aligned}&~~~~\widetilde{{{\varvec{\beta }}}}_\text {N}-{\varvec{\beta }}_{0}\\&=(\widetilde{{{\varvec{\beta }}}}_\text {N}-\hat{{{\varvec{\beta }}}}_\text {N})+( \hat{{{\varvec{\beta }}}}_\text {N}-{\varvec{\beta }}_{0})\\&=-\frac{1}{E(\ddot{\phi }_{h}(\epsilon ))}{\varvec{\Sigma }}^{-1}\left( \frac{1}{N}\sum \nolimits _{i=1}^{N}{\varvec{X}}_{i}{\dot{\phi }}_{h}\left( \epsilon _{i}\right) \right) + O_{p}\left( \frac{1}{N}+n^{-1/2}\Vert \hat{{\varvec{\beta }}}_{{\mathcal {P}}}-\hat{{\varvec{\beta }}}_\text {N}\Vert \right) . \end{aligned}$$

Under the assumptions $n/\sqrt{N}\rightarrow \infty$ and $\Vert \hat{{\varvec{\beta }}}_\text {N}-\hat{{\varvec{\beta }}}_{{\mathcal {P}}}\Vert =O_{p}(n^{-1/2})$, we can obtain that $\sqrt{N}(\frac{1}{N}+n^{-1/2}\Vert \hat{{\varvec{\beta }}}_{{\mathcal {P}}}-\hat{{\varvec{\beta }}}_\text {N}\Vert )=o_{p}(1)$. Thus, we have that

$$\begin{aligned}&~~~~\sqrt{N}(\widetilde{{{\varvec{\beta }}}}_\text {N}-{\varvec{\beta }}_{0})\\&=-\frac{1}{E(\ddot{\phi }_{h}(\epsilon ))}{\varvec{\Sigma }}^{-1}\left( \frac{1}{\sqrt{N}} \sum \nolimits _{i=1}^{N}{\varvec{X}}_{i}{\dot{\phi }}_{h}\left( \epsilon _{i}\right) \right) +o_{p}(1)\\&\rightarrow _d N({\varvec{0}}, \xi (h){\varvec{\Sigma }}^{-1}), \end{aligned}$$

where $\xi (h)=\frac{E({\dot{\phi }}_{h}^2(\epsilon ))}{[E(\ddot{\phi }_{h}(\epsilon ))]^2}$. The proof of (b) is completed. $\square$