On the use of repeated measurement errors in linear regression models

Abstract

In a linear mean regression setting with repeated measurement errors, we develop asymptotic properties of a naive estimator to better clarify the effects of these errors. We then construct a group of unbiased estimating equations with independent repetitions and make use of these equations in two ways to obtain two estimators: a weighted averaging estimator and an estimator based on the generalized method of moments. The proposed estimators do not require any additional information about the measurement errors. We also prove the consistency and asymptotic normality of the two estimators. Our theoretical results are verified by simulation studies and a real data analysis.

Appendix

A.1 Proof of Lemma 1

Let $B\triangleq X^\mathrm {T}X$, $D\triangleq X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X+\Delta ^\mathrm {T}\Delta$, we have

$$\begin{aligned} (W^\mathrm {T}W)^{-1}=&(X^\mathrm {T}X+X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X+\Delta ^\mathrm {T}\Delta )^{-1}\triangleq (B+D)^{-1}\\ =&B^{-1}-B^{-1}DB^{-1}+B^{-1}DB^{-1}DB^{-1}-\ldots \\ =&(X^\mathrm {T}X)^{-1}-(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}\\&+(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}\Delta (X^\mathrm {T}X)^{-1}+\ldots \end{aligned}$$

Moreover, for any matrix $B_n$, $D_n$, if $B_n=O_p(a_n)$ and $D_n=O_p(b_n)$, then $B_nD_n=O_p(a_nb_n)$ and $B_n+D_n=O_p(a_n\vee b_n)$.

Thus the results follow immediately.

A.2 Proof of Lemma 2

According to the second order expansion in Lemma 1, we have

$$\begin{aligned} \hat{\beta }_\mathrm {naive}&=(W^\mathrm {T}W)^{-1}W^\mathrm {T}Y\\&=(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y+(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}Y\\&\quad -(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y\\&\quad -(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}\Delta (X^\mathrm {T}X)^{-1}X^\mathrm {T}Y\\&\quad +(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y\\&\quad -(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}Y+o_p(1). \end{aligned}$$

Thus we get the expectation of $\hat{\beta }_\mathrm {naive}$ referring to proposition 1.

A.3 Proof of Lemma 3

According to the first order expansion in Lemma 1, we have

$$\begin{aligned} \hat{\beta }_\mathrm {naive}= & {} (X^\mathrm {T}X)^{-1}X^\mathrm {T}Y+(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}Y\\&\quad -(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y+o_p(1).\end{aligned}$$

Let $A_0\triangleq {{\,\mathrm{E}\,}}YY^\mathrm {T}\equiv X\beta _0\beta _0^\mathrm {T}X+\sigma _1^2I_n$, we have

$$\begin{aligned} {{\,\mathrm{Var}\,}}(\hat{\beta }_\mathrm {naive})&={{\,\mathrm{E}\,}}(\hat{\beta }_\mathrm {naive}\hat{\beta }_\mathrm {naive}^\mathrm {T})-\beta _0\beta _0^\mathrm {T}\\&=(X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0X(X^\mathrm {T}X)^{-1}-\beta _0\beta _0^\mathrm {T}\\&\quad +\frac{\sigma _2^2}{s}{{\,\mathrm{tr}\,}}(A_0)(X^\mathrm {T}X)^{-2}-\frac{\sigma _2^2}{s}(X^\mathrm {T}X)^{-2}X^\mathrm {T}A_0^\mathrm {T}X(X^\mathrm {T}X)^{-1}\\&\quad -\frac{\sigma _2^2}{s}{{\,\mathrm{tr}\,}}(A_0X(X^\mathrm {T}X)^{-1}X^\mathrm {T})(X^\mathrm {T}X)^{-2}-\frac{\sigma _2^2}{s}(X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0^\mathrm {T}X(X^\mathrm {T}X)^{-2}\\&\quad +\frac{\sigma _2^2}{s}{{\,\mathrm{tr}\,}}((X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0X(X^\mathrm {T}X)^{-1})(X^\mathrm {T}X)^{-1}+\frac{\sigma _2^2}{s}(X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0^\mathrm {T}X(X^\mathrm {T}X)^{-2}\\&\quad -\frac{\sigma _2^2}{s}{{\,\mathrm{tr}\,}}(X(X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0)(X^\mathrm {T}X)^{-2}+\frac{\sigma _2^2}{s}(X^\mathrm {T}X)^{-2}X^\mathrm {T}A_0^\mathrm {T}X(X^\mathrm {T}X)^{-1}\\&\quad +\frac{\sigma _2^2}{s}{{\,\mathrm{tr}\,}}(X(X^\mathrm {T}X)^{-1}X^\mathrm {T}A_0X(X^\mathrm {T}X)^{-1}X^\mathrm {T})(X^\mathrm {T}X)^{-2}\\&=(X^\mathrm {T}X)^{-1}\sigma _1^2+{{\,\mathrm{tr}\,}}(\beta _0\beta _0^\mathrm {T}+\sigma _1^2(X^\mathrm {T}X)^{-1})\frac{\sigma _1^2}{s}(X^\mathrm {T}X)^{-1}\\&\quad +(n-p)\frac{\sigma _2^2}{s}\sigma _1^2(X^\mathrm {T}X)^{-2}. \end{aligned}$$

A.4 Proof of Theorem 1

From Lemma 3, we get the expansion of $\hat{\beta }_\mathrm {naive}$ as follows:

$$\begin{aligned} \hat{\beta }_\mathrm {naive}&=(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y+(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}Y\\&\quad -(X^\mathrm {T}X)^{-1}(X^\mathrm {T}\Delta +\Delta ^\mathrm {T}X)(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y+o_p(1).\end{aligned}$$

Apparently, we have the asymptotic normality of $(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y$.

Moreover, because of the element-wise independence of $\Delta$, we have $(\Delta ^\mathrm {T}Y)_i=\sum _{j=1}^{n}\Delta _{ji}y_j \overset{D}{\rightarrow } \mathrm {Normality}$ by central limit theorem and ${{\,\mathrm{Cov}\,}}((\Delta ^\mathrm {T}Y)_i,(\Delta ^\mathrm {T}Y)_k)=0$ by the calculation.

Similarly, we get the asymptotic normality of $(X^\mathrm {T}X)^{-1}X^\mathrm {T}\Delta (X^\mathrm {T}X)^{-1}X^\mathrm {T}Y$,

$(X^\mathrm {T}X)^{-1}\Delta ^\mathrm {T}X(X^\mathrm {T}X)^{-1}X^\mathrm {T}Y$ and thus $\sqrt{n}\hat{\beta }_\mathrm {naive}\overset{D}{\rightarrow } \mathrm {Normality}$. Therefore the result follows by lemma 2 and lemma 3.

A.5 Proof of Theorem 2

If our model satisfies the assumptions A*1-A*4 in Chen et al. (2016), then we can prove it by Theorem 1 (ii) in that paper. Using Taylor’s expansion, we have

$$\begin{aligned} G_{nl}(\beta )=G_{nl}(\beta _0)-\frac{1}{n}{W^j}^\mathrm {T}W^k(\beta -\beta _0).\end{aligned}$$

• 1. (A*1): $\sup \limits _{n\ge 1}n^{\frac{1}{4}}\sum _{l=\tau +1}^{\tau (n)}\Vert W_{nl}\Vert \rightarrow 0$ satisfies because of finite $\tau =C_s^2$.

• 2. (A*2): Let $\gamma _l\equiv \Vert \frac{1}{n}\sum _{i=1}^n x_ix_i^\mathrm {T}\Vert$, $\epsilon _n\rightarrow 0$ satisfying $\gamma _l\ge \epsilon _n>n^{\frac{-1}{4}}$, then there exists $\delta >0$ such that if $\Vert \beta _0-\beta \Vert <\delta$, we have

  $$\begin{aligned} \Vert G_l(\beta )\Vert \equiv \Vert \frac{1}{n}\sum _{i=1}^nx_ix_i^\mathrm {T}\Vert \Vert \beta _0-\beta \Vert \ge \gamma _l\Vert \beta _0-\beta \Vert \end{aligned}$$

  and $\min \limits _{l\in \mathcal {J}_n^*}\gamma _l \ge \epsilon _n>0$.

• 3. (A*3): For any $\delta _n=o(1)$ and $n\ge 1$, we have

  $$\begin{aligned} G_{nl}(\hat{\beta }_l)= & {} \frac{1}{n}{W^j}^\mathrm {T}Y-\frac{1}{n}{W^j}^\mathrm {T}Y=O,\\ \inf \limits _{\Vert \beta -\beta _0\Vert \le \delta _n}\Vert G_{nl}(\beta )\Vert\le & {} \inf \limits _{\Vert \beta -\beta _0\Vert \le \delta _n}(\Vert G_{nl}(\beta )-G_{nl}(\beta _0)\Vert +\Vert G_{nl}(\beta _0)\Vert ).\end{aligned}$$

  Moreover, we have

  $$\begin{aligned} \Vert G_{nl}(\beta )-G_{nl}(\beta _0)\Vert&=\Vert \frac{1}{n}{W^j}^\mathrm {T}W^k\Vert \Vert \beta -\beta _0\Vert \\&\le \Vert \frac{1}{n}{W^j}^\mathrm {T}W^k\Vert \delta _n\\&=o_p\left( \frac{1}{\sqrt{n}}\right) , \end{aligned}$$
  $$\begin{aligned} \Vert G_{nl}(\beta _0)\Vert =o_p\left( \frac{1}{\sqrt{n}}\right) ~~~(\mathrm {because~of~} {{\,\mathrm{E}\,}}G_{nl}(\beta _0)=0).\end{aligned}$$

  Thus, $G_{nl}(\hat{\beta }_l)-\inf \limits _{\Vert \beta -\beta _0\Vert \le \delta _n}\Vert G_{nl}(\beta )\Vert =o_p(\epsilon _nn^{-1/4})$ and $\max \limits _{l\in \mathcal {J}_n^*}(\Vert G_{nl}(\hat{\beta }_l)\Vert -\inf \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert G_{nl}(\beta )\Vert )=o_p(\epsilon _nn^{-1/4})$ satisfy.

• 4. (A*4): For any $\delta _n=o(1)$ and $n\ge 1$, we have

  $$\begin{aligned} \Vert G_{nl}(\beta )-G_l(\beta )\Vert&\ \le \Vert \frac{1}{n}\sum _{i=1}^n w_i^jy_i-{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^jy_i\right) \Vert +\Vert \frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}\beta \\&\quad -{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^j{w_i^k}^\mathrm {T}\beta \right) \Vert \\&=O_p\left( \frac{1}{\sqrt{n}}\right) \\&=o_p(\epsilon _n n^{-1/4}). \end{aligned}$$

  Then, $\max \limits _{l\in \mathcal {J}_n^*} \sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert G_{nl}(\beta )-\widetilde{G}_l(\beta )\Vert =o_p(\epsilon _n n^{-1/4})$ satisfies.

A.6 Proof of Theorem 3

Let’s divide the proof into two parts, one for the proof of asymptotic normality and the other for finding the optimal weighting matrix and asymptotic variance.

First part: if we could justify the assumptions A*1 and B1-B7 in Chen et al. (2016), then from Section 5 and Theorem 2 in that paper, we have

$$\begin{aligned} \sqrt{n}(\hat{\beta }_\mathrm {weight}-\beta _0)\overset{D}{\rightarrow } N(0,\Sigma )\end{aligned}$$

and

$$\begin{aligned} \sqrt{n}(\hat{\beta }_l-\beta _0)\overset{D}{\rightarrow } N(0,V_{l})\end{aligned}$$

where $\hat{\beta }_\mathrm {weight}=\sum _{l\in \mathcal {J}_n^*}W_{nl}\hat{\beta }_l$ and $\Sigma =\lim _{n\rightarrow \infty }\sum _{l=1}^\tau \sum _{m=1}^\tau W_{nl}V_{lm}W_{nm}^\mathrm {T}$. Thus the asymptotic normality of $\hat{\beta }_\mathrm {ma}$ satisfy obviously. Now let’s justify these assumptions:

• 1. (A*1): It has been proved in Theorem 2.

• 2. (B1): B1 follows from the proof of (A*3) in Theorem 2.

• 3. (B2): We already have $G_l(\beta )-\Gamma _l(\beta -\beta _0)=\frac{1}{n}\sum _{i=1}^nx_ix_i^\mathrm {T}(\beta -\beta _0)-\frac{1}{n}\sum _{i=1}^nx_ix_i^\mathrm {T}(\beta -\beta _0)=0$. Then, there exists a finite constant C such that for any $\beta$ within a neighborhood of $\beta _0$:

  $$\begin{aligned}\max \limits _{l\in \mathcal {J}_n^*} \Vert G_l(\beta )-\Gamma _l(\beta -\beta _0) \Vert \le C \Vert \beta -\beta _0 \Vert ^2\end{aligned}$$

  and where $\Gamma _l=\frac{-1}{n}X^\mathrm {T}X$ is full (column) rank.

• 4. (B3): (a) From iid samples and central limit theorem, we have:

  $$\begin{aligned}\Vert \sqrt{n}G_{nl}(\beta _0)\Vert =\Vert \sqrt{n}\left( \frac{1}{n}\sum _{i=1}^n w_i^jy_i-\frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}\beta \right) \Vert =O_p(1).\end{aligned}$$

          (b) $\forall \delta _n=o(n^{-1/4}),$

  $$\begin{aligned}&\sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert G_{nl}(\beta )-G_l(\beta )-G_{nl}(\beta _0)+G_l(\beta _0)\Vert \\&\quad =\sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert G_{nl}(\beta )-G_l(\beta )-G_{nl}(\beta _0)\Vert \\&\quad =\sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert -\frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}(\beta -\beta _0)-{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^jy_i\right) +{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^j{w_i^k}^\mathrm {T}\beta \right) \Vert \\&\quad =\sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert -\frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}(\beta -\beta _0)-{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^j{w_i^k}^\mathrm {T}\right) (\beta -\beta _0)\Vert \\&\quad =\sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert -\frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}-{{\,\mathrm{E}\,}}\left( \frac{1}{n}\sum _{i=1}^nw_i^j{w_i^k}^\mathrm {T}\right) \Vert \Vert (\beta -\beta _0)\Vert \\&\quad \le \Vert -\frac{1}{n}\sum _{i=1}^n w_i^j{w_i^k}^\mathrm {T}-{{\,\mathrm{E}\,}}(w_i^j{w_i^k}^\mathrm {T})\Vert \delta _n\\&\quad =o_p\left( \frac{1}{\sqrt{n}}\right) . \end{aligned}$$

  Thus,

  $$\begin{aligned} \max \limits _{l\in \mathcal {J}_n^*} \sup \limits _{\Vert \beta -\beta _0\Vert \le \delta _n} \Vert G_{nl}(\beta )-G_l(\beta )-G_{nl}(\beta _0)+G_l(\beta _0)\Vert =o_p(1/\sqrt{n}).\end{aligned}$$

• 5. (B4): We can see that the optimal weighting matrix is nonrandom. So, we can restrict $W_{nl}$ to nonrandom matrices. Then, let $W_{nl}^0=W_{nl}$, we have:

  (a) $\sum _{l\in \mathcal {J}_n^*}\Vert (W_{nl}-W_{nl}^0)\Gamma _l^{-1}\Vert =o_p(1),$

  (b) $\limsup _n\sum _{l\in \mathcal {J}_n^*}\Vert W_{nl}^0\Gamma _l^{-1}\Vert <\infty .$

• 6. (B5): (a)V is obviously positive definite and finite and so is $V_0$.

  (b) Let $f_n(w_i,y_i)=n^{-1/2}\sum _{l\in \mathcal {J}_n^*}c^\mathrm {T}W_{nl}^0\Gamma _l^{-1}g_l(\theta _0,w_i,y_i)$, we have $n{{\,\mathrm{E}\,}}|f_n(w_i,y_i)|^{2+\kappa }\rightarrow 0$ for $\forall c\in R^p$ and some $\kappa >0$ since the existence of fourth moments of $g_l(\theta _0,w_i,y_i)$ and the positive definite matrix $\Gamma _l=\frac{-1}{n}(X^\mathrm {T}X)^{-1}$.

• 7. (B6): $\beta _0$ is naturally in the interior of $\Theta$.

• 8. (B7): $\max \limits _{l\in \mathcal {J}_n^*}\Vert \hat{\beta }_l-\beta _0\Vert =o_p(n^{-1/4})$ satisfies.

Second part: $\hat{\beta }_\mathrm {weight}$ is a special case of classical minimum distance estimator, so we can get the best estimator by $\Xi =V^{-1}$ which justifies the optimal weighting matrices provided above by $W_{nl}^\mathrm {opt}=[R^\mathrm {T}\Xi R]^{-1}[R^\mathrm {T}\Xi ]_{l.}$.

A.7 Proof of Theorem 4

• 1. ${{\,\mathrm{E}\,}}(\varphi _i(\beta ))=i_\tau \otimes x_ix_i^\mathrm {T}(\beta _0-\beta )$. If $\beta \ne \beta _0$ and $W_\mathrm {gmm}{{\,\mathrm{E}\,}}(\varphi _i(\beta ))=\mathbf {0}$, we have $|W_\mathrm {gmm}|=|\bar{\Omega }|=0$. But $|\bar{\Omega }|\ne 0$ according to $\bar{\Omega }_{lm}$. So, $W_\mathrm {gmm}{{\,\mathrm{E}\,}}(\varphi _i(\beta ))=\mathbf {0}$ if and only if $\beta =\beta _0$.

• 2. $\Theta$ is apparently compact.

• 3. $\varphi _i(\beta )$ is continuous in $\beta$.

• 4.
  $$\begin{aligned} \begin{aligned} {{\,\mathrm{E}\,}}(\sup \limits _{\beta \in \Theta }\parallel \varphi _i(\beta )\parallel ^2)&\le C_s^2 {{\,\mathrm{E}\,}}(\sup \limits _{\beta \in \Theta }(w_i^{j^\mathrm {T}}w_i^j(y_i-{w_i^k}^\mathrm {T}\beta )^2))\\&=C_s^2(x_i^\mathrm {T}x_i+p\sigma _2^2){{\,\mathrm{E}\,}}(\sup \limits _{\beta \in \Theta }(y_i-{w_i^k}^\mathrm {T}\beta )^2)\\&< \infty . \end{aligned} \end{aligned}$$

  The last inequality is satisfied because $\beta \in \Theta$ and $y_i$, $w_i^j$ are finite dimension random variables.

Thus, the results follow from Theorem 2.6 in Newey and McFadden (1994).

A.8 Proof of Theorem 5

• 1. We have proved the consistency of $\hat{\beta }_\mathrm {gmm}$.

• 2. $\beta _0$ is in the interior of $\Theta$.

• 3. $\varphi _n(\beta )$ is continuously differentiable of $\beta$.

• 4. With iid $\varphi _i(\beta )$, we have $\sqrt{n}\varphi _n(\beta _0)\overset{D}{\rightarrow } N(\mathbf {0},\bar{\Omega })$ by central limit theorem.

• 5. $\nabla _{\beta }\varphi _n(\beta )=\frac{-1}{n}\sum _{i=1}^{n} \left( \begin{array}{c}...\\ w_i^kw_i^{j^\mathrm {T}}\\ ...\\ \end{array}\right) \overset{p}{\rightarrow }-i_{\tau }\otimes A^{-1}$ for any $\beta$.

Thus, the results follow from Theorem 3.2 and Theorem 5.2 in Newey and McFadden (1994).