Estimating moments in ANOVA-type mixed models

Abstract

In the paper, a simple projection-based method is systematically developed to estimate the qth ($q\ge 2$) order moments of random effects and errors in the ANOVA type mixed model (ANOVAMM), where the response may not be divided into independent sub-vectors. All the estimates are weakly consistent and the second-order moment estimates are strongly consistent. Besides, the derived estimates are different from those in mixed models with cluster design. Simulation studies are conducted to examine the finite sample performance of the estimates and two real data examples are analyzed for illustration.

Appendices

Appendix A: Regularity conditions

Condition C.1: The random errors $\{\epsilon _i\}_{i=1}^n$ are independent with the same qth order moment $\gamma _q^\epsilon$ for $q\ge 2$.

Condition C.2: (1). As $n\rightarrow \infty$, $n-rw\rightarrow \infty$ where $rw=\mathrm{rank} (W)$;

(2) $rw/n=o(1)$ and $s_{(n)}=o(n^{-1/4}\text{ log }^{-2}n)$ where $s_{(n)}=\max _{1\le i \le n} \sum _{1\le j \le n, j\ne i} |l_{ij}|$;

(3) For $k=1,\ldots ,s$, $s_{(n)}^{(k)}=o\left( m_k^{-1/4}\text{ log }^{-2}m_k\right)$ where $s_{(n)}^{(k)}=\max _{1\le j \le m_k} \sum _{j\ne j'} |a_{jj'}^{(k)}|$ with $a_{jj'}^{(k)}$ being the $(j,j')$th element of $Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k$ [see the identity in (17)].

Condition C.3: The random effects $\{\alpha _k\}_{k=1}^s$ are independent and independent of the random errors $\{\epsilon _i\}_{i=1}^n$. Besides, $\alpha _k$ has independent components $\alpha _{kj}$ (for $j=1,\ldots ,m_k$) with the same qth order moment $\gamma _q^{\alpha _{k}}\equiv E(\alpha _{k1}^q)=\cdots =E(\alpha _{km_k}^q)$.

Condition C.4: For $k=1,2,\ldots ,s$, $\lim _{n\rightarrow \infty } \frac{\sum _{i=1}^n \sum _{i'=1}^n \left( u_{i}^{(k)} {u_{i'}^{(k)}}^\mathrm{T}\right) ^2}{\left\{ \sum _{j=1}^{m_k} \sum _{i=1}^n \left( u_{ij}^{(k)}\right) ^2\right\} ^2}=0$.

Condition C.5: The eigenvalues of $\{Z_k Z_k^\mathrm{T}\}_{k=1}^s$ are bounded away from zero and infinity.

Condition C.6: There are some positive constants $c_1$ and $c_2$ so that $c_1 \le \mathrm{tr}(Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k)/m_k \le c_2$ for $k=1,\ldots ,s$.

Condition C.7: (1) As $n\rightarrow \infty$, $\sum _{i=1}^n \sum _{j=1}^n l_{ij}^q \rightarrow \infty$ for $q\ge 2$;

(2)As $n\rightarrow \infty$, $\sum _{i=1}^n \sum _{j=1}^{m_k} \left( u_{ij}^{(k)}\right) ^q \rightarrow \infty$ for $q\ge 2$.

Condition C.8: There are the finite $(q+\delta )$th order moments of $\epsilon _i$ and $\alpha _{k1}$ with $k=1,\ldots ,s$ for some positive constant $\delta$.

Remark 3

Most of the conditions are standard. Conditions C.2(2)(3) are also reasonable. Notice that $P_{W^\bot }$ is an idempotent matrix and $s_{(n)}\le \max _{1\le i \le n} l_{ii}\equiv l_{(n)}$. If $l_{ii}=1$, then $l_{ij}= 0$ for $j\ne i$; otherwise, $0\le l_{ij}<l_{(ii)}<1$. In fact, Condition C.2(2) is the requirement for off-diagonal elements in $P_{W^\bot }$, which is similar to condition C.3 in Cui (2004). Similarly for Condition C.2(3). Our conditions of establishing the (strong) consistency of estimates may not be the most general, and may be reduced.

Appendix B: Proofs

Some identities and lemmas are introduced first.

Identities:

$$\begin{aligned}&\sum _{i=1}^n \sum _{j=1}^n l_{ij}^2 = \mathrm{tr} (P_{W^\bot })=n-rw, \sum _{i=1}^n \sum _{j=1}^n \left( v_{ij}^{(k)}\right) ^2=\mathrm{tr} (P_{W_{-k}^\bot })=n-rw_{-k}, \nonumber \\&a_{jj'}^{(k)}\equiv \left( Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k\right) _{jj'}= \sum _{i=1}^n u_{ij}^{(k)} u_{ij'}^{(k)}, sum_{j=1}^{m_k} \sum _{i=1}^n \left( u_{ij}^{(k)}\right) ^2=\mathrm{tr} \left( Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k\right) ,\nonumber \\ \end{aligned}$$

(17)

Notice that the (i, j)th element of $\left( Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k\right) ^2$ is $\sum _{i'=1}^{m_k}u_{i}^{(k)} {u_{i'}^{(k)}}^\mathrm{T}u_{i'}^{(k)} {u_{j}^{(k)}}^\mathrm{T}$. Then

$$\begin{aligned}&\mathrm{tr} \left\{ (Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k)^2\right\} = \sum _{i=1}^n \sum _{i'=1}^n \left( u_{i}^{(k)} {u_{i'}^{(k)}}^\mathrm{T}\right) ^2,\nonumber \\&\quad \frac{\mathrm{tr} \left\{ (Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k)^2\right\} }{\left\{ \mathrm{tr} (Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k)\right\} ^2} = \frac{\sum _{i=1}^n \sum _{i'=1}^n \left( u_{i}^{(k)} {u_{i'}^{(k)}}^\mathrm{T}\right) ^2}{\left\{ \sum _{j=1}^{m_k} \sum _{i=1}^n \left( u_{ij}^{(k)}\right) ^2\right\} ^2}. \end{aligned}$$

These identities can be easily derived by some elementary calculations, and the details are omitted. They are of interest by themselves and are important to prove the theorems.

1.1 Lemmas:

Lemma 1

(Convergence of weighted sums of independent random variables) Assume that the independent random variables $\xi _i$ have the same mean $E(\xi _1)$ and there are positive constants $b_i$ such that $\lim _{n\rightarrow \infty } B_n =\infty$ with $B_n=\sum _{i=1}^n {b_i}$. If $\lim _{n\rightarrow \infty }\max _{1\le i \le n} b_i/B_n =0$ and $\sup _{n}\{N(n)/n\} <\infty$ where $N(n) \equiv \#(i:b_i^{-1}B_i \le n)$ denotes the number of i subject to $b_i^{-1}B_i\le n$, then $B_n^{-1} \sum _{i=1}^n b_i \xi _i \mathop {\rightarrow }\limits ^{a.s.} E(\xi _1)$. Moreover, if $c_1\le b_i\le c_2$ for some positive constants $c_1$ and $c_2$, then $B_n^{-1} \sum _{i=1}^n b_i \xi _i \mathop {\rightarrow }\limits ^{a.s.} E(\xi _1)$.

This lemma can be found in Jamison et al. (1965). $\square$

Lemma 2

(Convergence of weighted sums of independent random variables) Assume that $\xi _1,\xi _2,\ldots ,\xi _n,\ldots$ are i.i.d. random variables with $E(\xi _1)=0$ and $E|\xi _1|^r < \infty$ for some $r\in (1,2)$, the constants $b_i\ne 0$ for $i\ge 1$ and $0<B_1 \le B_2 \le \cdots \rightarrow \infty$. Then if $\{N(n)/n^r\}=O(1)$ with N(n) defined in Lemma 1, we have $B_n^{-1} \sum _{i=1}^n b_i \xi _i \mathop {\rightarrow }\limits ^{a.s.} E(\xi _1)$.

This is Theorem 2 in Chen et al. (1996). $\square$

Lemma 3

Assume that $\xi _1,\xi _2,\ldots ,\xi _n$ are independent random variables satisfying $E(\xi _i)=0$ and $\sup _i E|\xi _i|^r < \infty$ for some $r\ge 1$. Let $\{b_{ij}\}$ be a series of real numbers. Then

$$\begin{aligned} max_{1\le j \le n}\sum _{i=1}^n \left| b_{ij}\xi _i\right| =o\left\{ \text{ log }^2n \cdot \max \left( n^{1/r}d_n,\sqrt{s_n d_n}\right) \right\} \quad a.s. \end{aligned}$$

where $s_n=\max _{1\le j \le n}\sum _{i=1}^n |b_{ij}|$, $d_n=\max _{1\le i,j\le n}|b_{ij}|$.

It is Lemma 3 in Cui (2004). $\square$

Proof of Theorem 1

(1) Strong consistency of $\widehat{\gamma }_2^\epsilon$. Notice that $\widehat{\gamma }_2^\epsilon =\epsilon ^\mathrm{T}P_{W^\bot } \epsilon /(n-rw)$. For $i,j=1,\ldots ,n$, let $l_{ij}$ be the (i, j)th element of the matrix $P_{W^\bot }$. Then $l_{ij}$ is the jth element of $l_i$ defined in the text where $l_{ii}\ge 0$ and $\sum _{i=1}^n l_{ii}=n-rw$. Further, $P_{W^\bot }^\mathrm{T}=P_{W^\bot }$ and $P_{W^\bot }^2=P_{W^\bot }$ imply $\sum _{j=1}^n l_{ij}^2=l_{ii}$, which yields $0\le l_{ii} \le 1$ for all $1\le i \le n$ and for $i\ne j$, if $l_{ij}\ne 0$, $0<l_{ii}<1$. Notice that

$$\begin{aligned} \widehat{\gamma }_2^\epsilon =\frac{\sum _{i=1}^n l_{ii} \epsilon _i^2}{n-rw}+\frac{\sum _{i\ne j} l_{ij} \epsilon _i \epsilon _j}{n-rw} \equiv II_1+II_2. \end{aligned}$$

Without loss of generality, assume $l_{ii}>0$ for $i=1,\ldots ,n'$ with $n' \le n$. Thus, $II_1={\sum _{i=1}^{n'} l_{ii} \epsilon _i^2}/{(n-rw)}$ and $n-rw=\sum _{i=1}^{n'} l_{ii}$. According to Conditions C.1 and C.2 (1) and Lemma 1, $II_1 \mathop {\rightarrow }\limits ^{a.s.} {\gamma }_2^\epsilon$.

Further, $II_2\mathop {\rightarrow }\limits ^{a.s.} 0$. In fact, let $s_{(n)}=\max _{1\le i \le n}\left\{ \sum _{1\le j \le n, j\ne i}|l_{ij}|\right\}$, $d_{(n)}=\max _{1\le i\ne j \le n} |l_{ij}|$, and then one has $d_{(n)}\le s_{(n)}$. According to the nonnegative definite property of $P_{W^\bot }$, $0\le d_{(n)} \le s_{(n)} \le l_{(n)} \le 1$ where $l_{(n)}=\max _{1\le i \le n}l_{ii}$. Then by Conditions C.1, C.2(2) and Lemma 3, $\max _{1\le i \le n} |\sum _{j\ne i} l_{ij} \epsilon _j|=o(1)$ a.s. Moreover, noting that $0\le II_2 \le s_{(n)} \cdot \sum _{i=1}^n |\epsilon _i | /(n-rw)$, one has $II_2\mathop {\rightarrow }\limits ^{a.s.}0$.

(2) Strong consistency of $\widehat{\gamma }_2^{\alpha _k}$ for $k=1,2,\ldots ,s$. Recalling (3) and (6), one has

$$\begin{aligned} \widehat{\gamma }_2^{\alpha _k}= & {} \frac{\sum _{i=1}^n \left( {u_i^{(k)}} \alpha _k \right) ^2 }{\sum _{i=1}^n \sum _{j=1}^{m_k} \left( u_{ij}^{(k)}\right) ^2 } +\frac{\sum _{i=1}^n \left( {v_i^{(k)}}^\mathrm{T}\epsilon \right) ^2 - {\gamma }_2^\epsilon \cdot \sum _{i=1}^n \sum _{j=1}^n \left( v_{ij}^{(k)}\right) ^2}{\sum _{i=1}^n \sum _{j=1}^{m_k} \left( u_{ij}^{(k)}\right) ^2 } \nonumber \\&-(\widehat{\gamma }_2^\epsilon -{\gamma }_2^\epsilon ) \cdot \frac{ \sum _{i=1}^n \sum _{j=1}^n \left( v_{ij}^{(k)}\right) ^2}{\sum _{i=1}^n \sum _{j=1}^{m_k} \left( u_{ij}^{(k)}\right) ^2 } +2 \frac{ \sum _{i=1}^n \left( u_{i}^{(k)} \alpha _k {v_{i}^{(k)}}^\mathrm{T}\epsilon \right) }{\sum _{i=1}^n \sum _{j=1}^{m_k} \left( u_{ij}^{(k)}\right) ^2 } \nonumber \\\equiv & {} IIk_1+IIk_2- (\widehat{\gamma }_2^\epsilon -{\gamma }_2^\epsilon ) \cdot IIk_3 +2 IIk_4. \end{aligned}$$

(18)

Notice that by the identity (17)

$$\begin{aligned} IIk_1= & {} \frac{\sum _{j=1}^{m_k}\sum _{i=1}^n \left( {u_{ij}^{(k)}} \right) ^2 \alpha _{kj}^2 }{\sum _{j=1}^{m_k} \sum _{i=1}^n \left( u_{ij}^{(k)}\right) ^2 } {+} \frac{\sum _{j=1}^{m_k} \alpha _{kj} \left\{ \sum _{1\le j' \le m_k, j'\ne j'} \left( \sum _{i=1}^n {u_{ij}^{(k)}} {u_{ij'}^{(k)}} \right) \alpha _{kj'} \right\} }{\sum _{j=1}^{m_k} \sum _{i=1}^n \left( u_{ij}^{(k)}\right) ^2 } \\= & {} \frac{\sum _{j=1}^{m_k} a_{jj}^{(k)} \alpha _{kj}^2 }{\sum _{j=1}^{m_k}a_{jj}^{(k)}} + \frac{\sum _{j=1}^{m_k} \alpha _{kj} \left\{ \sum _{1\le j' \le m_k, j'\ne j'} a_{jj'}^{(k)} \alpha _{kj'} \right\} }{\sum _{j=1}^{m_k}a_{jj}^{(k)}}\\\equiv & {} IIk_{11}+IIk_{12}, \end{aligned}$$

where $a_{jj}^{(k)}\ge \sum _{j' \ne j} |a_{jj'}^{(k)}|$ by the nonnegative definite property of the matrix $Z_k^\mathrm{T}P_{W_{-k}^\bot } Z_k$. Thus, similar to the derivation for the strong consistency of $\widehat{\gamma }_2^\epsilon$ and under Condition C.5, one can obtain $IIk_{11}\mathop {\rightarrow }\limits ^{a.s.} {\gamma }_2^{\alpha _k}$ by Lemma 1 and $IIk_{12}\mathop {\rightarrow }\limits ^{a.s.} 0$ by Lemma 3, Conditions C.2(1), C.2(3), C.5 and C.6. These yield $c_1\le \sum _{j=1}^{m_k} a_{jj}^{(k)} /m_k \le c_2$ for some positive constants $c_1,c_2$. Thus, $IIk_{1}\mathop {\rightarrow }\limits ^{a.s.} {\gamma }_2^{\alpha _k}$.

Further, noting that $0\le \sum _{j\ne i}|v_{ij}^{(k)}| \le v_{ii}^{(k)}\le 1$, Condition C.2(2) indicates that $rw_{-k}/n=o(1)$ and Conditions C.2(2),(3) and C.5 imply that $\tilde{s}_{(n)}=o(n^{-1/4}\text{ log }^{-2}n)$ where $\tilde{s}_{(n)}=\max _{1\le i \le n} \sum _{j\ne i} |v_{ij}^{(k)}|$, one has ${\sum _{i=1}^n \left( {v_i^{(k)}}^\mathrm{T}\epsilon \right) ^2}/{\sum _{i=1}^n \left( {v_{ij}^{(k)}}^2 \right) }\mathop {\rightarrow }\limits ^{a.s.} {\gamma }_2^\epsilon$. Moreover, Condition C.5 yields $IIk_3=O(1)$ which leads to $IIk_2\mathop {\rightarrow }\limits ^{a.s.} 0$. Noting that $E(IIk_4)=0$ and

$$\begin{aligned} IIk_4=\frac{\sum _{j=1}^{m_k} \alpha _{kj} \left[ \sum _{j'=1}^n \epsilon _{j'} \left\{ \sum _{i=1}^n \left( u_{ij}^{(k)} v_{ij'}^{(k)}\right) \right\} \right] }{\sum _{j=1}^{m_k} \sum _{i=1}^n \{u_{ij}^{(k)}\}^2}, \end{aligned}$$

one has $IIk_4 \mathop {\rightarrow }\limits ^{a.s.} 0$ by Conditions C.2(3), C.3, C.5, C.6 and Lemma 3. $\square$

Proof of Theorem 2

(1) The consistency of $\widehat{\gamma }_q^{\epsilon }$ with $q\ge 2$. According to the derivation of $\widehat{\gamma }_q^{\epsilon }$, one has

$$\begin{aligned} E\left\{ \sum _{i=1}^n \left( l_i^\mathrm{T}Y \right) ^q \right\}= & {} \left( \sum _{i=1}^n \sum _{j=1}^n l_{ij}^q \right) \cdot E(\epsilon _1^q) \nonumber \\&+\sum _{S(n;0,q-2;q)} \sum _{i=1}^n \left( \mathop \Pi \limits _{j=1}^n l_{l_ij}^{s_j} \right) \cdot \left\{ E(\epsilon _1^{s_1}) \ldots E(\epsilon _1^{s_n}) \right\} , \end{aligned}$$

and then it is sufficient to verify

$$\begin{aligned}&\dfrac{\sum _{i=1}^n \left\{ \left( l_i^\mathrm{T}\epsilon \right) ^q - E\left( l_i^\mathrm{T}\epsilon \right) ^q \right\} }{\sum _{i=1}^n \sum _{j=1}^n l_{ij}^q} =o_p(1), \end{aligned}$$

(19)

$$\begin{aligned}&\dfrac{ \sum _{S(n;0,q-2;q)} \sum _{i=1}^n \left( \Pi _{j=1}^n l_{l_ij}^{s_j} \right) \cdot \left\{ \gamma ^\epsilon _{s_1} \ldots \gamma ^\epsilon _{s_n}- \widehat{\gamma }_\epsilon ^{s_1} \ldots \widehat{\gamma }_\epsilon ^{s_n} \right\} }{\sum _{i=1}^n \sum _{j=1}^n l_{ij}^q} =o_p(1). \end{aligned}$$

(20)

In fact, notice that $0\le s_1,\ldots , s_n \le q$ and $s_1+\ldots s_n=q$ with the finite q, for each $(s_1,\ldots ,s_n)$ with finite items, $\frac{\sum _{i=1}^n \left( \Pi _{j=1}^n l_{l_{ij}}^{s_j} \right) }{\sum _{i=1}^n \sum _{j=1}^n l_{ij}^q}=O(1)$ under condition C.7(1), and then by the consistency of $\widehat{\gamma }_{s_j}^{\epsilon }$ with $s_j<q$, (20) is derived, and $\frac{ \Pi _{j=1}^n \sum _{i=1}^n l_{l_{ij}}^{s_j} \left\{ \epsilon _j^{s_j}-E\left( \epsilon _j^{s_j}\right) \right\} }{\sum _{i=1}^n \sum _{j=1}^n l_{ij}^q}=o_p(1)$ by condition C.7(1) and C.8, which yields (19). (2) The consistency of $\widehat{\gamma }^{\alpha _{k}}_q$ with $q\ge 2$. Recalling the estimate $\widehat{\gamma }^{\alpha _{k}}_q$ in (8), one has $\widehat{\gamma }^{\alpha _{k}}_q = \widehat{II}_1^{\alpha q} -\widehat{II}_2^{\alpha q} -\sum _{t=0}^{q-1} C_q^t \cdot \widehat{II}_3^{\alpha qt}$ where

$$\begin{aligned} \widehat{II}_1^{\alpha q}= & {} \frac{\sum _{i=1}^n \left( {v_i^{(k)}}^\mathrm{T}Y \right) ^q}{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q} =\frac{\sum _{i=1}^n \left( {u_i^{(k)}} \alpha _k + {v_i^{(k)}}^\mathrm{T}\epsilon \right) ^q}{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}, \\ \widehat{II}_2^{\alpha q}= & {} \frac{\sum _{S(m_k;0,q-2;q)} \left[ \sum _{i=1}^n \Pi _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^{s_{j'}} \cdot \left\{ \widehat{\gamma }^{\alpha _{k}}_{s_1} \cdots \widehat{\gamma }^{\alpha _{k}}_{s_{m_k}} \right\} \right] }{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q},\\ \widehat{II}_3^{\alpha qt}= & {} \frac{\sum _{i=1}^n \left[ \sum _{S(m_k;0,t;t)} \Pi _{j'=1}^{m_k} \left\{ \left( u_{ij'}^{(k)}\right) ^{s_{j'}} \widehat{\gamma }^{\alpha _{k}}_{s_{j'}} \right\} \right] \left[ \sum _{S(n;0,q-t;q-t)} \Pi _{j=1}^{n} \left\{ \left( v_{ij}^{(k)}\right) ^{s_{j}} \widehat{\gamma }^{\epsilon }_{s_j} \right\} \right] }{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}\\\equiv & {} \frac{\sum _{i=1}^n \widehat{II}_{31zi}^{\alpha qt} \cdot \widehat{II}_{32zi}^{\alpha qt}}{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}, \end{aligned}$$

and ${II}_2^{\alpha q}$, ${II}_3^{\alpha qt}$, ${II}_{31zi}^{\alpha qt}$, ${II}_{32zi}^{\alpha qt}$ are the corresponding items of $\widehat{II}_2^{\alpha q}$, $\widehat{II}_3^{\alpha qt}$, $\widehat{II}_{31zi}^{\alpha qt}$, $\widehat{II}_{32zi}^{\alpha qt}$ in which the estimates $\widehat{\gamma }^{\alpha _k}_{s_{j_k}}$, $\widehat{\gamma }^\epsilon _{s_{j'}}$ are replaced by their true values ${\gamma }^{\alpha _k}_{s_{j_k}}$ and ${\gamma }^\epsilon _{s_{j'}}$ with $s_{j_k}=1,\ldots ,q$, and $s_{j'}=1,\ldots ,q-1$. Besides, notice that $E\left( \widehat{II}_1^{\alpha q}\right) =E(\alpha _{k1}^q)-II_2^{\alpha q}-\sum _{t=0}^{q-1} C_q^t \cdot II_3^{\alpha qt}$, and by the consistency of $\widehat{\gamma }^{\alpha _k}_{s_{j_k}}$ in Theorem 1, condition C.7(2) and the fact that q is finite, $\widehat{II}_2^{\alpha q}-{II}_2^{\alpha q}=o_p(1)$. Thus, it suffices to derive that $\widehat{II}_1^{\alpha q}-{II}_1^{\alpha q} = o_p(1)$ and $\widehat{II}_3^{\alpha qt}-{II}_3^{\alpha qt} = o_p(1)$. In fact,

$$\begin{aligned}&\displaystyle \widehat{II}_1^{\alpha q}-{II}_1^{\alpha q} = \sum _{t=0}^q C_q^t \cdot II_{nt}^{\alpha q}, \\&\displaystyle II_{nt}^{\alpha q}= \frac{\sum _{i=1}^n \left\{ \left( {u_i^{(k)}} \alpha _k\right) ^t \left( {v_i^{(k)}}^\mathrm{T}\epsilon \right) ^{q-t}- E\left( {u_i^{(k)}} \alpha _k\right) ^t \cdot E\left( {v_i^{(k)}}^\mathrm{T}\epsilon \right) ^{q-t}\right\} }{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}, \end{aligned}$$

and then by conditions C.1, C.5, C.7(2), C.8, we have $II_{nt}^{\alpha q}=o_p(1)$ for $t=0,\ldots ,q$.

For $t=0$, $\widehat{II}_{31zi}^{\alpha qt}=1$ and then by the consistency of $\widehat{\gamma }_{s_{j_k}}^{\epsilon }$ with $s_{j_k}<q$, the finiteness of q, conditions C.5, C.7(2)

$$\begin{aligned} \widehat{II}_3^{\alpha qt}-{II}_3^{\alpha qt}= & {} \frac{ \sum _{S(m_k;0,q;q)} \sum _{i=1}^n \Pi _{j'=1}^{n} \left( v_{ij'}^{(k)}\right) ^{s_{j'}} \left\{ \widehat{E(\epsilon _{1}^{s_{j'}})} - E\left( \epsilon _{1}^{s_{j'}}\right) \right\} }{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}=o_p(1). \end{aligned}$$

Similarly for $1\le t \le q$ and by condition C.5 and the fact that $P_{W_{-k}^\bot }$ is idempotent,

$$\begin{aligned} \frac{\sum _{i=1}^n II_{31zi}^{\alpha qt} \left( \widehat{II}_{32zi}^{\alpha qt}-II_{32zi}^{\alpha qt}\right) }{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}=o_p(1), \quad \frac{ \sum _{i=1}^n \left( \widehat{II}_{31zi}^{\alpha qt}-II_{31zi}^{\alpha qt}\right) \cdot \widehat{II}_{32zi}^{\alpha qt}}{\sum _{i=1}^n \sum _{j'=1}^{m_k} \left( u_{ij'}^{(k)}\right) ^q}=o_p(1). \end{aligned}$$

Combining these with the fact that $\widehat{II}_{31zi}^{\alpha qt} \cdot \widehat{II}_{32zi}^{\alpha qt}-II_{31zi}^{\alpha qt} \cdot II_{32zi}^{\alpha qt}=\left( \widehat{II}_{31zi}^{\alpha qt}-II_{31zi}^{\alpha qt}\right) \cdot \widehat{II}_{32zi}^{\alpha qt}+II_{31zi}^{\alpha qt} \left( \widehat{II}_{32zi}^{\alpha qt}-II_{32zi}^{\alpha qt}\right)$, one has $\widehat{II}_3^{\alpha qt}-{II}_3^{\alpha qt}=o_p(1)$. $\square$