Batch Size Selection for Variance Estimators in MCMC

Abstract

We consider batch size selection for a general class of multivariate batch means variance estimators, which are computationally viable for high-dimensional Markov chain Monte Carlo simulations. We derive the asymptotic mean squared error for this class of estimators. Further, we propose a parametric technique for estimating optimal batch sizes and discuss practical issues regarding the estimating process. Vector auto-regressive, Bayesian logistic regression, and Bayesian dynamic space-time examples illustrate the quality of the estimation procedure where the proposed optimal batch sizes outperform current batch size selection methods.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Andrews D (1991) Heteroskedasticity and autocorrelation consistent covariant matrix estimation. Econometrica 59:817–858

    MathSciNet  MATH  Article  Google Scholar 

  2. Atchadé YF (2011) Kernel estimators of asymptotic variance for adaptive Markov chain Monte Carlo. The Annals of Statistics 39(2):990–1011

    MathSciNet  MATH  Article  Google Scholar 

  3. Boone EL, Merrick JR, Krachey MJ (2014) A Hellinger distance approach to MCMC diagnostics. J Stat Comput Simul 84(4):833–849

    MathSciNet  MATH  Article  Google Scholar 

  4. Brockmann M, Gasser T, Herrmann E (1993) Locally adaptive bandwidth choice for kernel regression estimators. J Am Stat Assoc 88(424):1302–1309

    MathSciNet  MATH  Article  Google Scholar 

  5. Bühlmann P (1996) Locally adaptive lag-window spectral estimation. J Time Ser Anal 17(3):247–270

    MathSciNet  MATH  Article  Google Scholar 

  6. Chan KW, Yau CY (2017) Automatic optimal batch size selection for recursive estimators of time-average covariance matrix. J Am Stat Assoc 112 (519):1076–1089

    MathSciNet  Article  Google Scholar 

  7. Damerdji H (1991) Strong consistency and other properties of the spectral variance estimator. Manag Sci 37:1424–1440

    MATH  Article  Google Scholar 

  8. Damerdji H (1995) Mean-square consistency of the variance estimator in steady-state simulation output analysis. Oper Res 43:282–291

    MathSciNet  MATH  Article  Google Scholar 

  9. Elith J, Leathwick J, Hastie T (2008) A working guide to boosted regression trees. J Anim Ecol 77(4):802–813

    Article  Google Scholar 

  10. Finley AO, Banerjee S (2013) spBayes: univariate and multivariate spatial modeling R package version 0.3-7. http://CRAN.R-project.org/package=spBayes

  11. Finley AO, Banerjee S, Gelfand AE (2012) Bayesian dynamic modeling for large space-time datasets using gaussian predictive processes. J Geogr Syst 14(1):29–47

    Article  Google Scholar 

  12. Finley A, Banerjee S, Gelfand A (2015) spBayes for large univariate and multivariate point-referenced spatio-temporal data models. Journal of Statistical Software 63:1–28

    Article  Google Scholar 

  13. Flegal JM, Haran M, Jones GL (2008) Markov chain Monte Carlo: can we trust the third significant figure? Stat Sci 23:250–260

    MathSciNet  MATH  Article  Google Scholar 

  14. Flegal JM, Hughes J, Vats D, Dai N (2017) mcmcse: Monte Carlo standard errors for MCMC R package version 1.3-2. http://cran.r-project.org/web/packages/mcmcse/index.html

  15. Flegal JM, Jones GL (2010) Batch means and spectral variance estimators in Markov chain Monte Carlo. The Annals of Statistics 38:1034–1070

    MathSciNet  MATH  Article  Google Scholar 

  16. Geyer CJ (2011) Introduction to Markov chain Monte Carlo. In: Handbook of Markov chain Monte Carlo. CRC, London

  17. Glynn PW, Whitt W (1992) The asymptotic validity of sequential stopping rules for stochastic simulations. The Annals of Applied Probability 2:180–198

    MathSciNet  MATH  Article  Google Scholar 

  18. Janssen PH, Stoica P (1987) On the expectation of the product of four matrix-valued Gaussian random variables. Eindhoven University of Technology

  19. Jones GL (2004) On the Markov chain central limit theorem. Probab Surv 1:299–320

    MathSciNet  MATH  Article  Google Scholar 

  20. Jones GL, Haran M, Caffo BS, Neath R (2006) Fixed-width output analysis for Markov chain Monte Carlo. J Am Stat Assoc 101:1537–1547

    MathSciNet  MATH  Article  Google Scholar 

  21. Jones GL, Hobert JP (2001) Honest exploration of intractable probability distributions via Markov chain Monte Carlo. Stat Sci 16:312–334

    MathSciNet  MATH  Article  Google Scholar 

  22. Jones MC, Marron JS, Sheather SJ (1996) A brief survey of bandwidth selection for density estimation. J Am Stat Assoc 91(433):401–407

    MathSciNet  MATH  Article  Google Scholar 

  23. Kuelbs J, Philipp W (1980) Almost sure invariance principles for partial sums of mixing b-valued random variables. Ann Probab 8:1003–1036

    MathSciNet  MATH  Article  Google Scholar 

  24. Leathwick J, Elith J, Chadderton W, Rowe D, Hastie T (2008) Dispersal, disturbance and the contrasting biogeographies of New Zealand’s diadromous and non-diadromous fish species. J Biogeogr 35(8):1481–1497

    Article  Google Scholar 

  25. Liu Y, Flegal J (2018) Weighted batch means estimators in Markov chain Monte Carlo. Electronic Journal of Statistics 12:3397–3442

    MathSciNet  MATH  Article  Google Scholar 

  26. Loader CR (1999) Bandwidth selection: classical or plug-in? Ann Stat 27:415–438

    MathSciNet  MATH  Article  Google Scholar 

  27. Newey WK, West KD (1987) A simple, positive semi-definite, heteroskedasticity and autocorrelation consistent covariance matrix. Econometrica 55:703–708

    MathSciNet  MATH  Article  Google Scholar 

  28. Plummer M, Best N, Cowles K, Vines K (2006) CODA: convergence diagnosis and output analysis for MCMC. R news 6(1):7–11

    Google Scholar 

  29. Politis DN (2003) Adaptive bandwidth choice. Journal of Nonparametric Statistics 15(4-5):517–533

    MathSciNet  MATH  Article  Google Scholar 

  30. Politis DN (2011) Higher-order accurate, positive semidefinite estimation of large-sample covariance and spectral density matrices. Econometric Theory 27(4):703–744

    MathSciNet  MATH  Article  Google Scholar 

  31. Politis DN, Romano JP (1995) Bias-corrected nonparametric spectral estimation. J Time Ser Anal 16(1):67–103

    MathSciNet  MATH  Article  Google Scholar 

  32. Politis DN, Romano JP (1996) On flat-top kernel spectral density estimators for homogeneous random fields. Journal of Statistical Planning and Inference 51(1):41–53

    MathSciNet  MATH  Article  Google Scholar 

  33. Politis DN, Romano JP (1999) Multivariate density estimation with general flat-top kernels of infinite order. J Multivar Anal 68(1):1–25

    MathSciNet  MATH  Article  Google Scholar 

  34. Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. Journal of the Royal Statistical Society. Series B (Methodological) 53:683–690

    MathSciNet  MATH  Article  Google Scholar 

  35. Silverman BW (1999) Density Estimation for Statistics and Data Analysis. Chapman & Hall Ltd

  36. Song WT, Schmeiser BW (1995) Optimal mean-squared-error batch sizes. Manag Sci 41:110–123

    MATH  Article  Google Scholar 

  37. Taylor (2018) Sum of autocovariances for AR(p) model. Cross Validated. https://stats.stackexchange.com/q/372006. (version: 2018-10-17)

  38. Thompson MB (2010) A comparison of methods for computing autocorrelation time. arXiv:1011.0175

  39. Tjøstheim D (1990) Non-linear time series and Markov chains. Adv Appl Probab 22:587–611

    MathSciNet  MATH  Article  Google Scholar 

  40. Vats D, Flegal JM (2018) Lugsail lag windows and their application to MCMC. arXiv:1809.04541

  41. Vats D, Flegal JM, Jones GL (2018) Strong consistency of multivariate spectral variance estimators in Markov chain Monte Carlo. Bernoulli 24:1860–1909

    MathSciNet  MATH  Article  Google Scholar 

  42. Vats D, Flegal JM, Jones GL (2019) Multivariate output analysis for Markov chain Monte Carlo. Biometrika 106:321–337

    MathSciNet  MATH  Article  Google Scholar 

  43. Woodroofe M (1970) On choosing a delta-sequence. The Annals of Mathematical Statistics 41(5):1665–1671

    MathSciNet  MATH  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to James M. Flegal.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Proof of Theorem 1

This section presents a proof of Theorem 1 and the optimal batch size results for the generalized OBM estimators. We will use the fact that

$$ \text{MSE}\left( \hat{\Sigma}_{w, ij} \right) = \left( \text{Bias}(\hat{\Sigma}_{w,ij}) \right)^{2} + \text{Var}\left( \hat{\Sigma}_{w, ij} \right). $$

The bias and variance results of the generalized OBM estimators are important in their own right, and are presented here separately. Denote \(\lim _{n\rightarrow \infty } f(n)/g(n)=0\) by f(n) = o(g(n)) and if f(n)/g(n) stays uniformly bounded for all n, then we write f(n) = O(g(n)). Recall notation \(\bar {Y}_{l}(k)=k^{-1}{\sum }_{t=1}^{k}Y_{l+t}\) for l = 0,...,nk and \(\bar {Y} = n^{-1} {\sum }_{t=1}^{n} Y_{t}\). Let \(\bar {Y}_{l}^{(i)}(k)\) and \(\bar {Y}^{(i)}\) denote the i th component of \(\bar {Y}_{l}(k)\) and \(\bar {Y}\), respectively.

Theorem 2

Suppose \(\text {E}_{F}\|g\|^{2+\delta } < \infty \) for δ > 0 and \({\sum }_{k=1}^{b}k{\Delta }_{2}w_{n}(k)=1\). If {Xt} is a polynomially ergodic Markov chain of order ξ > (2 + 𝜖)(1 + 2/δ) for some 𝜖 > 0, then

$$ \text{Bias}\left( \hat{\Sigma}_{w,ij} \right) = \sum\limits_{k=1}^{b}{\Delta}_{2} w_{n}(k) {\Gamma}_{ij} + o\left( \frac{b}{n} \right) + o\left( \frac{1}{b} \right). $$

Proof

By Vats and Flegal (2018, Theorem 2), \(|{\Gamma }_{ij}| < \infty \). Then under Assumption 1, and following (Song and Schmeiser 1995), for all i and j,

$$ \text{Cov}[\bar{Y}_{l}^{(i)}(k),\bar{Y}_{l}^{(j)}(k)]-\text{Cov}[\bar{Y}^{(i)},\bar{Y}^{(j)}]=\frac{n-k}{kn}\left( {\Sigma}_{ij}+\frac{n+k}{kn}{\Gamma}_{ij}+o\left( \frac{1}{k^{2}}\right)\right). $$
(4)

Since \({\sum }_{k=1}^{b}k{\Delta }_{2}w_{n}(k)=1\), by Eq. 4,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}\left( \hat{\Sigma}_{w,ij}\right)&=& \sum\limits_{k=1}^{b}\frac{(n-k+1)(n-k)k{\Delta}_{2}w_{n}(k)}{n^{2}} {\Sigma}_{ij} \\ && + \sum\limits_{k=1}^{b}\frac{(n-k+1)(n^{2}-k^{2}){\Delta}_{2}w_{n}(k)}{n^{3}} {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right)\\ &=&{\Sigma}_{ij}+ \sum\limits_{k=1}^{b}\frac{(n-k+1)(n^{2}-k^{2}){\Delta}_{2}w_{n}(k)}{n^{3}} {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right)\\ &=&{\Sigma}_{ij}+ \sum\limits_{k=1}^{b}{\Delta}_{2}w_{n}(k) {\Gamma}_{ij}+o\left( \frac{b}{n}\right)+o\left( \frac{1}{b}\right). \end{array} $$

Next, we obtain \(\text {Var}\left (\hat {\Sigma }_{w, ij} \right )\). The proof here is under a more general strong invariance principle. For a function \(f: \mathsf {X} \to \mathbb {R}^{p}\), assume there exists a p × p lower triangular matrix L, a non-negative increasing function ψ on the positive integers, a finite random variable D, and a sufficiently rich probability space Ω such that for almost all ω ∈Ω and for all n > n0,

$$ \left\lVert \sum\limits_{t=1}^{n} f(X_{t}) - n \mathrm{E}_{F} f - LB(n)\right\rVert<D(\omega)\psi(n) \quad \text{with probability 1.} $$
(5)

Under the conditions of Lemma 1, Vats et al. (2018) establish (5) with ψ(n) = n1/2−λ for some λ > 0. Appendix B contains a number of preliminary results, followed by the proof of Theorem 3 in Appendix C.

Theorem 3

Suppose (5) holds for f = g and f = g2 (where the square is element-wise) such that \(\mathrm {E}_{F}D^{4} < \infty \) and Assumption 1 holds. If

  1. 1.

    \( {\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2} = O\left (1/b^{2}\right ), \)

  2. 2.

    \(b\psi ^{2}(n)\log n \left ({\sum }_{k=1}^{b}|{\Delta }_{2}w_{n}(k)| \right )^{2}\rightarrow 0\), and

  3. 3.

    \(\psi ^{2}(n){\sum }_{k=1}^{b}|{\Delta }_{2}w_{n}(k)|\rightarrow 0\), then,

$$ \begin{array}{@{}rcl@{}} \text{Var} &&\left( \hat{\Sigma}_{w,ij}\right)\\ &=&[{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}] \bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3} \frac{1}{n}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u} {\Delta}_{2}w_{t+u} \left( \frac{2}{3}u^{3}+u^{2}t\right) \frac{1}{n}\bigg]+o\left( \frac{b}{n}\right)\\ &:=&\left( [{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}] S \frac{b}{n}+o(1) \right) \frac{b}{n}. \end{array} $$

Appendix B: Preliminaries

Proposition 1

If variable X and Y are jointly normally distributed with

$$ \left[ \begin{array}{l} X \\ Y \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}l_{11}& l_{12}\\ l_{12}& l_{22} \end{array}\right]\right), $$

then \(\mathrm {E}[X^{2}Y^{2}]=2l_{12}^{2}+l_{11}l_{22}\).

Proposition 2

Janssen and Stoica (1987) If X1, X2, X3, and X4 are jointly normally distributed with mean 0, then

$$ \mathrm{E}[X_{1}X_{2}X_{3}X_{4}]= \mathrm{E}[X_{1}X_{2}] \mathrm{E}[X_{3}X_{4}] + \mathrm{E}[X_{1}X_{3}] \mathrm{E}[X_{2}X_{4}] + \mathrm{E}[X_{1}X_{4}] \mathrm{E}[X_{2}X_{3}]. $$

Recall B = {B(t),t ≥ 0} is a p-dimensional standard Brownian motion. Let B(i)(t) be the i th component of vector B(t). Denote \(\bar {B}=n^{-1}B(n)\), \(\bar {B}_{l}(k)=k^{-1}[B(l+k)-B(l)]\). Let Σ = LLT, where L is a lower triangular matrix. Let C(t) = LB(t) and C(i)(t) be the i th component of C(t). Suppose \(\bar {C}^{(i)}_{l}(k)=k^{-1}(C^{(i)}(l+k)-C^{(i)}(l))\), and \(\bar {C}^{(i)}=n^{-1}C^{(i)}(n)\).

We now present some preliminary results and notation for the proof of Theorem 3.

Lemma 2

For any 0 < c2 < c1 < 1, define

$$ A_{2}=\frac{(c_{1}b)^{2}}{n^{2}}\mathrm{E}\left[\left[\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)\right]^{2}\right], \text{ and} $$
$$ A_{3} = -\frac{c_{1}c_{2}b^{2}}{n^{2}} \mathrm{E} \left[\sum\limits_{l=0}^{n-c_{1}b}(\bar{C}_{l}^{(i)}(c_{1}b) - \bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b) - \bar{C}^{(j)})\right]\left[\sum\limits_{l=0}^{n-c_{2}b}(\bar{C}_{l}^{(i)}(c_{2}b) - \bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{2}b) - \bar{C}^{(j)})\right]. $$

Then, A2 and A3 simplify to

$$ A_{2} = \left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{c_{1}b}{n}+{\Sigma}_{ij}^{2}-4{\Sigma}_{ij}^{2}\cdot\frac{c_{1}b}{n}\right]+o\left( \frac{b}{n}\right) \text{ and }, $$
$$ A_{3} = \frac{(c_{2} - 3c_{1})c_{2}}{3c_{1}}({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})\cdot \frac{b}{n} + 2\left( c_{1}+c_{2}\right)\cdot{\Sigma}_{ij}^{2}\cdot\frac{b}{n} - {\Sigma}_{ij}^{2} +o\left( \frac{b}{n}\right). $$

Proof

Denote

$$ a_{1}=\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right)^{2} \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)^{2}, $$
(6)
$$ a_{2}=\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)}), $$
(7)
$$ a_{3}=\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right). $$

Then

$$ \begin{array}{@{}rcl@{}} A_{2}&= \frac{c_{1}b^{2}}{n^{2}} \mathrm{E}[a_{1}+2a_{2}+2a_{3}]. \end{array} $$
(8)

First we calculate E[a1] in Eq. 6. Let \(U_{t}^{(i)}=B^{(i)}(t)-B^{(i)}(t-1)\), then \(U_{t}^{(i)} {\overset {iid}{\sim }} N(0,1)\) for t = 1, 2,...,n and

$$ \bar{B}_{l}^{(i)}(c_{1}b)-\bar{B}^{(i)}=\frac{(n-c_{1}b)}{nc_{1}b}\sum\limits_{t=l+1}^{l + c_{1}b}U^{(i)}_{t}-\frac{1}{n}\sum\limits_{t=1}^{l}U_{t}^{(i)}-\frac{1}{n}\sum\limits_{t=l+c_{1}b+1}^{n}U_{t}^{(i)}. $$

Notice that \(\mathrm {E}[\bar {B}_{l}^{(i)}(c_{1}b)-\bar {B}^{(i)}]=0\) for l = 0,..., (nc1b) and

$$ \text{Var} \left[\bar{B}_{l}^{(i)}(c_{1}b)-\bar{B}^{(i)} \right]=\Big(\frac{n-c_{1}b}{nc_{1}b}\Big)^{2}c_{1}b+\frac{n-c_{1}b}{n^{2}}=\frac{n-c_{1}b}{c_{1}bn}, $$

therefore

$$ \bar{B}_{l}(c_{1}b)-\bar{B}_{n}\sim N\left( 0,\ \frac{n-c_{1}b}{c_{1}bn}I_{p}\right) \Rightarrow \bar{C}_{l}(c_{1}b)-\bar{C}_{n} \sim N\left( 0,\ \frac{n-c_{1}b}{c_{1}bn}LL^{T}\right). $$
(9)

Now consider \(\mathrm {E}[(\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)})^{2}(\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)})^{2}]= {\mathrm {E}[Z_{i}^{2}}{Z_{j}^{2}}]\) where \(Z_{i}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}\) and \(Z_{j}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}\). Recall Σ = LLT, then

$$ \left[ \begin{array}{l} Z_{i} \\ Z_{j} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \frac{n-c_{1}b}{c_{1}bn}\left[\begin{array}{ll}{\Sigma}_{ii}& {\Sigma}_{ij}\\ {\Sigma}_{ij}& {\Sigma}_{jj} \end{array}\right]\right). $$

Apply Proposition 1,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}\left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right)^{2} \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)^{2} \right] & = \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}[{\Sigma}_{ij}^{2} + {\Sigma}_{ij}^{2}+{\Sigma}_{ii}{\Sigma}_{jj}]. \end{array} $$
(10)

Replacing Eqs. 10 in 6

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{1}]&= \sum\limits_{l=0}^{n-c_{1}b}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+o\left( \frac{n}{b}\right). \end{array} $$
(11)

To calculate E[a2] for s = 1, 2,..., (c1b − 1), we require

$$ \mathrm{E} \left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right]. $$

Notice that

$$ \begin{array}{@{}rcl@{}} \text{Cov}(\bar{C}_{l}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{l+s}(c_{1}b)-\bar{C}_{n}) &=L\cdot \mathrm{E} \left[ \left( \bar{B}_{l}(c_{1}b)-\bar{B}_{n} \right) \left( \bar{B}_{l+s}(c_{1}b)-\bar{B}_{n} \right)^{T} \right]\cdot L^{T}. \end{array} $$

Consider each entry of \(\mathrm {E}[(\bar {B}_{l}(c_{1}b)-\bar {B})(\bar {B}_{l+s}(c_{1}b)-\bar {B})^{T}]\). For ij,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+s}(c_{1}b)-\bar{B}^{(j)} \right] = \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right]\cdot \mathrm{E} \left[\bar{B}^{(j)}_{l+s}(c_{1}b)-\bar{B}^{(j)} \right]=0. $$
(12)

For i = j,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{l+s}(c_{1}b)-\bar{B}^{(i)} \right] \\ & =& \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b) \bar{B}^{(i)}_{l+s}(c_{1}b) \right] + \mathrm{E} \left[\bar{B}^{(i)} \bar{B}^{(i)} \right] - \mathrm{E} \left[ \bar{B}^{(i)} \bar{B}^{(i)}_{l+s}(c_{1}b) \right] - \mathrm{E} \left[ \bar{B}^{(i)} \bar{B}^{(i)}_{l}(c_{1}b) \right] \\ & = &\frac{1}{c_{1}b^{2}} \mathrm{E}\left[ \left( B^{(i)}(l+c_{1}b) - B^{(i)}(l) \right) \left( B^{(i)}(l+s+c_{1}b) - B^{(i)}(l + s) \right) \right] \\ && + \frac{1}{n^{2}}\mathrm{E}\left[ \left( B^{(i)}(n) \right)^{2} \right] - \frac{1}{nc_{1}b}\mathrm{E}\left[B^{(i)}(n) \left( B^{(i)} (l+c_{1}b+s) - B^{(i)}(l+s)\right) \right]\\ && - \frac{1}{nc_{1}b}\mathrm{E}\left[B^{(i)}(n) \left( B^{(i)} (l+c_{1}b) - B^{(i)}(l)\right) \right] \\ & =& \frac{n - c_{1}b}{c_{1}bn} - \frac{s}{n^{2}}. \end{array} $$
(13)

Combine Eqs. 13 and 12,

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{l+s}(c_{1}b)-\bar{C}_{n} \right) =\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)\cdot{\Sigma}. $$

Given Eqs. 912 and 13 and let \(Z_{1}=\bar {C}_{l}(c_{1}b)^{(i)}-\bar {C}^{(i)}_{n}\), \(Z_{2}=\bar {C}_{l}(c_{1}b)^{(j)}-\bar {C}^{(j)}_{n}\), \(Z_{3}=\bar {C}_{l+s}(c_{1}b)^{(i)}-\bar {C}^{(i)}_{n}\), \(Z_{4}=\bar {C}_{l+s}(c_{1}b)^{(j)}-\bar {C}^{(j)}_{n}\), (Z1,Z2,Z3,Z4)T has a 4-dimensional normal distribution with mean 0, and covariance matrix,

$$ \left[\begin{array}{llll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ii}& \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ij} &\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ii}&\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ij}\\ & \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{jj}& \left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{ij}&\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right){\Sigma}_{jj}\\ & &\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ii}&\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{ij} \\&&&\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}_{jj} \end{array}\right]. $$

Only upper triangle entries are presented due to symmetry of the matrix. By Proposition 2,

$$ \begin{array}{@{}rcl@{}} & \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] =\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2} \left[{\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2} \right] . \end{array} $$
(14)

Plug Eqs. 14 in 7,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E}[a_{2}] \\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E} \left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}{\Sigma}_{ii}{\Sigma}_{jj}+\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}{\Sigma}_{ij}^{2}\right]. \end{array} $$
(15)

Notice that

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}-\frac{s}{{c_{1}^{2}}b^{2}}\right)^{2}\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\frac{s^{2}}{{c_{1}^{4}}b^{4}}+\left( \frac{2}{{c_{1}^{2}}b^{2}n}-\frac{2}{{c_{1}^{3}}b^{3}}\right)s+\left( \frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}\right)\right]\\ &=&\sum\limits_{s=1}^{c_{1}b-1}\left[-\frac{s^{3}}{{c_{1}^{4}}b^{4}}+\left( \frac{n}{{c_{1}^{4}}b^{4}}+\frac{1}{{c_{1}^{3}}b^{3}}+\frac{1}{{c_{1}^{4}}b^{4}}-\frac{2}{{c_{1}^{2}}b^{2}n}\right)s^{2}\right. + \left( \frac{3}{{c_{1}^{2}}b^{2}}-\frac{2n}{{c_{1}^{3}}b^{3}}+\frac{2}{{c_{1}^{2}}b^{2}n}-\frac{2}{{c_{1}^{3}}b^{3}}-\frac{1}{n^{2}}\right)s\\ && \left.+\left( \frac{n}{{c_{1}^{2}}b^{2}}+\frac{3}{n}-\frac{3}{c_{1}b}+\frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}-\frac{c_{1}b}{n^{2}}\right)\right]\\ &=&-\frac{1}{{c_{1}^{4}}b^{4}}\left( \frac{{c_{1}^{4}}b^{4}}{4}-\frac{{c_{1}^{3}}b^{3}}{2}+\frac{{c_{1}^{2}}b^{2}}{4}\right)+\left( \frac{n}{{c_{1}^{4}}b^{4}}+\frac{1}{{c_{1}^{3}}b^{3}}+\frac{1}{{c_{1}^{4}}b^{4}}-\frac{2}{{c_{1}^{2}}b^{2}n}\right)\left( \frac{{c_{1}^{3}}b^{3}}{3}-\frac{{c_{1}^{2}}b^{2}}{2}+\frac{c_{1}b}{6}\right)\\ && +\left( \frac{3}{b^{2}}-\frac{2n}{b^{3}}+\frac{2}{b^{2}n}-\frac{2}{b^{3}}-\frac{1}{n^{2}}\right)\left( \frac{b^{2}}{2}-\frac{b}{2}\right)\\ && +\left( \frac{n}{{c_{1}^{2}}b^{2}}+\frac{3}{n}-\frac{3}{c_{1}b}+\frac{1}{{c_{1}^{2}}b^{2}}+\frac{1}{n^{2}}-\frac{2}{c_{1}bn}-\frac{c_{1}b}{n^{2}}\right)\left( c_{1}b-1\right)\\ &=&\frac{n}{{c_{1}^{4}}b^{4}}\cdot\frac{{c_{1}^{3}}b^{3}}{3}-\frac{2n}{{c_{1}^{3}}b^{3}}\cdot\frac{{c_{1}^{2}}b^{2}}{2}+\frac{n}{{c_{1}^{2}}b^{2}}\cdot c_{1}b\\ &=&\frac{1}{3}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$
(16)

Plug Eqs. 16 in 15

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{2}]&={\Sigma}_{ij}^{2}\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\left[\frac{1}{3}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)\right]. \end{array} $$
(17)

Similarly as E[a2], we calculate E[a3] by first calculating

$$ \mathrm{E}\left[ \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}^{(i)}_{l+s}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}^{(j)}_{l+s}(c_{1}b)-\bar{C}^{(j)} \right) \right], $$

for s = c1b,..., (nc1b). We will show that

$$ \left[ \begin{array}{l} \bar{C}_{l}(b)-\bar{C} \\ \bar{C}_{l+s}(b)-\bar{C} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-b}{bn}\right){\Sigma}& -\frac{1}{n}{\Sigma}\\ -\frac{1}{n}{\Sigma}& \left( \frac{n-b}{bn}\right){\Sigma} \end{array}\right]\right). $$
(18)

Continuing as in Eq. 12

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C},\ \bar{C}_{l+s}(c_{1}b)-\bar{C} \right)=L\cdot\left( -\frac{1}{n}\right)I_{p}\cdot L^{T}=-\frac{1}{n}\cdot{\Sigma}. $$
(19)

The joint distribution in Eq. 18 follows from Eqs. 19 and 9. Denote \(Z_{1}=\bar {C}_{l}(c_{1}b)^{(i)}-\bar {C}^{(i)}\), \(Z_{2}=\bar {C}_{l}(c_{1}b)^{(j)}-\bar {C}^{(j)}\), \(Z_{3}=\bar {C}_{l+s}(c_{1}b)^{(i)}-\bar {C}^{(i)}\), \(Z_{4}=\bar {C}_{l+s}(c_{1}b)^{(j)}-\bar {C}^{(j)}\). By Proposition 2,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{3}]&=\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] = \sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}{\Sigma}_{ij}^{2} + \frac{{\Sigma}_{ii}{\Sigma}_{jj}}{n^{2}} + \frac{{\Sigma}_{ij}^{2}}{n^{2}}\right]. \end{array} $$
(20)

Notice

$$ \begin{array}{@{}rcl@{}} &&\sum\limits_{s=c_{1}b}^{n-c_{1}b}\sum\limits_{l=1}^{n-c_{1}b+1-s}\frac{1}{n^{2}} \\ &=&\frac{1}{n^{2}}\cdot\sum\limits_{s=c_{1}b}^{n-c_{1}b}(n-c_{1}b-s+1)\\ &=&-\frac{1}{n^{2}}\left( \frac{n^{2}}{2}-c_{1}bn+\frac{n}{2}\right)+\left( \frac{1}{n}-\frac{c_{1}b}{n^{2}}+\frac{1}{n^{2}}\right)(n-2c_{1}b+1)\\ &=&o\left( \frac{n}{b}\right). \end{array} $$
(21)

Plug Eqs. 21 in 20,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{3}]&={\Sigma}_{ij}^{2}\sum\limits_{s=b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+o\left( \frac{n}{b}\right). \end{array} $$
(22)

Plug Eqs. 1117 and 22 in 8,

$$ \begin{array}{@{}rcl@{}} A_{2}& =& \frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot \left( \mathrm{E} [a_{1}] + 2\mathrm{E} [a_{2}] + 2\mathrm{E} [a_{3}] \right)\\ & = & \frac{c_{1}b^{2}}{n^{2}}\cdot\Bigg[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)\\ && + {\Sigma}_{ij}^{2}\left( \sum\limits_{l=0}^{n-c_{1}b}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+2\sum\limits_{s=1}^{c_{1}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}+2\sum\limits_{s=c_{1}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}\right)\Bigg]\\ &=&\frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right)+{\Sigma}_{ij}^{2}\left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}(n-c_{1}b+1)^{2}\right]\\ &= & \frac{{c_{1}^{2}}b^{2}}{n^{2}}\cdot\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{n}{c_{1}b}+o\left( \frac{n}{c_{1}b}\right)+{\Sigma}_{ij}^{2}\left( \frac{n^{2}}{{c_{1}^{2}}b^{2}}-\frac{4n}{c_{1}b}\right)\right]\\ &=&\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{c_{1}b}{n}+{\Sigma}_{ij}^{2}-4{\Sigma}_{ij}^{2}\cdot\frac{c_{1}b}{n}\right]+o\left( \frac{b}{n}\right). \end{array} $$

That proves the first part of the lemma. We now move on to term A3. Let

$$ OL^{(i)} = \left( \bar{C}_{p}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{q}^{(i)}(c_{2}b)-\bar{C}^{(i)} \right) \text{ and } OL^{(j)} = \left( \bar{C}_{p}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right) \left( \bar{C}_{q}^{(j)}(c_{2}b)-\bar{C}^{(j)} \right), $$

for p, q satisfying qp and q + c1bp + c2b. Also, define

$$ a_{4} = \sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}(c_{1}b)^{(i)}-\bar{C}^{(i)})(\bar{C}_{l}(c_{1}b)^{(j)}-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(j)}), $$
$$ a_{5} = \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}(\bar{C}_{l}(c_{1}b)^{(i)}-\bar{C}^{(i)})(\bar{C}_{l}(c_{1}b)^{(j)}-\bar{C}^{(j)})(\bar{C}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C}^{(j)}). $$

We have

$$ \begin{array}{@{}rcl@{}} A_{3}&& \\ &=&- \frac{c_{1}c_{2}b^{2}}{n^{2}}\left[\sum\limits_{l=0}^{n-c_{1}b} \left( \bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)} \right)\right] \left[\sum\limits_{l=0}^{n-c_{2}b} \left( \bar{C}_{l}^{(i)}(c_{2}b)-\bar{C}^{(i)} \right) \left( \bar{C}_{l}^{(j)}(c_{2}b)-\bar{C}^{(j)} \right) \right]\\ &=& - \frac{c_{1}c_{2}b^{2}}{n^{2}}\cdot\mathrm{E}[((c_{1}-c_{2})b+1)(n-b+1)\cdot OL^{(i)}OL^{(j)} + 2a_{4} + 2a_{5}] \end{array} $$
(23)

First consider E[OL(i)OL(j)] in Eq. 23. We will show that

$$ \left[ \begin{array}{l} \bar{C}_{p}(c_{1}b)-\bar{C}_{n} \\ \bar{C}_{q}(c_{2}b)-\bar{C}_{n} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}\\ \left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$
(24)

For ij,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b) - \bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{q}(c_{2}b) - \bar{B}^{(j)} \right] = \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b)-\bar{B}^{(i)} \right] \mathrm{E} \left[\bar{B}^{(j)}_{q}(c_{2}b)-\bar{B}^{(j)}\right] = 0. $$
(25)

For i = j and p, q satisfying qp and q + c2bp + c1b, following steps similar to Eq. 13,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{p}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{q}(c_{2}b)-\bar{B}^{(i)} \right] = \frac{n - c_{1}b}{nc_{1}b}. $$
(26)

By Eqs. 25 and 26

$$ \text{Cov} \left( \bar{C}_{p}(c_{1}b)-\bar{C}_{n},\ \bar{C}_{q}(c_{2}b)-\bar{C}_{n} \right)= \frac{n-c_{1}b}{c_{1}bn}\cdot{\Sigma}. $$
(27)

Equation 27 yields the joint distribution in Eq. 24. Denote \(Z_{1}=\bar {C}_{p}(c_{1}b)^{(i)}-\bar {C}^{(i)}\), \(Z_{2}=\bar {C}_{p}(c_{1}b)^{(j)}-\bar {C}^{(j)}\), \(Z_{3}=\bar {C}_{q}(c_{2}b)^{(i)}-\bar {C}^{(i)}\), \(Z_{4}=\bar {C}_{q}(c_{2}b)^{(j)}-\bar {C}^{(j)}\). Then

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[((c_{1}-c_{2})b+1)(n-c_{1}b+1)OL^{(i)}OL^{(j)} \right] \\ & =& ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \mathrm{E}[Z_{1}Z_{2}Z_{3}Z_{4}] \\ &=&((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2}({\Sigma}_{ij}^{2}+{\Sigma}_{ii}{\Sigma}_{jj})\\ && +((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}. \end{array} $$
(28)

Simplifying further,

$$ \begin{array}{@{}rcl@{}} ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)^{2} &=(c_{1}-c_{2})\frac{n}{{c_{1}^{2}}b}+o\left( \frac{n}{b}\right) , \text{ and} \end{array} $$
(29)

and

$$ \begin{array}{@{}rcl@{}} ((c_{1}-c_{2})b+1)(n-c_{1}b+1)\cdot \left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) &=\frac{c_{1}-c_{2}}{c_{1}c_{2}}\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$
(30)

Plug Eqs. 29 and 30 in 28,

$$ \begin{array}{@{}rcl@{}} &&\mathrm{E} \left[((c_{1}-c_{2})b+1)(n-c_{1}b+1)OL^{(i)}OL^{(j)} \right]\\ &=&\left[\frac{(c_{1}-c_{2})}{{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+\frac{c_{1}-c_{2}}{c_{1}c_{2}}{\Sigma}_{ij}^{2}\right]\cdot\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$
(31)

We calculate E[a4] by showing that

$$ \left[ \begin{array}{l} \bar{C}_{l}(c_{1}b) - \bar{C} \\ \bar{C}_{l+(c_{1}-c_{2})b+s}(c_{2}b) - \bar{C} \end{array} \right] \!\sim\! N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& \left( \frac{1}{c_{1}b}-\frac{1}{n}-\frac{s}{{c_{2}^{2}}b^{2}}\right){\Sigma}\\ \left( \frac{1}{c_{1}b}-\frac{1}{n} - \frac{s}{c_{2}b^{2}}\right){\Sigma}& \qquad\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$
(32)

All we need to obtain is the covariance matrix. Continuing as before in Eq. 12, for ij,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] =0. $$
(33)

For i = j, we need to calculate \(\mathrm {E}[\bar {B}^{(i)}_{l}(c_{1}b)-\bar {B}^{(i)}][\bar {B}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{1}b)-\bar {B}^{(i)}]\) for s = 1,..., (c2b − 1). Continuing as before in Eq. 13,

$$ \begin{array}{@{}rcl@{}} &\mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(i)}_{l+(c_{1}-c_{2})b+s}(c_{1}b)-\bar{B}^{(i)} \right]= \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}. \end{array} $$
(34)

By Eqs. 33 and 34,

$$ \text{Cov} \left( \bar{C}_{l}(b)-\bar{C},\ \bar{C}_{l+s}(b)-\bar{C} \right) = \left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)\cdot{\Sigma}. $$
(35)

Therefore (32) follows from Eqs. 9 and 35. Again denote \(Z_{1}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}\), \(Z_{2}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}\), \(Z_{3}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar {C}^{(i)}\), \(Z_{4}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar {C}^{(j)}\),

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{4}]\!\!\!&&\\ &=&\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}E[Z_{1}Z_{2}Z_{3}Z_{4}]\\ &=&\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}+\left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)^{2}{\Sigma}_{ii}{\Sigma}_{jj}+\left( \frac{1}{c_{1}b} - \frac{1}{n} - \frac{s}{c_{1}c_{2}b^{2}}\right)^{2}{\Sigma}_{ij}^{2}\right]. \end{array} $$
(36)

It is easy to show that

$$ \begin{array}{@{}rcl@{}} &\sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) =\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$
(37)

Following the steps in Eq. 16, we get

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=1}^{c_{2}b-1}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{1}{c_{1}b}-\frac{1}{n}-\frac{s}{c_{1}c_{2}b^{2}}\right)^{2} & =\frac{c_{2}}{{c_{1}^{2}}3}\cdot\frac{n}{b}+o\left( \frac{n}{b}\right). \end{array} $$
(38)

Plug Eqs. 37 and 38 in 36

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{4}]&=\frac{c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\frac{n}{b}+{\Sigma}_{ij}^{2}\frac{n}{c_{1}b}+o\left( \frac{n}{b}\right). \end{array} $$
(39)

Finally, we calculate E[a5]. For s = c2b,..., (nc1b), we will show that

$$ \left[ \begin{array}{l} \bar{C}_{l}(c_{1}b)-\bar{C} \\ \bar{C}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{C} \end{array} \right] \sim N\left( \begin{bmatrix} 0\\ 0\end{bmatrix},\ \left[\begin{array}{ll}\left( \frac{n-c_{1}b}{c_{1}bn}\right){\Sigma}& -\frac{1}{n}{\Sigma}\\ -\frac{1}{n}{\Sigma}& \left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma} \end{array}\right]\right). $$
(40)

For ij,

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] = 0. $$
(41)

For i = j, we need to calculate \(\mathrm {E}[\bar {B}^{(i)}_{l}(b)-\bar {B}^{(i)}][\bar {B}^{(i)}_{l+(1-c)b+s}(cb)-\bar {B}^{(i)}]\) for s = cb,..., (nb).. Similar to the steps in Eq. 13, we get

$$ \mathrm{E} \left[\bar{B}^{(i)}_{l}(c_{1}b)-\bar{B}^{(i)} \right] \left[\bar{B}^{(j)}_{l+(c_{1}-c_{2})b+s}(c_{2}b)-\bar{B}^{(j)} \right] = -\frac{1}{n} . $$
(42)

By Eqs. 41 and 42,

$$ \text{Cov} \left( \bar{C}_{l}(c_{1}b)-\bar{C},\ \bar{C}_{l+(c_{1} - c_{2})b + s}(c_{2}b)-\bar{C} \right)= -\frac{1}{n}\cdot{\Sigma}. $$
(43)

Therefore (40) follows from Eqs. 9 and 43. Again denote \(Z_{1}=\bar {C}_{l}^{(i)}(c_{1}b)-\bar {C}^{(i)}\), \(Z_{2}=\bar {C}_{l}^{(j)}(c_{1}b)-\bar {C}^{(j)}\), \(Z_{3}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar {C}^{(i)}\), \(Z_{4}=\bar {C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar {C}^{(j)}\)

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{5}]\!\!\!&&\\ &=&\sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s} \mathrm{E} \left[(\bar{C}_{l}^{(i)}(c_{1}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{1}b)-\bar{C}^{(j)})(\bar{C}_{l+(c_{1}-c_{2})b+s}^{(i)}(c_{2}b)-\bar{C}^{(i)})(\bar{C}_{l+(c_{1}-c_{2})b+s}^{(j)}(c_{2}b)-\bar{C}^{(j)}) \right]\\ &=&\sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left[\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right){\Sigma}_{ij}^{2}+\frac{1}{n^{2}}{\Sigma}_{ii}{\Sigma}_{jj}+\frac{1}{n^{2}}{\Sigma}_{ij}^{2}\right]. \end{array} $$
(44)

Straightforward algebra shows,

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\left( \frac{n-c_{1}b}{c_{1}bn}\right)\left( \frac{n-c_{2}b}{c_{2}bn}\right) &=\frac{1}{2c_{1}c_{2}}\frac{n^{2}}{b^{2}}-\left( \frac{3}{2c_{1}}+\frac{3}{2c_{2}}\right)\frac{n}{b}+o\left( \frac{n}{b}\right), \end{array} $$
(45)

and

$$ \begin{array}{@{}rcl@{}} \sum\limits_{s=c_{2}b}^{n-c_{1}b}\sum\limits_{l=0}^{n-c_{1}b-s}\frac{1}{n^{2}} &=o\left( \frac{n}{b}\right). \end{array} $$
(46)

Plug Eqs. 45 and 46 in 44,

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[a_{5}]&={\Sigma}_{ij}^{2} \left( \frac{1}{2c_{1}c_{2}}\frac{n^{2}}{b^{2}}-\left( \frac{3}{2c_{1}}+\frac{3}{2c_{2}}\right)\frac{n}{b}\right). \end{array} $$
(47)

Replace Eqs. 3139 and 47 in 23

$$ \begin{array}{@{}rcl@{}} A_{3}& = & \mathrm{E}\left[-\frac{c_{1}c_{2}b^{2}}{n^{2}}\left[\sum\limits_{l=0}^{n-b}(\bar{C}_{l}^{(i)}(b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(b)-\bar{C}^{(j)})\right]\left[\sum\limits_{l=0}^{n-cb}(\bar{C}_{l}^{(i)}(cb)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(cb)-\bar{C}^{(j)})\right]\right]\\ &=&- \frac{c_{1}c_{2}b^{2}}{n^{2}}\cdot \mathrm{E} \left[((c_{1}-c_{2})b+1)(n-b+1)\cdot OL^{(i)}OL^{(j)} +2a_{4} + 2a_{5} \right] \\ &=&-c_{1}c_{2}\frac{b}{n}\cdot \left[\frac{(c_{1}-c_{2})}{{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+\frac{c_{1}-c_{2}}{c_{1}c_{2}}{\Sigma}_{ij}^{2}\right.\\ &&+ \frac{2c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})+{\Sigma}_{ij}^{2}\frac{2}{c_{1}} + \left. {\Sigma}_{ij}^{2} \left( \frac{1}{c_{1}c_{2}}\frac{n}{b}-\left( \frac{3}{c_{1}}+\frac{3}{c_{2}}\right)\right) \right]+o\left( \frac{b}{n}\right)\\ &=&-c_{1}c_{2} \frac{b}{n} \left[\frac{3c_{1} - c_{2}}{3{c_{1}^{2}}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot - 2\left( \frac{c_{1}+c_{2}}{c_{1}c_{2}}\right)\cdot{\Sigma}_{ij}^{2} +\frac{1}{c_{1}c_{2}} \frac{n}{b}\cdot{\Sigma}_{ij}^{2} \right]+o\left( \frac{b}{n}\right) \\ &=& \frac{(c_{2} - 3c_{1})c_{2}}{3c_{1}}({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})\cdot \frac{b}{n} + 2\left( c_{1}+c_{2}\right)\cdot{\Sigma}_{ij}^{2}\cdot\frac{b}{n} - {\Sigma}_{ij}^{2} +o\left( \frac{b}{n}\right). \end{array} $$

With elements \(\tilde {\Sigma }_{wL, ij}\), define

$$ \tilde{\Sigma}_{wL}=\frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{n}(k) \left[\bar{C}_{l}(k)-\bar{C}\right]\left[\bar{C}_{l}(k)-\bar{C}\right]^{T}. $$

Lemma 3

If Assumption 1 holds and \({\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2}\leq O\left (b^{-2}\right )\) then

$$ Var[\tilde{\Sigma}_{wL,ij}]=({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\cdot\frac{1}{n}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u}{\Delta}_{2}w_{t+u}\left( \frac{2}{3}u^{3}+u^{2}t\right)\cdot\frac{1}{n}\bigg]+o\left( \frac{b}{n}\right). $$

Proof

Note \(({\Delta }_{2}w_{k})^{2}\leq {\sum }_{k=1}^{b}({\Delta }_{2}w_{k})^{2} = O\left (b^{-2}\right ),\) hence ak = b ⋅Δ2wk = O(1). Consider

$$ \tilde{\Sigma}_{wL,ij} = \frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{n}(k) \left[\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)} \right] \left[\bar{C}^{(j)}_{l}(k)-\bar{C}^{(j)} \right]. $$

Let ck = k/b for k = 1,...,b, also denote ak = b ⋅Δ2wk for simplicity. Then,

$$ \begin{array}{@{}rcl@{}} \tilde{\Sigma}_{wL,ij} &=\sum\limits_{k=1}^{b}c_{k}a_{k}\left( \frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b} \left[\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)} \right] \left[\bar{C}^{(j)}_{l}(k)-\bar{C}^{(j)} \right]\right). \end{array} $$

Define \(A_{1,ij}^{(k)}\) and \(A_{2,ij}^{(ut)}\) below and apply Lemma 2,

$$ \begin{array}{@{}rcl@{}} A_{1,ij}^{(k)} & =& \mathrm{E}\left[\frac{(c_{k}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{k=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)})\right)^{2}\right]\\ &=&\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{c_{k}b}{n}+{\Sigma}_{ij}^{2}+o\left( \frac{b}{n}\right), \end{array} $$
(48)

and

$$ \begin{array}{@{}rcl@{}} A_{2,ij}^{(ut)}& = & \mathrm{E}\left[\frac{(c_{u+t}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\right.\\ && \left.\cdot\left( \sum\limits_{q=0}^{n-c_{u+t}b}(\bar{C}^{(i)}_{q}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}^{(j)}_{q}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\frac{b}{n}+\frac{c_{u+t}}{c_{u}}{\Sigma}_{ij}^{2}+o\left( \frac{b}{n}\right). \end{array} $$
(49)

To calculate \(\text {Var}[\tilde {\Sigma }_{wL,ij}]\), we will calculate \(\mathrm {E}[\tilde {\Sigma }_{wL,ij}^{2}]\) and \((\mathrm {E}[\tilde {\Sigma }_{wL,ij}])^{2}\). Plugging Eqs. 48 and 49 in the expression of \(\mathrm {E}[\tilde {\Sigma }_{wL,ij}^{2}]\) results in

$$ \begin{array}{@{}rcl@{}} \mathrm{E}[\tilde{\Sigma}_{wL,ij}^{2}]\!\!&&\\ & =& \mathrm{E}\left[\sum\limits_{k=1}^{b}\left( c_{k}a_{k}\cdot\left[\frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(k)-\bar{C}^{(j)})\right]\right)^{2}\right.\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}c_{t+u}^{2}a_{t+u}\cdot\frac{b^{2}}{n^{2}}\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\\ && \left.\cdot\left( \sum\limits_{q=0}^{n-c_{t+u}b}(\bar{C}_{q}^{(i)}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}_{q}^{(j)}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\cdot E\left[\frac{(c_{k}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{l=0}^{n-c_{k}b}(\bar{C}_{l}^{(i)}(k)-\bar{C}^{(i)})(\bar{C}_{l}^{(j)}(k)-\bar{C}^{(j)})\right)^{2}\right]\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{t+u}\cdot E\left[\frac{(c_{u+t}b)^{2}}{n^{2}}\cdot\left( \sum\limits_{p=0}^{n-c_{t}b}(\bar{C}_{p}^{(i)}(c_{t}b)-\bar{C}^{(i)})(\bar{C}_{p}^{(j)}(c_{t}b)-\bar{C}^{(j)})\right)\right.\\ && \cdot\left.\left( \sum\limits_{q=0}^{n-c_{t+u}b}(\bar{C}_{q}^{(i)}(c_{t+u}b)-\bar{C}^{(i)})(\bar{C}_{q}^{(j)}(c_{t+u}b)-\bar{C}^{(j)})\right)\right]\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}A_{1,ij}^{(k)}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}A_{2,ij}^{(ut)}\\ &=&o\left( \frac{b}{n}\right)+\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\left[\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{c_{k}b}{n}+{\Sigma}_{ij}^{2}\right]\\ &&+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\cdot\frac{b}{n}+\frac{c_{u+t}}{c_{u}}{\Sigma}_{ij}^{2}\right]\\ &=&\left[\sum\limits_{k=1}^{b}{c_{k}^{2}}{a_{k}^{2}}\cdot{\Sigma}_{ij}^{2}+2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\frac{c_{u+t}}{c_{u}}\cdot{\Sigma}_{ij}^{2}\right]+\left[\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\left( \frac{2}{3}[{\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2}]-4{\Sigma}_{ij}^{2}\right)\cdot\frac{b}{n}\right.\\ && +\left.2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\frac{b}{n}\right]+o\left( \frac{b}{n}\right)\\ &=&\left( \sum\limits_{k=1}^{b}a_{k}c_{k} \right)^{2}{\Sigma}_{ij}^{2}+\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\left( \frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right)\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\cdot\frac{b}{n}+o\left( \frac{b}{n}\right). \end{array} $$
(50)

By Eq. 9,

$$ \mathrm{E}[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)})]=\frac{n-c_{k}b}{c_{k}bn}{\Sigma}_{ij}. $$
(51)

Using Eq. 51,

$$ \begin{array}{@{}rcl@{}} (\mathrm{E}[\tilde{\Sigma}_{wL,ij}])^{2}) \!\!&&\\ &=&\left( \frac{1}{n}\sum\limits_{k=1}^{b}\sum\limits_{l=0}^{n-k}k^{2}{\Delta}_{2}w_{k} \mathrm{E} \left[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)}) \right]\right)^{2}\\ &=&\left( \sum\limits_{k=1}^{b}c_{k}a_{k}\left[\frac{c_{k}b}{n}\sum\limits_{l=0}^{n-c_{k}b} \mathrm{E} \left[(C_{l}^{(i)}(c_{k}b)-\bar{C}^{(i)})(C_{l}^{(j)}(c_{k}b)-\bar{C}^{(j)}) \right]\right]\right)^{2}\\ &=&\left( \sum\limits_{k=1}^{b}c_{k}a_{k}\left[\frac{c_{k}b}{n}\cdot(n-c_{k}b+1)\cdot\frac{n-c_{k}b}{c_{k}bn}\cdot{\Sigma}_{ij}\right]\right)^{2}\\ &=&{\Sigma}_{ij}^{2}\left[ \left( \sum\limits_{k=1}^{b}a_{k}c_{k} \right)^{2}-\sum\limits_{k=1}^{b}4{a_{k}^{2}}{c_{k}^{3}}\cdot\frac{b}{n}-2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}a_{u}a_{u+t}(2{c_{u}^{2}}c_{u+t}+2c_{u}c_{u+t}^{2})\cdot\frac{b}{n}\right]+o\left( \frac{b}{n}\right). \end{array} $$
(52)

Combine Eqs. 50 and 52,

$$ \begin{array}{@{}rcl@{}} \text{Var}[\tilde{\Sigma}_{wL,ij}] &=& \mathrm{E}[\tilde{\Sigma}_{wL,ij}^{2}] - (\mathrm{E}[\tilde{\Sigma}_{wL,ij}])^{2}\\ &=&\sum\limits_{k=1}^{b}{c_{k}^{3}}{a_{k}^{2}}\Bigg(\left[\frac{2}{3}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-4{\Sigma}_{ij}^{2}\right]+4{\Sigma}_{ij}^{2}\Bigg)\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}\Bigg({c_{u}^{2}}a_{u}a_{u+t}\left[\left( c_{u+t}-\frac{c_{u}}{3}\right)({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})-\left( 2c_{u+t}+\frac{2c_{u+t}^{2}}{c_{u}}\right){\Sigma}_{ij}^{2}\right]\\ && +a_{u}a_{u+t}(2{c_{u}^{2}}c_{u+t}+2c_{u}c_{u+t}^{2}){\Sigma}_{ij}^{2}\Bigg)\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&\sum\limits_{k=1}^{b}\frac{2}{3} {c_{k}^{3}}{a_{k}^{2}}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+2\sum\limits_{t=1}^{b}\sum\limits_{u=1}^{b-t}\left( {c_{u}^{2}}c_{u+t}-\frac{1}{3}{c_{u}^{3}}\right)a_{u}a_{u+t}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&\sum\limits_{k=1}^{b}\frac{2}{3}\left( \frac{k}{b}\right)^{3}(b{\Delta}_{2}w_{k})^{2}({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}\\ && +2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}\left[\left( \left( \frac{u}{b}\right)^{2}\frac{u+t}{b}-\frac{1}{3}\left( \frac{u}{b}\right)^{3}\right)b{\Delta}_{2}w_{u}\cdot b{\Delta}_{2}w_{u+t}\right]({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)\\ &=&({\Sigma}_{ii}{\Sigma}_{jj}+{\Sigma}_{ij}^{2})\cdot\bigg[\frac{2}{3}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\cdot\frac{1}{n} + 2\sum\limits_{t=1}^{b-1}\sum\limits_{u=1}^{b-t}{\Delta}_{2}w_{u}\cdot{\Delta}_{2}w_{t+u}\!\cdot\!\left( \frac{2}{3}u^{3} + u^{2}t\right)\!\cdot\!\frac{1}{n}\bigg] + o\left( \frac{b}{n}\right). \end{array} $$

Lemma 4

(Lemma 14 Vats et al. 2018) Suppose (5) holds for f = g and Assumption 1 hold. If, as \(n\to \infty \),

$$ b\psi(n)^{2} \log n\left( \sum\limits_{k=1}^{b}|{\Delta}_{2}w_{n}(k)|\right)^{2}\rightarrow 0 \quad \text{ and } \quad \psi(n)^{2}\sum\limits_{k=1}^{b}|{\Delta}_{2}w_{n}(k)|\rightarrow 0, $$

then \(\hat {\Sigma }_{w}\rightarrow \tilde {\Sigma }_{wL}\) as \(n \to \infty \) w.p. 1.

Lemma 5

Suppose (5) holds for f = g and f = g2 (where the square is element-wise) such that \(\mathrm {E}_{F}D^{4} < \infty \) and Assumption 1 holds. Further, suppose \(\psi ^{2}(n)b^{-1}\log n \rightarrow 0\), then

$$ \mathrm{E} \left[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij} \right]^{2}\rightarrow\ 0\ \text{as}\ n\rightarrow\infty. $$

Proof

An observation of Lemma B.4 of Jones et al. (2006), Lemmas 12, 13 and 14 of Flegal and Jones (2010) and Lemma 5 of Liu and Flegal (2018) show that Lemma 5 hold. □

Appendix C: Proof of Theorem 3

Define

$$ \eta= \text{Var}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]+2\mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]. $$

We show \(\eta \rightarrow 0\) as \(n\rightarrow \infty \). From Lemma 5, Cauchy-Schwarz inequality, and Var[X] ≤EX2.

$$ \begin{array}{@{}rcl@{}} |\eta|&=&|\text{Var}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}] + 2\mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]|\\ &&\leq\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2}+2\sqrt{\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2}\cdot\mathrm{E}[\tilde{\Sigma}_{w,L,ij} - \mathrm{E}\tilde{\Sigma}_{wL, ij}]^{2}}\\ & =& \mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{w,L,ij}]^{2} + 2(\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2})^{1/2}\cdot (\text{Var}[\tilde{\Sigma}_{wL,ij}])^{1/2} \end{array} $$

By the conditions of Lemma 3,

$$ \frac{1}{n}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}k^{3}\leq\frac{b^{3}}{n}\sum\limits_{k=1}^{b}({\Delta}_{2}w_{k})^{2}\leq O \left( \frac{b}{n} \right). $$

Hence we have

$$ \text{Var}[\tilde{\Sigma}_{wL,ij}]=(({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S \frac{b}{n}+o(1))\cdot\frac{b}{n}. $$

By Lemma 5, \(\mathrm {E}[\hat {\Sigma }_{w,ij}-\tilde {\Sigma }_{wL,ij}]^{2}=o(1)\), therefore

$$ \begin{array}{@{}rcl@{}} |\eta|&\leq&\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{,L,ij}]^{2}+2(\mathrm{E}[\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}]^{2})^{1/2}\cdot (\text{Var}[\tilde{\Sigma}_{wL,ij}])^{1/2}\\ &=&o(1)+2\sqrt{o(1)\cdot \left[(({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S+o(1))\cdot\frac{b}{n} \right]} = o(1). \end{array} $$
(53)

Since \(b/n\rightarrow 0\) as \(n\rightarrow \infty \), plug in Eq. 53

$$ \begin{array}{@{}rcl@{}} \text{Var}[\hat{\Sigma}_{w,ij}]& =& \mathrm{E}[\hat{\Sigma}_{w,ij} - \mathrm{E}\hat{\Sigma}_{w,ij}]^{2}\\ & =& \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})+(\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}) - (\mathrm{E}\hat{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij})]^{2}\\ &&+ 2 \mathrm{E}[[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij}) - \mathrm{E}(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})]\cdot [\tilde{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{w,ij}]]\\ & =& \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{w,L,ij}) - \mathrm{E}(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})]^{2} + \mathrm{E}[\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}]^{2}\\ && + 2 \mathrm{E}[(\hat{\Sigma}_{w,ij}-\tilde{\Sigma}_{wL,ij})\cdot (\tilde{\Sigma}_{w,ij} - \mathrm{E}\tilde{\Sigma}_{w,ij})]\\ & =& \mathrm{E}[\tilde{\Sigma}_{wL,ij} - \mathrm{E}\tilde{\Sigma}_{wL,ij}]^{2}+\eta\\ &=&({\Sigma}_{ii}{\Sigma}_{jj} + {\Sigma}_{ij}^{2})S\cdot\frac{b}{n}+o\left( \frac{b}{n}\right)+o(1). \end{array} $$

Appendix D: Performance Without a Pilot Run

Table 3 Coverage probabilities for 90% confidence regions over 1000 replications with no pilot runs for Bayesian logistic regression example
Table 4 Coverage probabilities for 90% confidence regions over 1000 replications with no pilot runs for the Bayesian dynamic space-time example

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Vats, D. & Flegal, J.M. Batch Size Selection for Variance Estimators in MCMC. Methodol Comput Appl Probab (2021). https://doi.org/10.1007/s11009-020-09841-7

Download citation

Keywords

  • Asymptotic variance estimation
  • Batch size
  • Markov chain Monte Carlo
  • Overlapping batch means

Mathematics Subject Classification (2010)

  • 60J22
  • 62F15