Robust distributed multicategory angle-based classification for massive data

Abstract

Multicategory classification problems are frequently encountered in practice. Considering that the massive data sets are increasingly common and often stored locally, we first provide a distributed estimation in the multicategory angle-based classification framework and obtain its excess risk under general conditions. Further, under varied robustness settings, we develop two robust distributed algorithms to provide robust estimations of the multicategory classification. The first robust distributed algorithm takes advantage of median-of-means (MOM) and is designed by the MOM-based gradient estimation. The second robust distributed algorithm is implemented by constructing the weighted-based gradient estimation. The theoretical guarantees of our algorithms are established via the non-asymptotic error bounds of the iterative estimations. Some numerical simulations demonstrate that our methods can effectively reduce the impact of outliers.

Appendix A proof of theoretical results

1.1 A.1 Proof of Theorem 1

Given a compact convex set ${\mathcal {B}}=\{\varvec{\beta }:\Vert \varvec{\beta }\Vert _2^2\le c_0\}$, by duality with convex $\ell$ and $\Vert \varvec{\beta }\Vert _2^2$, minimizing the optimization model (5) is equivalent to solving the optimization problem

$$\begin{aligned} \min _{\varvec{\beta }}\frac{1}{\vert {\mathcal {D}}_m\vert } \sum _{i=1}^{\vert {\mathcal {D}}_m\vert } \ell (({\varvec{x}}_i\otimes {\varvec{W}}_{y_i})^{\top }\varvec{\beta }) \end{aligned}$$

(A1)

subject to $\Vert \varvec{\beta }\Vert _2^2\le c_0$. Here, $c_0$ is a positive constant with respect to $\lambda _m>0$ for $m=1,\ldots ,M$. That is to say, it is sufficient to consider the optimization problem (A1) on the compact convex set ${\mathcal {B}}\in {\mathbb {R}}^{(k-1)p}$.

According to the condition of

$$\begin{aligned} \frac{\gamma _1}{2}\Vert \varvec{\beta }-\varvec{\beta }^\prime \Vert _2^2\le {\mathcal {R}}(\varvec{\beta })-{\mathcal {R}}(\varvec{\beta }^\prime ) -\langle \nabla {\mathcal {R}}(\varvec{\beta }^\prime ),\varvec{\beta } -\varvec{\beta }^\prime \rangle \end{aligned}$$

in Assumption 3, we have

$$\begin{aligned} \nabla ^2{\mathcal {R}}(\varvec{\beta })\succeq \gamma _1I, \end{aligned}$$

where I denotes the $(k-1)p\times (k-1)p$ identity matrix.

Based on the above argument under Assumption 1-3, according to Corollary 2 in Zhang et al. (2013), we can directly obtain that

$$\begin{aligned} E\big [\Vert \varvec{{\widehat{\beta }}}-\varvec{\beta }^{*}\Vert _2^2\big ]\le C_0\bigg (\frac{1}{nM\gamma _1^2}+\frac{1}{n^2}\Big (\frac{\log (k-1)p}{\gamma _1^4}+\frac{1}{\gamma _1^6}\Big ) +\frac{1}{n^2M}+\frac{1}{n^3}\bigg ), \nonumber \\ \end{aligned}$$

(A2)

where $C_0$ is a numerical constant.

By the Lipschitz continuity of the loss $\ell$ in Assumption 1, we get that

$$\begin{aligned} {\mathcal {R}}(\varvec{{\widehat{\beta }}})-{\mathcal {R}}(\varvec{\beta }^*) =E[L_0({\varvec{X}},Y)\Vert \varvec{{\widehat{\beta }}}-\varvec{\beta }^*\Vert _2]. \end{aligned}$$

(A3)

Further, applying the Cauchy-Schwarz inequality yields that

$$\begin{aligned} E[L_0({\varvec{X}},Y)\Vert \varvec{{\widehat{\beta }}}-\varvec{\beta }^*\Vert _2] \le E[L_0^2({\varvec{X}},Y)]^{\frac{1}{2}}E[\Vert \varvec{{\widehat{\beta }}}-\varvec{\beta }^*\Vert _2^2]^{\frac{1}{2}}. \end{aligned}$$

(A4)

Combine (A3) and (A4), we have

$$\begin{aligned} {\mathcal {E}}_{\ell }(\varvec{{\widehat{\beta }}}, \varvec{\beta }^{*}) \le L_0E[\Vert \varvec{{\widehat{\beta }}}-\varvec{\beta }^*\Vert _2^2]^{\frac{1}{2}}. \end{aligned}$$

(A5)

Substitute (A2) into (A5), we finally obtain that

$$\begin{aligned} {\mathcal {E}}_{\ell }(\varvec{{\widehat{\beta }}}, \varvec{\beta }^{*}) \le C\bigg (\frac{1}{\sqrt{nM}\gamma _1}+\frac{1}{n}\Big (\frac{\sqrt{\log (k-1)p}}{\gamma _1^2} +\frac{1}{\gamma _1^3}\Big )\bigg ), \end{aligned}$$

where C is a numerical constant.

1.2 A.2 Proof of Proposition 3 and Theorem 4

.

Proof of Proposition 3

According to the nature of the MOM-based gradient defined in (10), for given $\widetilde{\varvec{\beta }}$ with $\Vert \widetilde{\varvec{\beta }}-\varvec{\beta }^0\Vert \le C_0$, we first observe that the event

$$\begin{aligned} \Big \{\vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert >\varepsilon _{m,r}\Big \} \end{aligned}$$

which implies that at least L/2 of $({\varvec{g}}_m^{(l)}(\widetilde{\varvec{\beta }})-\lambda _m\widetilde{\varvec{\beta }})_{r}$ has to be outside $\varepsilon _{m,r}$ distance to $(\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_{r}$. Namely,

$$\begin{aligned}{} & {} \Bigg \{\vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }})) _r\vert>\varepsilon _{m,r}\Big \}\nonumber \\{} & {} \quad \subset \left\{ \sum _{l=1}^L{\varvec{1}}\{\vert ({\varvec{g}}_m^{(l)}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert >\varepsilon _{m,r}\}\ge \frac{L}{2}\right\} . \end{aligned}$$

(A6)

Define ${\mathcal {L}}_m=\{l\in \{1,\ldots ,L\}:{\mathcal {G}}_m^{(l)}\cap {\mathcal {O}}_m=\emptyset \}$, and let $T_{m,r}^{(l)}={\varvec{1}}\{\vert ({\varvec{g}}_m^{(l)}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_{r}\vert >\varepsilon _{m,r}\}$ and $p_{m,r}^{(l)}=E(T_{m,r}^{(l)})$, then the above (A6) implies that

$$\begin{aligned}{} & {} P\Big (\vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert >\varepsilon _{m,r}\Big )\\{} & {} \quad \le P\left( \sum _{l=1}^LT_{m,r}^{(l)}\ge \frac{L}{2}\right) \\{} & {} \quad \le P\left( \sum _{k\in {\mathcal {L}}_m}T_{m,r}^{(l)}+\vert {\mathcal {O}}_m\vert \ge \frac{L}{2}\right) \\{} & {} \quad \le P\left( \sum _{k\in {\mathcal {L}}_m}(T_{m,r}^{(l)}-E(T_{m,r}^{(l)}))\ge \left( \frac{1}{2}-C_1-p_{m,r}^{(l)}\right) L\right) \\{} & {} \quad \le e^{-2L(\frac{1}{2}-C_1-p_{m,r}^{(l)})^2}. \end{aligned}$$

The last inequality is derived by the one-sided Hoeffding’s inequality.

Note that

$$\begin{aligned} p_{m,r}^{(l)}=E(T_{m,r}^{(l)}) =P\Big (\vert ({\varvec{g}}_m^{(l)}(\widetilde{\varvec{\beta }}) -\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert >\varepsilon _{m,r}\Big ) \le \frac{(Cov(\nabla \ell (\widetilde{\varvec{\beta }})))_{rr}L}{n\varepsilon _{m,r}^2} \end{aligned}$$

for some $l\in {\mathcal {L}}$. Choose $\varepsilon _{m,r}=\sqrt{\frac{(2+c)(Cov(\nabla \ell (\widetilde{\varvec{\beta }})))_{rr}L}{n(1-2C_1)}}$ for all $c>0$, we can obtain that

$$\begin{aligned}{} & {} P\Bigg (\vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert >\sqrt{\frac{(2+c)(Cov(\nabla \ell (\widetilde{\varvec{\beta }})))_{rr}L}{n(1-2C_1)}}\Bigg )\\{} & {} \quad \le e^{-2L(\frac{1}{2}-C_1-\frac{1-2C_1}{2+c})^2}. \end{aligned}$$

This means that

$$\begin{aligned} \vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert \le \sqrt{\frac{(2+c)(Cov(\nabla \ell (\widetilde{\varvec{\beta }})))_{rr}L}{n(1-2C_1)}} \end{aligned}$$

with probability at least $1-e^{-\frac{L(1-2C_1)^2c^2}{2(2+c)^2}}$.

Thus, for given $\widetilde{\varvec{\beta }}$ with $\Vert \widetilde{\varvec{\beta }}-\varvec{\beta }^0\Vert \le C_0$, we can get that

$$\begin{aligned}{} & {} \Vert \varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }})\Vert _2\\{} & {} \quad \le \Vert \varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }})\Vert _ 2+\lambda _m\Vert \widetilde{\varvec{\beta }}\Vert _2\\{} & {} \quad \le \sqrt{(k-1)p}\max _{1\le r \le (k-1)p}\vert (\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\widetilde{\varvec{\beta }}) -\lambda _m\widetilde{\varvec{\beta }}-\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert +Cn^{-1/2}\\{} & {} \quad \le \max _{1\le r \le (k-1)p}\sqrt{\frac{(2+c)(k-1)p(Cov(\nabla \ell (\widetilde{\varvec{\beta }})))_{rr}L}{n(1-2C_1)}}+Cn^{-1/2} \end{aligned}$$

with probability at least $1-e^{-\frac{L(1-2C_1)^2c^2}{2(2+c)^2}}$, where C is a numerical constant. $\square$

.

Proof of Theorem 4

Let $\varvec{\delta }^{m,t}=\varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\varvec{\beta }^{m,t}) -\nabla {\mathcal {R}}(\varvec{\beta }^{m,t})\in {\mathbb {R}}^{(k-1)p}$, by the fact of $\nabla {\mathcal {R}}(\varvec{\beta }^{*})=0$ and our update rule of the MOM-based gradient estimation, i.e., $\varvec{\beta }^{m,t+1}=\varvec{\beta }^{m,t} -\eta \varvec{{\widehat{g}}}_m^{\textrm{MOM}}(\varvec{\beta }^{m,t})$, we have

$$\begin{aligned}{} & {} \Vert \varvec{\beta }^{m,t+1}-\varvec{\beta }^{*}\Vert _2\nonumber \\{} & {} \quad =\Vert \varvec{\beta }^{m,t}-\eta \varvec{{\widehat{g}}}_m^ {\textrm{MOM}}(\varvec{\beta }^{m,t})-\varvec{\beta }^{*} +\eta \nabla {\mathcal {R}}(\varvec{\beta }^{*})\Vert _2\nonumber \\{} & {} \quad =\Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*} -\eta [\nabla {\mathcal {R}}(\varvec{\beta }^{m,t}) -(\nabla {\mathcal {R}}(\varvec{\beta }^{*})]-\eta \varvec{\delta }^{m,t}\Vert _2\nonumber \\{} & {} \quad \le \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}-\eta [\nabla {\mathcal {R}}(\varvec{\beta }^{m,t}) -(\nabla {\mathcal {R}}(\varvec{\beta }^{*})]\Vert _2+\eta \Vert \varvec{\delta }^{m,t}\Vert _2. \end{aligned}$$

(A7)

Let ${\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})=\nabla {\mathcal {R}}(\varvec{\beta }^{m,t}) -\nabla {\mathcal {R}}(\varvec{\beta }^{*})$ for brevity and then use Lemma 3.11 introduced in Bubeck (2015), we can directly get that

$$\begin{aligned}{} & {} \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*} -\eta {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})\Vert _2^2\\{} & {} \quad =\Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}\Vert _2^2 +\eta ^2\Vert {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*}) \Vert _2^2-2\eta ({\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*}))^\top (\varvec{\beta } ^{m,t}-\varvec{\beta }^{*})\\{} & {} \quad \le \Vert \varvec{\beta }^{m(t)}-\varvec{\beta }^{*}\Vert _2^2 +\eta ^2\Vert {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})\Vert _2^2 -2\eta \Big [\frac{\gamma _1\gamma _2}{\gamma _1+\gamma _2}\Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}\Vert _2^2\\{} & {} \quad +\frac{1}{\gamma _1+\gamma _2}\Vert {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})\Vert _2^2\Big ]\\{} & {} \quad =\left( 1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}\right) \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}\Vert _2^2 +\left( \eta ^2-\frac{2\eta }{\gamma _1+\gamma _2}\right) \Vert {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})\Vert _2^2. \end{aligned}$$

Choose $\eta =\frac{2}{\sqrt{M}(\gamma _1+\gamma _2)}$ with respect to the number of local machines M as the step size, there is

$$\begin{aligned} \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*} -\eta {\varvec{G}}(\varvec{\beta }^{m,t},\varvec{\beta }^{*})\Vert _2^2 \le \left( 1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}\right) \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}\Vert _2^2. \end{aligned}$$

(A8)

Thus, substitute (A8) into (A7), we have

$$\begin{aligned} \Vert \varvec{\beta }^{m,t+1}-\varvec{\beta }^{*}\Vert _2 \le \sqrt{1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}}\Vert \varvec{\beta }^{m,t} -\varvec{\beta }^{*}\Vert _2+\eta \Vert \varvec{\delta }^{m,t}\Vert _2. \end{aligned}$$

Using Proposition 3 and the condition $Cov(\nabla \ell )$ is finite, for given $\varvec{\beta }^{m,t}$ with $\Vert \varvec{\beta }^{m,t}-\varvec{\beta }^0\Vert \le C_0$, we further have

$$\begin{aligned} \Vert \varvec{\beta }^{m,t+1}-\varvec{\beta }^{*}\Vert _2\le & {} \sqrt{1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}}\Vert \varvec{\beta }^{m,t} -\varvec{\beta }^{*}\Vert _2\nonumber \\{} & {} +C\eta \sqrt{\frac{(2+c)(k-1)pL}{n(1-2C_1)}}+C\eta n^{-1/2}\nonumber \\= & {} \sqrt{1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}}\Vert \varvec{\beta }^{m,t} -\varvec{\beta }^{*}\Vert _2+\frac{2\varepsilon _{_\textrm{MOM}}}{\gamma _1+\gamma _2} \end{aligned}$$

with probability at least $1-e^{-\frac{L(1-2C_1)^2c^2}{2(2+c)^2}}$ for all $c>0$, where

$$\begin{aligned} \varepsilon _{_\textrm{MOM}}=C\sqrt{\frac{(2+c)(k-1)pL}{N(1-2C_1)}}+CN^{-1/2}. \end{aligned}$$

and C is a numerical constant.

Write $\kappa =\sqrt{1-\frac{2\eta \gamma _1\gamma _2}{\gamma _1+\gamma _2}}$, by the fact of $\kappa <1$ for $0<\gamma _1<\gamma _2$, we can obtain that

$$\begin{aligned} \Vert \varvec{\beta }^{m,t}-\varvec{\beta }^{*}\Vert _2\le & {} \kappa \Vert \varvec{\beta }^{m,t-1}-\varvec{\beta }^{*}\Vert _2+\frac{2\varepsilon _{_\textrm{MOM}}}{\gamma _1+\gamma _2}\\\le & {} \kappa ^2\Vert \varvec{\beta }^{m,t-2}-\varvec{\beta }^{*}\Vert _2 +\frac{2(1+\kappa )\varepsilon _{_\textrm{MOM}}}{\gamma _1+\gamma _2}\\{} & {} \vdots \\\le & {} \kappa ^t\Vert \varvec{\beta }^{m,0}-\varvec{\beta }^{*}\Vert _2 +\frac{2(1+\kappa +\ldots +\kappa ^t)\varepsilon _{_\textrm{MOM}}}{\gamma _1+\gamma _2}\\\le & {} \kappa ^t\Vert \varvec{\beta }^{0}-\varvec{\beta }^{*}\Vert _2 +\frac{2\varepsilon _{_\textrm{MOM}}}{(1-\kappa )(\gamma _1+\gamma _2)} \end{aligned}$$

with probability at least $1-Te^{-\frac{L(1-2C_1)^2c^2}{2(2+c)^2}}$ for all $c>0$, and C is a numerical constant.

Finally, by the norm inequality of $\Vert \varvec{\beta }^{T}-\varvec{\beta }^{*}\Vert _2 \le \frac{1}{M}\sum _{m=1}^M\Vert \varvec{\beta }^{m,T}-\varvec{\beta }^{*}\Vert _2$, we have

$$\begin{aligned} \Vert \varvec{\beta }^{T}-\varvec{\beta }^{*}\Vert _2 \le \kappa ^T\Vert \varvec{\beta }^{0}-\varvec{\beta }^{*}\Vert _2 +\frac{2\varepsilon _{_\textrm{MOM}}}{(1-\kappa )(\gamma _1+\gamma _2)} \end{aligned}$$

with probability at least $1-Te^{-\frac{L(1-2C_1)^2c^2}{2(2+c)^2}}$. $\square$

1.3 A.3 Proof of Proposition 5 and Theorem 6

Before the proof of Proposition 5, we introduce Theorem 3.2 proposed by Minsker and Ndaoud (2021) and write it as the following form to provide the error bounds of the weighted-based gradient estimation.

Lemma 7

Suppose $\max _{1\le m\le M}\vert {\mathcal {O}}_m\vert \le C_2L$ for $0\le C_2<1$ and event $\Omega _m^\nu$ holds. Then, we have

$$\begin{aligned}{} & {} \vert (\varvec{{\widehat{g}}}_m^{\textrm{WT}}(\varvec{\beta }) -\lambda _m\varvec{\beta }-\nabla {\mathcal {R}}(\varvec{\beta }))_r\vert \\{} & {} \quad \le \frac{C_\nu \sqrt{Cov(\nabla \ell (\varvec{\beta }))_{rr}}}{(1-C_2)^\nu } \left( \sqrt{\frac{1+c_1}{n}}+\sqrt{\frac{L}{n}} +\quad \frac{\vert {\mathcal {O}}_m\vert }{\sqrt{nL}\left( \sqrt{\alpha ({\mathcal {O}})}\right) ^{\nu -1}}\right) \end{aligned}$$

with probability at least $1-2e^{-c_1}-Le^{-c_2n/L}$ for $m=1,2,\ldots ,M$ and $r=1,2,\ldots ,(k-1)p$. Here, $C_\nu >0$, $c_1>0$, $c_2>0$, and

$$\begin{aligned} \Omega _m^{\nu }=\bigcup _{r=1}^{(k-1)p}\left\{ \sum _{l=1}^L1/(\varvec{\sigma }_m^{(l)}(\varvec{\beta }))_r^\nu \le (4\sqrt{(Cov(\nabla \ell (\varvec{\beta })))_{rr}}/(1-C_2))^\nu \right\} . \end{aligned}$$

Thus, for given $\widetilde{\varvec{\beta }}$ with $\Vert \widetilde{\varvec{\beta }}-\varvec{\beta }^0\Vert \le C_0$, by Lemma 7, we can get that directly

$$\begin{aligned}{} & {} \Vert \varvec{{\widehat{g}}}_m^{\textrm{WT}}(\widetilde{\varvec{\beta }}) -\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }})\Vert _2\\{} & {} \quad \le \Vert \varvec{{\widehat{g}}}_m^{\textrm{WT}}(\widetilde{\varvec{\beta }})-\lambda _m\widetilde{\varvec{\beta }} -\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }})\Vert _2+\lambda _m\Vert \widetilde{\varvec{\beta }}\Vert _2\\{} & {} \quad \le \sqrt{(k-1)p}\max _{1\le r\le (k-1)p}\vert (\varvec{{\widehat{g}}}_m^{\textrm{WT}}(\widetilde{\varvec{\beta }})-\lambda _m\widetilde{\varvec{\beta }} -\nabla {\mathcal {R}}(\widetilde{\varvec{\beta }}))_r\vert +Cn^{-1/2}\\{} & {} \quad \le \max _{1\le r\le (k-1)p}\frac{C_\nu \sqrt{(k-1)pCov(\nabla \ell (\widetilde{\varvec{\beta }}))_{rr}}}{(1-C_2)^\nu } \Big (\sqrt{\frac{1+c_1}{n}}+\sqrt{\frac{L}{n}} +\frac{\vert {\mathcal {O}}_m\vert }{\sqrt{nL}(\sqrt{\alpha ({\mathcal {O}})})^{\nu -1}}\Big )\\{} & {} \quad +Cn^{-1/2} \end{aligned}$$

with probability at least $1-(2e^{-c_1}+Le^{-c_2n/L})(k-1)p$.

The proof of Theorem 6 is similar to that given Theorem 4 above and so is omitted. $\square$