Recursive non-parametric kernel classification rule estimation for independent functional data

Abstract

In this paper we propose an automatic selection of the bandwidth of the recursive non-parametric estimation of the kernel classification rule function defined by the stochastic approximation algorithm, when the explanatory data are curves and the response is categorical. We established a central limit theorem for our proposed recursive estimators, the proposed recursive estimators will be very competitive to the non-recursive one in terms of estimation error but much better in terms of computational costs. The proposed estimators are used first on simulated waveform curves and then on real phoneme data.

Proofs

Throughout this section we use the following notation:

$$\begin{aligned}&Q_n=\prod _{j=1}^{n}\left( 1-\gamma _j\right) ,\nonumber \\&a_{n,g}\left( \chi ,h\right) =Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}\mathbb {1}_{\left\{ Y_k=g\right\} }K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) ,\nonumber \\&f_n\left( \chi ,h\right) =Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) ,\nonumber \\&a_g\left( \chi \right) :={\mathbb {P}}\left[ Y=g \cap \mathcal {X}=\chi \right] ,\nonumber \\&f\left( \chi \right) :={\mathbb {P}}\left[ \mathcal {X}=\chi \right] . \end{aligned}$$

(18)

Moreover, we let,

$$\begin{aligned} {\widehat{a}}_n^*\left( \chi ,h\right)= & {} \frac{Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k} \right) \mathbb {1}_{\left\{ Y_k^*=g\right\} }}{Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}{\mathbb {E}}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] },\\ {\widehat{f}}_n^*\left( \chi ,h\right)= & {} \frac{Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k} \right) }{Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}{\mathbb {E}} \left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] }. \end{aligned}$$

Before giving the outlines of the proofs, we state the following technical lemma, which is proved in Mokkadem et al. (2009a), and which is widely applied throughout the proofs.

Lemma 1

Let $\left( v_n\right) \in \mathcal {GS}\left( v^*\right)$, $\left( \gamma _n\right) \in \mathcal {GS}\left( -\alpha \right)$, and $m>0$ such that $m-v^*\xi >0$ where $\xi$ is defined in (8). We have

$$\begin{aligned} \lim _{n \rightarrow +\infty }v_nQ_n^{m}\sum _{k=1}^nQ_k^{-m}\frac{\gamma _k}{v_k} =\frac{1}{m-v^*\xi }. \end{aligned}$$

Moreover, for all positive sequence $\left( b_n\right)$ such that $\lim _{n \rightarrow +\infty }b_n=0$, and all $\delta \in {\mathbb {R}}$,

$$\begin{aligned} \lim _{n \rightarrow +\infty }v_nQ_n^{m}\left[ \sum _{k=1}^n Q_k^{-m} \frac{\gamma _k}{v_k}b_k+\delta \right] =0. \end{aligned}$$

Let us underline that the application of Lemma 1 which requires Assumption (A4)(iv) on the limit of $(n\gamma _n)$ as n goes to infinity.

Our proofs are organized as follows. Proposition 1, Theorem 1 and Theorem 2 are proved respectively, in Appendices 1, 2 and 3 . In all our proofs C and M stand for any arbitrary positive constants.

1.1 Proof of Proposition 1

Let us first use the following decomposition

$$\begin{aligned}&{\mathbb {E}}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) \nonumber \\&\quad = \frac{{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] }{{\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] }-p_g\left( \chi \right) -\frac{{\mathbb {E}}\left\{ a_{n,g}\left( \chi ,h\right) \left[ f_n\left( \chi ,h\right) -{\mathbb {E}}\left( f_n\left( \chi ,h\right) \right) \right] \right\} }{\left\{ {\mathbb {E}} \left[ f_n\left( \chi ,h\right) \right] \right\} ^2}\nonumber \\&\qquad +\frac{{\mathbb {E}}\left\{ {\widehat{p}}_{n,g}\left( \chi ,h\right) \left[ f_n\left( \chi ,h\right) -{\mathbb {E}}\left( f_n\left( \chi ,h\right) \right) \right] ^2\right\} }{\left\{ {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] \right\} ^2}. \end{aligned}$$

(19)

Computing the expectation of $f_n\left( \chi ,h\right)$

First, we have

$$\begin{aligned} {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right]= & {} Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}{\mathbb {E}}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \nonumber \\= & {} Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}\left\{ \int _0^1K\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) } \left( u\right) \right\} \nonumber \\= & {} Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}\left\{ F\left( h_k\right) \left[ K\left( 1\right) -\int _0^1K^{\prime }\left( u\right) \tau _{h_k} \left( u\right) du\right] \right\} . \end{aligned}$$

Then, since we have $\lim _{n\rightarrow \infty }\left( n\gamma _n\right) >\mathcal {F}_a$, the application of Lemma 1 gives

$$\begin{aligned} {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] =\frac{1}{1-\left( \mathcal {F}_a -a\right) \xi }M_{\chi , 1}h_n^{-1}F\left( h_n\right) \left[ 1+o\left( 1\right) \right] . \end{aligned}$$

(20)

Computing the expectation of $a_{n,g}\left( \chi ,h\right)$

Now, we have

$$\begin{aligned} {\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right]= & {} Q_n \sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}{\mathbb {E}} \left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} } -p_g\left( \chi \right) \right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&+p_g\left( \chi \right) {\mathbb {E}} \left[ f_n\left( \chi ,h\right) \right] . \end{aligned}$$

Taylor’s expansion of $\phi$ around 0 ensures that

$$\begin{aligned}&{\mathbb {E}}\left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} }-p_g\left( \chi \right) \right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&\quad ={\mathbb {E}}\left[ \left( p_g\left( \mathcal {X}_k\right) -p_g\left( \chi \right) \right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&\quad ={\mathbb {E}}\left[ \phi \left( \left\| \chi -\mathcal {X}_k\right\| \right) K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&\quad =\int _0^1\phi \left( h_ku\right) K\left( u\right) d{\mathbb {P}}^{ \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \\&\quad =h_k\phi ^{\prime }\left( 0\right) \int _0^1uK\left( u\right) d{\mathbb {P}}^{ \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) +o\left( h_k\right) . \end{aligned}$$

Moreover, it follows from the proof of Lemma 2 in Ferraty et al. (2007), the assumption $\left( A2\right)$ and Fubini’s Theorem

$$\begin{aligned} \int _0^1uK\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) =F\left( h_k\right) \left[ K\left( 1\right) -\int _0^1\left( uK\left( u\right) \right) ^{\prime } \tau _{h_k}\left( u\right) du\right] , \end{aligned}$$

Then, since we have $\lim _{n\rightarrow \infty }\left( n\gamma _n\right) >\mathcal {F}_a$, the application of Lemma 1 gives

$$\begin{aligned} {\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right]= & {} \left\{ \frac{1}{1-\mathcal {F}_a\xi }\phi ^{\prime }\left( 0\right) M_{\chi , 0} +p_g\left( \chi \right) \frac{1}{1-\left( \mathcal {F}_a-a\right) \xi }M_{\chi , 1}h_n^{-1} \right\} \nonumber \\&F\left( h_n\right) \left[ 1+o\left( 1\right) \right] . \end{aligned}$$

(21)

The combination of (20) and (21) gives

$$\begin{aligned} \frac{{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] }{{\mathbb {E}} \left[ f_n\left( \chi ,h\right) \right] } -p_g\left( \chi \right)= & {} h_n\phi ^{\prime }\left( 0\right) \frac{1-\left( \mathcal {F}_a-a\right) \xi }{1 -\mathcal {F}_a\xi }\frac{M_{\chi , 0}}{M_{\chi , 1}}\left[ 1+o(1)\right] . \end{aligned}$$

(22)

Computing the variance of $f_n\left( \chi ,h\right)$

First, we have

$$\begin{aligned} Var\left[ f_n\left( \chi ,h\right) \right]= & {} Q_n^2\sum _{k=1}^nQ_k^{-2} \gamma _k^2h_k^{-2}Var\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] ,\\= & {} Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2} \left\{ {\mathbb {E}}\left[ K^2\left( \frac{\left\| \chi -\mathcal {X}_k \right\| }{h_k}\right) \right] -{\mathbb {E}}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] ^2\right\} \\= & {} Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2} \left\{ \int _0^1K^2\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \right. \\&\left. -\left[ \int _0^1K \left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \right] ^2\right\} \\= & {} Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}F\left( h_k\right) \left\{ \left[ K^2\left( 1\right) -\int _0^1\left( K^2\left( u\right) \right) ^{\prime } \tau _{h_k}\left( u\right) du\right] \right. \\&\left. -F\left( h_k\right) \left[ K\left( 1\right) -\int _0^1K^{\prime } \left( u\right) \tau _{h_k}\left( u\right) du\right] ^2\right\} . \end{aligned}$$

Then, since we have $\lim _{n\rightarrow \infty }\left( n\gamma _n\right) >\left( \mathcal {F}_a+\alpha \right) /2-a$, the application of Lemma 1 gives

$$\begin{aligned} Var\left[ f_n\left( \chi ,h\right) \right]= & {} \frac{1}{2-\left( \mathcal {F}_a+\alpha -2a\right) \xi }\frac{\gamma _n}{h_n^{2}}F\left( h_n\right) M_{\chi , 2}\left[ 1+o\left( 1\right) \right] . \end{aligned}$$

(23)

Computing the variance of $a_{n,g}\left( \chi ,h\right)$

First, we have

$$\begin{aligned}&Var\left[ a_{n,g}\left( \chi ,h\right) \right] =Q_n^2\sum _{k=1}^nQ_k^{-2} \gamma _k^2h_k^{-2}Var\left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] ,\nonumber \\&\quad =Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}\left\{ {\mathbb {E}} \left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] -{\mathbb {E}} \left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] ^2\right\} \nonumber \\&\quad =Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}\left\{ {\mathbb {E}} \left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} }-p_g\left( \chi \right) \right) K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] +p_g\left( \chi \right) {\mathbb {E}}\left[ K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right. \nonumber \\&\quad \left. -\left( {\mathbb {E}}\left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} } -p_g\left( \chi \right) \right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] +p_g\left( \chi \right) {\mathbb {E}} \left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right) ^2\right\} \nonumber \\&\quad =Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}\left\{ {\mathbb {E}} \left[ \phi \left( \left\| \chi -\mathcal {X}_k\right\| \right) K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] +p_g\left( \chi \right) {\mathbb {E}} \left[ K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right. \nonumber \\&\quad \left. -\left( {\mathbb {E}}\left[ \phi \left( \left\| \chi -\mathcal {X}_k\right\| \right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] +p_g\left( \chi \right) {\mathbb {E}}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right) ^2\right\} . \end{aligned}$$

Taylor’s expansion of $\phi$ around 0 ensures that, for $i\in \left\{ 1,2\right\}$

$$\begin{aligned}&{\mathbb {E}}\left[ \phi \left( \left\| \chi -\mathcal {X}_k\right\| \right) K^{i} \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&\quad =\int _0^1\phi \left( h_ku\right) K^{i} \left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \nonumber \\&\quad =h_k\phi ^{\prime }\left( 0\right) \int _0^1uK^{i}\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) } \left( u\right) +o\left( h_k\right) \nonumber \\&\quad =h_k\phi ^{\prime }\left( 0\right) F\left( h_k\right) \left[ K^i\left( 1\right) -\int _0^1\left( uK^i\left( u\right) \right) ^{\prime }\tau _{h_k}\left( u\right) du\right] . \end{aligned}$$

Moreover, we have for $i\in \left\{ 1,2\right\}$

$$\begin{aligned} {\mathbb {E}}\left[ K^i\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right]= & {} \int _0^1K^i\left( u\right) d{\mathbb {P}}^{ \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \\= & {} F\left( h_k\right) \left[ K^i\left( 1\right) -\int _0^1\left( K^i \left( u\right) \right) ^{\prime }\tau _{h_k}\left( u\right) du\right] \end{aligned}$$

Then, since we have $\lim _{n\rightarrow \infty }\left( n\gamma _n\right) >\left( \mathcal {F}_a+\alpha -a\right) /2$, the application of Lemma 1 gives

$$\begin{aligned} Var\left[ a_{n,g}\left( \chi ,h\right) \right]= & {} \frac{1}{2-\left( \mathcal {F}_a +\alpha -2a\right) \xi }\frac{\gamma _n}{h_n^{2}}F\left( h_n\right) M_{\chi , 2}\nonumber \\&\left\{ 1-\frac{2-\left( \mathcal {F}_a+\alpha -2a\right) \xi }{2-\left( 2\mathcal {F}_a+\alpha -2a\right) \xi }F\left( h_n\right) p_g^2 \left( \chi ,h\right) \frac{M_{\chi , 1}^2}{M_{\chi , 2}}\right\} \left[ 1+o\left( 1\right) \right] .\nonumber \\ \end{aligned}$$

(24)

Computing the covariance between $a_{n,g}\left( \chi ,h\right)$ and $f_n\left( \chi ,h\right)$

First, we have

$$\begin{aligned} {\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) f_n\left( \chi ,h\right) \right]= & {} Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}{\mathbb {E}} \left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \nonumber \\&+Q_n^2\sum _{\begin{array}{c} k,k^{\prime }=1\\ k\not =k^{\prime } \end{array}}^nQ_k^{-1} \Pi _{k^{\prime }}^{-1}\gamma _k\gamma _{k^{\prime }}h_k^{-1}{\mathbb {E}} \left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K\left( \frac{\left\| \chi -\mathcal {X}_k \right\| }{h_k}\right) \right] \nonumber \\&\times h_{k^{\prime }}^{-1}{\mathbb {E}}\left[ \mathbb {1}_{\left\{ Y_k^{\prime } =g\right\} }K\left( \frac{\left\| \chi -\mathcal {X}_{k^{\prime }}\right\| }{h_{k^{\prime }}}\right) \right] . \end{aligned}$$

(25)

Moreover, we have

$$\begin{aligned} {\mathbb {E}}\left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K^2 \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right]= & {} {\mathbb {E}}\left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} } -p_g\left( \chi \right) \right) K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\&+p_g\left( \chi \right) {\mathbb {E}}\left[ K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] . \end{aligned}$$

Taylor’s expansion of $\phi$ around 0 ensures that

$$\begin{aligned} {\mathbb {E}}\left[ \left( \mathbb {1}_{\left\{ Y_k=g\right\} } -p_g\left( \chi \right) \right) K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right]= & {} {\mathbb {E}}\left[ \left( p_g\left( X_k\right) -p_g\left( \chi \right) \right) K^2 \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\= & {} {\mathbb {E}}\left[ \phi \left( \left\| \chi -\mathcal {X}_k\right\| \right) K^2 \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\= & {} \int _0^1\phi \left( h_ku\right) K^2\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) \\= & {} h_k\phi ^{\prime }\left( 0\right) \int _0^1uK^2\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) } \left( u\right) +o\left( h_k\right) . \end{aligned}$$

Moreover, it follows from the proof of Lemma 2 in Ferraty et al. (2007), the assumption $\left( A2\right)$ and Fubini’s Theorem

$$\begin{aligned} \int _0^1uK^2\left( u\right) d{\mathbb {P}}^{\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }\left( u\right) =F\left( h_k\right) \left[ K^2\left( 1\right) -\int _0^1\left( uK^2\left( u\right) \right) ^{\prime } \tau _{h_k}\left( u\right) du\right] , \end{aligned}$$

Then, we get

$$\begin{aligned} {\mathbb {E}}\left[ \mathbb {1}_{\left\{ Y_k=g\right\} }K^2 \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right]= & {} h_k \phi ^{\prime }\left( 0\right) F\left( h_k\right) \left[ K^2\left( 1\right) -\int _0^1\left( uK^2\left( u\right) \right) ^{\prime }\tau _{h_k}\left( u\right) du \right] \nonumber \\&+p_g\left( \chi \right) F\left( h_k\right) \left[ K^2\left( 1\right) -\int _0^1\left( K^2\left( u\right) \right) ^{\prime }\tau _{h_k}\left( u\right) du\right] .\nonumber \\ \end{aligned}$$

(26)

Then, in view of (25), (26) and since we have $\lim _{n\rightarrow \infty }\left( n\gamma _n\right) >\left( \mathcal {F}_a+\alpha -a\right) /2$, the application of Lemma 1 gives

$$\begin{aligned}&Cov\left( a_{n,g}\left( \chi ,h\right) ,f_n\left( \chi ,h\right) \right) \nonumber \\&\quad =\frac{1}{2-\left( \mathcal {F}_a+\alpha -a\right) \xi }\frac{\gamma _n}{h_n^2}F\left( h_n\right) M_{\chi , 2}p_g\left( \chi \right) \left[ 1+o\left( 1\right) \right] . \end{aligned}$$

(27)

Computing the expectation of ${\widehat{p}}_{n,g}\left( \chi ,h\right)$

First, it follows from (23) and (24), that

$$\begin{aligned}&{\mathbb {E}}\left\{ a_{n,g}\left( \chi ,h\right) \left[ f_n\left( \chi ,h\right) -{\mathbb {E}}\left( f_n\left( \chi ,h\right) \right) \right] \right\} =O\left( \frac{\gamma _n}{F\left( h_n\right) }\right) . \end{aligned}$$

(28)

$$\begin{aligned}&{\mathbb {E}}\left\{ {\widehat{p}}_{n,g}\left( \chi ,h\right) \left[ f_n\left( \chi ,h\right) -{\mathbb {E}}\left( f_n\left( \chi ,h\right) \right) \right] ^2\right\} =O\left( \frac{\gamma _n}{F\left( h_n\right) }\right) . \end{aligned}$$

(29)

Then (9) follows from (19), (22), (28) and (29).

Computing the variance of ${\widehat{p}}_{n,g}\left( \chi ,h\right)$

We have

$$\begin{aligned}&Var\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] \nonumber \\&\quad \simeq \frac{Var\left[ a_{n,g}\left( \chi ,h\right) \right] }{\left\{ {\mathbb {E}} \left[ f_n\left( \chi ,h\right) \right] \right\} ^2}-4\frac{{\mathbb {E}} \left[ a_{n,g}\left( \chi ,h\right) \right] Cov\left( a_{n,g}\left( \chi ,h\right) ,f_n \left( \chi ,h\right) \right) }{\left\{ {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] \right\} ^3}\nonumber \\&\qquad +3Var\left[ f_n\left( \chi ,h\right) \right] \frac{\left\{ {\mathbb {E}} \left[ a_{n,g}\left( \chi ,h\right) \right] \right\} ^2}{\left\{ {\mathbb {E}} \left[ f_n\left( \chi ,h\right) \right] \right\} ^4}. \end{aligned}$$

(30)

Then, the combination of (20), (21), (23), (24), (27) and (30), ensures that

$$\begin{aligned} Var\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right]= & {} \frac{\left( 1 -\left( \mathcal {F}_a-a\right) \xi \right) ^2}{\left( 2-\left( \mathcal {F}_a +\alpha -2a\right) \xi \right) }\frac{M_{\chi , 2}}{M_{\chi , 1}^2}\frac{\gamma _n}{F\left( h_n\right) }\\&\left\{ 1-\frac{2-\left( \mathcal {F}_a+\alpha -2a\right) \xi }{2 -\left( 2\mathcal {F}_a+\alpha -2a\right) \xi }F\left( h_n\right) p_g^2 \left( \chi ,h\right) \frac{M_{\chi , 1}^2}{M_{\chi , 2}}\right\} \left[ 1+o\left( 1\right) \right] . \end{aligned}$$

1.2 Proof of Theorem 1

Let us at first assume that, if $a\ge \left( \alpha +\mathcal {F}_a\right) /2$, then

$$\begin{aligned} \sqrt{\gamma _n^{-1} F\left( h_n\right) }\left( {\widehat{p}}_{n,g}\left( \chi \right) -{\mathbb {E}}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] \right) {\mathop {\rightarrow }\limits ^{\mathcal {D}}}\mathcal {N}\left( 0, \frac{\left( 1-\left( \mathcal {F}_a-a\right) \xi \right) ^2}{\left( 2-\left( \mathcal {F}_a+\alpha -2a\right) \xi \right) }\frac{M_{\chi , 2}}{M_{\chi , 1}^2}\right) .\nonumber \\ \end{aligned}$$

(31)

In the case when $\gamma _n^{-1}h_n^2F\left( h_n\right) \rightarrow c$, Part 1 of Theorem 1 follows from the combination of (9) and (31). In the case $\gamma _n^{-1}h_n^2F\left( h_n\right) \rightarrow \infty$, (11) implies that

$$\begin{aligned} h_n^{-2}\left( {\widehat{p}}_{n,g}\left( \chi ,h\right) -{\mathbb {E}} \left( {\widehat{p}}_{n,g}\left( \chi ,h\right) \right) \right) {\mathop {\rightarrow }\limits ^{{\mathbb {P}}}}0, \end{aligned}$$

and the application of (9) gives Part 2 of Theorem 1.

We now prove (31). In view of (18), we have

$$\begin{aligned} {\widehat{p}}_{n,g}\left( \chi ,h\right) -{\mathbb {E}} \left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right]= & {} \frac{a_{n,g}\left( \chi ,h\right) }{f_n\left( \chi ,h\right) } -\frac{{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] }{{\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] } +o\left( \sqrt{\frac{\gamma _n}{F\left( h_n\right) }}\right) . \end{aligned}$$

Moreover, since we have

$$\begin{aligned}&{\frac{a_{n,g}\left( \chi ,h\right) }{f_n\left( \chi ,h\right) } -\frac{{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] }{{\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] } =\frac{1}{f_n\left( \chi ,h\right) {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] }}\\&\quad \left\{ {\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] \left( a_{n,g}\left( \chi ,h\right) -{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] \right) -{\mathbb {E}}\left[ a_{n,g}\left( \chi ,h\right) \right] \right. \\&\qquad \left. \left( f_n\left( \chi ,h\right) -{\mathbb {E}}\left[ f_n\left( \chi ,h\right) \right] \right) \right\} . \end{aligned}$$

Using Slutsky’s theorem and (9) and (10), we get (31).

1.3 Proof of Theorem 2

The proof is based on the following decomposition:

$$\begin{aligned}&{\Pr }^{\mathcal {S}}\left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) } \left\{ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) -{\widehat{p}}_{n,g} \left( \chi ,b\right) \right\} \le y\right) \\&\quad -\Pr \left( \sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) }\left\{ {\widehat{p}}_{n,g}\left( \chi ,h\right) -p_g\left( \chi \right) \right\} \le y\right) \\&\quad =\mathcal {T}_1\left( y\right) +\mathcal {T}_2\left( y\right) +\mathcal {T}_3\left( y\right) , \end{aligned}$$

where

$$\begin{aligned} \mathcal {T}_1\left( y\right)= & {} {\Pr }^{\mathcal {S}}\left( \sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) }\left\{ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) -{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} \le y\right) \\&-\Phi \left( \frac{y-\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) } \left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) \right] -{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) Var^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g} \left( \chi ,h\right) \right] }}\right) ,\\ \mathcal {T}_2\left( y\right)= & {} \Phi \left( \frac{y-\sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) }\left\{ \mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) Var \left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] }}\right) \\&-\Pr \left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\left\{ {\widehat{p}}_{n,g} \left( \chi ,h\right) -p_g\left( \chi \right) \right\} \le y\right) ,\\ \mathcal {T}_3\left( y\right)= & {} \Phi \left( \frac{y-\sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) }\left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g} \left( \chi ,h\right) \right] -{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) Var^{\mathcal {S}} \left[ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) \right] }}\right) \\&-\Phi \left( \frac{y-\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) } \left\{ \mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) Varr \left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] }}\right) , \end{aligned}$$

where $\mathrm{E}^{\mathcal {S}}$ and $Var^{\mathcal {S}}$ denote expectation and variance, conditionally on the sample $\mathcal {S}$, and $\psi$ denotes the standard normal distribution function. The first part of Theorem 1 ensures that

$$\begin{aligned} \mathcal {T}_2\left( y\right) \rightarrow 0\quad \text{ a.s. }\quad \forall y\in {\mathbb {R}}. \end{aligned}$$

(32)

Moreover,

$$\begin{aligned} \mathcal {T}_1\left( y\right) \rightarrow 0\quad \text{ a.s. }\quad \forall y\in {\mathbb {R}}, \end{aligned}$$

(33)

follows from the next lemma.

Lemma 2

Assume that Assumptions $\left( A1\right)$–$\left( A5\right)$ hold. Then, we have

$$\begin{aligned} \frac{{\widehat{p}}_{n,g}^*\left( \chi ,h\right) -\mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }{\sqrt{Var^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) \right] }}{\mathop {\rightarrow }\limits ^{d}} \mathcal {N}\left( 0,1\right) . \end{aligned}$$

Now, from (32), (33) and Polya’s theorem (see, e.g., Serfling 1980, p. 18) together with the continuity of the function $\psi$, we arrive at

$$\begin{aligned} \mathrm{sup}_{y\in {\mathbb {R}}}\left| \mathcal {T}_1\left( y\right) \right| +\mathrm{sup}_{y\in {\mathbb {R}}}\left| \mathcal {T}_2\left( y\right) \right| \rightarrow 0\quad \text{ a.s. }\quad \forall y\in {\mathbb {R}}. \end{aligned}$$

Finally, it remains to study the term $\mathcal {T}_3\left( y\right)$. Using the fact that, for any $a>0$ and $c\in {\mathbb {R}}$,

$$\begin{aligned} \mathrm{sup}_{y\in {\mathbb {R}}}\left| \psi \left( c+ay\right) -\psi \left( y\right) \right| \le \left| c\right| +\mathrm{max}\left\{ a,a^{-1}\right\} -1, \end{aligned}$$

and considering

$$\begin{aligned} a=\sqrt{\frac{Var\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] }{Var^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }} \end{aligned}$$

and

$$\begin{aligned} c=\frac{\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\left\{ \mathrm{E} \left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) -\mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) \right] +{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) Var^{\mathcal {S}} \left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }}, \end{aligned}$$

we get

$$\begin{aligned}&\mathrm{sup}_{y\in {\mathbb {R}}}\left| \mathcal {T}_3\left( y\right) \right| \nonumber \\&\quad \le \left| \frac{\sqrt{\gamma _n^{-1}F_{\chi } \left( h_n\right) }\left\{ \mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) -\mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g} \left( \chi ,h\right) \right] +{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} }{\sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) Var^{\mathcal {S}} \left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }}\right| \nonumber \\&\qquad +\mathrm{max}\left\{ \sqrt{\frac{Var\left[ {\widehat{p}}_{n,g} \left( \chi ,h\right) \right] }{Var^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^* \left( \chi ,h\right) \right] }},\sqrt{\frac{Var^{\mathcal {S}} \left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }{Var \left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] }}\right\} -1. \end{aligned}$$

(34)

The combination of (10), (34) and the following two lemmas ensure the convergence of $\mathrm{sup}_{y\in {\mathbb {R}}}\left| \mathcal {T}_3\left( y\right) \right|$.

Lemma 3

Assume that Assumptions $\left( A1\right)$–$\left( A5\right)$ hold. Then

$$\begin{aligned} \sqrt{\frac{Var\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] }{Var^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] }}\rightarrow 1\quad \text{ a.s. } \end{aligned}$$

Lemma 4

Assume that Assumptions $\left( A1\right)$–$\left( A5\right)$ hold. Then

$$\begin{aligned} \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\left\{ \mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) -\mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}^*_{n,g}\left( \chi ,h\right) \right] +{\widehat{p}}_{n,g}\left( \chi ,b\right) \right\} \rightarrow 0\quad \text{ a.s. } \end{aligned}$$

1.3.1 Proof of Lemma 2

Using the fact that, for x such that ${\widehat{f}}_{n}^*\left( \chi ,h\right) \not =0$, we have the following decomposition:

$$\begin{aligned} {\widehat{p}}_{n,g}^*\left( \chi ,h\right) -p_g\left( \chi \right)= & {} \mathcal {D}_n^*\left( \chi ,h\right) \frac{f\left( \chi \right) }{{\widehat{f}}_{n}^*\left( \chi ,h\right) }, \end{aligned}$$

(35)

with

$$\begin{aligned} \mathcal {D}_n^*\left( \chi ,h\right)= & {} \frac{1}{f\left( \chi \right) } \left\{ {\widehat{a}}_{n}^*\left( \chi ,h\right) -p_g\left( \chi \right) {\widehat{f}}_{n}^*\left( \chi ,h\right) \right\} . \end{aligned}$$

It follows from (35), that the asymptotic behavior of ${\widehat{p}}^*_{n,g}\left( \chi ,h\right) -p_g\left( \chi \right)$ can be deduced from the one of $\mathcal {D}^*_n\left( \chi ,h\right)$. Let us set

$$\begin{aligned} Z_k^*\left( \chi \right) =\frac{\gamma _k}{Q_kh_k} \left\{ \mathbb {1}_{\left\{ Y_k^*=g\right\} }-p_g\left( \chi \right) \right\} K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \end{aligned}$$

and

$$\begin{aligned} T_k^*\left( \chi \right) =Z_k^*\left( \chi \right) -\mathrm{E}^{\mathcal {S}}\left[ Z_k^*\left( \chi \right) \right] . \end{aligned}$$

(36)

Hence, we readily infer that

$$\begin{aligned} \mathcal {D}_n^*\left( \chi ,h\right) -\mathrm{E}^{\mathcal {S}} \left[ \mathcal {D}_n^*\left( \chi ,h\right) \right]= & {} \frac{Q_n}{f \left( \chi \right) \mathrm{E}\left[ {\widehat{f}}_n\left( \chi ,h\right) \right] } \sum _{k=1}^nT_k^*\left( \chi \right) . \end{aligned}$$

Moreover, since we have

$$\begin{aligned} Var^{\mathcal {S}}\left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] =\varepsilon _k^2,\quad \mathrm{E}^{\mathcal {S}}\left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} } \right] ={\widehat{p}}_{n,g}\left( \chi _k,b\right) , \end{aligned}$$

and given that $\mathrm{lim}_{n\rightarrow \infty }\left( n\gamma _n\right) >\left( \mathcal {F}_a +\gamma \right) /2-a$, the application of Lemma 1 together with (20), (26) and (36), ensures that

$$\begin{aligned} v_n^{*2}= & {} \sum _{k=1}^nVar^{\mathcal {S}}\left( T_{k}^*\left( \chi \right) \right) =\sum _{k=1}^n\frac{\gamma _k^2}{Q_k^{2}h_k^{2}}Var^{\mathcal {S}} \left[ \left\{ \mathbb {1}_{\left\{ Y_k^*=g\right\} }-p_g\left( \chi \right) \right\} K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \\= & {} O\left( \sum _{k=1}^n\frac{\gamma _k^2}{Q_k^{2}h_k^{2}}F_{\chi }\left( h_k\right) \right) =O\left( \frac{\gamma _nF_{\chi }\left( h_n\right) }{Q_n^2h_n^2}\right) . \end{aligned}$$

Now, we have, for all $p>0$,

$$\begin{aligned} \mathrm{E}\left[ \left| Z_k^*\left( \chi \right) \right| ^{2+p}\right]= & {} O\left( \frac{F_{\chi }\left( h_k\right) }{h_k^{1+p}}\right) , \end{aligned}$$

and, since $\mathrm{lim}_{n\rightarrow \infty }\left( n\gamma _n\right) >\left( \mathcal {F}_a +\gamma \right) /2-a$, there exists $p>0$ such that $\mathrm{lim}_{n\rightarrow \infty } \left( n\gamma _n\right) >\frac{1+p}{2+p}\left( \left( \mathcal {F}_a+\gamma \right) /2 -a\right)$. The application of Lemma 1, ensures that

$$\begin{aligned} \sum _{k=1}^n\mathrm{E}^{\mathcal {S}}\left[ \left| T_{k}^*\left( \chi \right) \right| ^{2+p} \right]= & {} O\left( \sum _{k=1}^n \frac{\gamma _k^{2+p}}{Q_k^{2+p}}\mathrm{E}^{\mathcal {S}} \left[ \left| Z_k^*\left( \chi \right) \right| ^{2+p}\right] \right) \\= & {} O\left( \sum _{k=1}^n \frac{Q_k^{-2-p}\gamma _k^{2+p}F_{\chi }\left( h_k\right) }{h_k^{1+p}}\right) \\= & {} O\left( \frac{\gamma _n^{1+p}F_{\chi }\left( h_n\right) }{Q_n^{2+p}h_n^{1+p}} \right) \nonumber , \end{aligned}$$

it comes that

$$\begin{aligned} \frac{1}{\left| v_n^{*}\right| ^{2+p}}\sum _{k=1}^n\mathrm{E}^{\mathcal {S}}\left[ \left| T_{k}^*\left( \chi \right) \right| ^{2+p}\right]= & {} O\left( {\left[ \gamma _nh_n^{-1}F_{\chi }\left( h_n\right) \right] }^{p/2}\right) =o\left( 1\right) . \end{aligned}$$

The convergence in (31) then follows from the application of Lyapounov’s theorem.

1.3.2 Proof of Lemma 3

Now, we consider the following decomposition:

$$\begin{aligned} Var^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right]\simeq & {} \frac{Var^{\mathcal {S}}\left[ {\widehat{a}}_{n}^*\left( \chi ,h\right) \right] }{\left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{f}}^*_{n}\left( \chi ,h\right) \right] \right\} ^2}\nonumber \\&-4\frac{\mathrm{E}^{\mathcal {S}}\left[ {\widehat{a}}^*_{n}\left( \chi ,h\right) \right] Cov^{\mathcal {S}} \left( {\widehat{a}}^*_{n}\left( \chi ,h\right) ),{\widehat{f}}^*_{n} \left( \chi ,h\right) \right) }{\left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{f}}^*_{n}\left( \chi ,h\right) \right] \right\} ^3}\nonumber \\&+3Var^{\mathcal {S}}\left[ {\widehat{f}}^*_{n}\left( \chi ,h\right) \right] \frac{\left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{a}}^*_{n}\left( \chi ,h\right) \right] \right\} ^2}{\left\{ \mathrm{E}^{\mathcal {S}}\left[ {\widehat{f}}^*_{n}\left( \chi ,h\right) \right] \right\} ^4}. \end{aligned}$$

(37)

Moreover, we have for

$$\begin{aligned}&Var^{\mathcal {S}}\left[ {\widehat{a}}_{n}^*\left( \chi ,h\right) \right] =\frac{Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}K^2 \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) Var^{\mathcal {S}} \left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] }{\left( Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}\mathrm{E} \left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right) ^2}, \qquad \end{aligned}$$

(38)

$$\begin{aligned}&\mathrm{E}^{\mathcal {S}}\left[ {\widehat{f}}_{n}^*\left( \chi ,h\right) \right] =\frac{Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) }{Q_n\sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}\mathrm{E}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] } \end{aligned}$$

(39)

and

$$\begin{aligned}&{\mathrm{E}^{\mathcal {S}}\left[ {\widehat{a}}_{n}^*\left( \chi ,h\right) {\widehat{f}}_{n}^*\left( \chi ,h\right) \right] }\nonumber \\&\quad =\frac{Q_n^2\sum _{k=1}^nQ_k^{-2}\gamma _k^2h_k^{-2}K^2\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \mathrm{E}^{\mathcal {S}} \left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] }{\left( Q_n\sum _{k=1}^nQ_k^{-1} \gamma _kh_k^{-1}\mathrm{E}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right) ^2}\nonumber \\&\quad +\, \frac{Q_n^2\sum _{\begin{array}{c} k,k^{\prime }=1\\ k\not =k^{\prime } \end{array}}^nQ_k^{-1} \Pi _{k^{\prime }}^{-1}\gamma _k\gamma _{k^{\prime }}h_k^{-1}h_{k^{\prime }}^{-1}K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k^{\prime }}\right) \mathrm{E}^{\mathcal {S}}\left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] }{\left( Q_n \sum _{k=1}^nQ_k^{-1}\gamma _kh_k^{-1}\mathrm{E}\left[ K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \right] \right) ^2}. \end{aligned}$$

(40)

We recall again that

$$\begin{aligned} Var^{\mathcal {S}}\left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] =\varepsilon _k^2,\quad \mathrm{E}^{\mathcal {S}}\left[ \mathbb {1}_{\left\{ Y_k^*=g\right\} }\right] ={\widehat{p}}_{n,g}\left( \chi _k,b\right) , \end{aligned}$$

the combination of (37), (38), (39), (40) and some computational analysis ensures that

$$\begin{aligned} Var^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] =Var\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] \left( 1+o\left( 1\right) \right) . \end{aligned}$$

1.3.3 Proof of Lemma 4

This proof follows the same steps as those used in Ferraty et al. (2010) and more recently in Slaoui (2019) by using the following decomposition:

$$\begin{aligned} \mathrm{E}^{\mathcal {S}}\left[ {\widehat{p}}_{n,g}^*\left( \chi ,h\right) \right] -{\widehat{p}}_{n,g}\left( \chi ,b\right) ={\mathbb {I}}_1+{\mathbb {I}}_2+{\mathbb {I}}_3, \end{aligned}$$

where

$$\begin{aligned} {\mathbb {I}}_1= & {} {\mathbb {J}}_1^{-1}\sum _{k=1}^n\frac{Q_n\gamma _k}{Q_kh_k}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \left\{ {\widehat{a}}_{n,g}\left( \chi _k,b\right) -{\widehat{p}}_{n,g} \left( \chi ,b\right) {\widehat{f}}_{n}\left( \chi _k,b\right) \right. \\&-\left. \mathrm{E}\left[ {\widehat{a}}_{n,g}\left( \chi _k,b\right) \right] +\mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,b\right) {\widehat{f}}_{n} \left( \chi _k,b\right) \right] \right\} ,\\ {\mathbb {I}}_2= & {} {\mathbb {J}}_1^{-1}\sum _{k=1}^n\frac{Q_n\gamma _k}{Q_kh_k}K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \left\{ \mathrm{E}\left[ {\widehat{a}}_{n,g}\left( \chi _k,b\right) \right] -\mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,b\right) {\widehat{f}}_{n} \left( \chi _k,b\right) \right] \right. \\&\left. -{a}\left( \chi _k\right) +p_g\left( \chi \right) {f}\left( \chi _k\right) \right\} ,\\ {\mathbb {I}}_3= & {} {\mathbb {J}}_1^{-1}\sum _{k=1}^n\frac{Q_n\gamma _k}{Q_kh_k}K \left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) \left\{ {a}_g\left( \chi \right) -p_g\left( \chi \right) f\left( \chi _k\right) \right\} ,\\ {\mathbb {J}}_1= & {} \sum _{k=1}^n\frac{Q_n\gamma _k}{Q_kh_k}K\left( \frac{\left\| \chi -\mathcal {X}_k\right\| }{h_k}\right) {\widehat{f}}_{n}\left( \chi _k,b\right) . \end{aligned}$$

Moreover, we can check that ${\mathbb {I}}_3=\mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,h\right) \right] -p_g\left( \chi \right) +o\left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\right)$ a.s., whereas ${\mathbb {I}}_1$ and ${\mathbb {I}}_2$ are $o\left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\right)$ a.s. by the following two Lemmas.

Lemma 5

Assume that Assumptions $\left( A1\right)$–$\left( A5\right)$ hold. Then

$$\begin{aligned}&\mathrm{sup}_{\left\| \chi -\chi _1\right\| \le h}\left| \mathrm{E}\left[ {\widehat{a}}_{n}\left( \chi ,b\right) \right] -\mathrm{E}\left[ {\widehat{p}}_{n,g}\left( \chi ,b\right) {\widehat{f}}_{n} \left( \chi _1,b\right) \right] -a_g\left( \chi _1\right) \right. \\&\quad \left. +p_g\left( \chi \right) f \left( \chi _1\right) \right| =o\left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\right) \quad \text{ a.s. } \end{aligned}$$

Lemma 6

Assume that Assumptions $\left( A1\right)$–$\left( A5\right)$ hold. Then

$$\begin{aligned} \mathrm{sup}_{\left\| \chi -\chi _1\right\| \le h}\left| \mathrm{E}\left[ {\widehat{a}}_{n} \left( \chi _1,b\right) )\right] -\mathrm{E}\left[ {\widehat{p}}_{n,g} \left( \chi ,b\right) {\widehat{f}}_{n}\left( \chi _1,b\right) \right] \right. \\ \left. -{\widehat{a}}_{n}\left( \chi _1,b\right) +{\widehat{p}}_{n,g} \left( \chi ,b\right) {\widehat{f}}_{n}\left( \chi _1,b\right) \right| =o\left( \sqrt{\gamma _n^{-1}F_{\chi }\left( h_n\right) }\right) \quad \text{ a.s. } \end{aligned}$$

The proof of Lemmas 5 and 6 are obtained by following the same lines and decompositions as the one considered in the proof of Lemmas A.5 and A.6 in Ferraty et al. (2010) and more recently in Slaoui (2019).