Adaptive robust local online density estimation for streaming data

Abstract

Accurate online density estimation is crucial to numerous applications that are prevalent with streaming data. Existing online approaches for density estimation somewhat lack prompt adaptability and robustness when facing concept-drifting and noisy streaming data, resulting in delayed or even deteriorated approximations. To alleviate this issue, in this work, we first propose an adaptive local online kernel density estimator (ALoKDE) for real-time density estimation on data streams. ALoKDE consists of two tightly integrated strategies: (1) a statistical test for concept drift detection and (2) an adaptive weighted local online density estimation when a drift does occur. Specifically, using a weighted form, ALoKDE seeks to provide an unbiased estimation by factoring in the statistical hallmarks of the latest learned distribution and any potential distributional changes that could be introduced by each incoming instance. A robust variant of ALoKDE, i.e., R-ALoKDE, is further developed to effectively handle data streams with varied types/levels of noise. Moreover, we analyze the asymptotic properties of ALoKDE and R-ALoKDE, and also derive their theoretical error bounds regarding bias, variance, MSE and MISE. Extensive comparative studies on various artificial and real-world (noisy) streaming data demonstrate the efficacies of ALoKDE and R-ALoKDE in online density estimation and real-time classification (with noise).

Appendix

1.1 Appendix 1: Proof of Theorem 1

Equation (6) is equivalent to the following equation,

$$\begin{aligned} \begin{aligned} \lambda _{t}&= \mathop {\mathrm{argmin}}_{\lambda \in [0,1]} \int [\lambda E(\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})) + (1-\lambda )E(\hat{f}_{t}({{\varvec{x}}}))\\&\quad -f({{\varvec{x}}})]^2 d {{\varvec{x}}} + \int Var \{\lambda \hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})+ (1-\lambda )\hat{f}_{t}({{\varvec{x}}}) \} d {{\varvec{x}}}\\&=\mathop {\mathrm{argmin}}_{\lambda \in [0,1]} \lambda ^2 \int [E(\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})) - f({{\varvec{x}}})]^2 d {{\varvec{x}}} \\&\quad +(1-\lambda )^2 \int [E(\hat{f}_{t}({{\varvec{x}}})) - f({{\varvec{x}}})]^2 d {{\varvec{x}}} \\&\quad +2 \lambda (1-\lambda ) \int [E(\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})) - f({{\varvec{x}}})]\\&[E(\hat{f}_{t}({{\varvec{x}}})) - f({{\varvec{x}}})] d {{\varvec{x}}}\\&\quad + \lambda ^2 \int Var \{ \hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}}) \} d {{\varvec{x}}} + (1-\lambda )^2 \int Var \{ \hat{f}_{t}({{\varvec{x}}}) \} d {{\varvec{x}}}\\&\quad + 2 \lambda (1-\lambda ) \int Cov \{ \hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}}),\hat{f}_{t}({{\varvec{x}}}) \} d {{\varvec{x}}}\\&=\mathop {\mathrm{argmin}}_{\lambda \in [0,1]} \lambda ^2 A_t + (1-\lambda )^2 B_t + 2 \lambda (1-\lambda ) C_t\\&=\mathop {\mathrm{argmin}}_{\lambda \in [0,1]} (A_t + B_t - 2C_t)\lambda ^2 - 2(B_t - C_t)\lambda \end{aligned} \end{aligned}$$

(13)

In fact, $A_{t}$ is the MISE of $\hat{f}_{t}^{kde }$, $B_t$ is the MISE of $\hat{f}_{t}$, and $C_t$ is the covariance between $\hat{f}_{t}^{kde }$ and $\hat{f}_{t}$. Technically, they can be efficiently computed by the second-order Taylor series approximation [29], where we can use an estimate of the curvature $\int |f''({{\varvec{x}}})|^2 d {{\varvec{x}}}$ and choose $h_{opt}$ accordingly.

Furthermore, since $x^2 + y^2 \ge 2xy$, $Var\{X - Y\} = Var\{X\} + Var\{Y\} - 2 Cov\{X, Y\} \ge 0$, $Var\{X\} + Var\{Y\} \ge 2 Cov\{X, Y\}$, we obtain $A_t + B_t \ge 2C_t$. Note that the equality holds if and only if $\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})=\hat{f}_{t}({{\varvec{x}}})$, which is unlikely to happen since we utilize a local sampling strategy to construct $\hat{f}_{t}^{kde }$ when concept drift occurs. Then, the solution of the optimization problem in Eq. (7) is discussed below:

• Case 1: if $\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}})=\hat{f}_{t}({{\varvec{x}}})$, then $A_t = B_t = C_t$, we have $\forall \lambda _t \in [0,1]$;

• Case 2: if $\hat{f}_{t}^{kde }({{\varvec{x}}};h_{\mathcal {D}_{t}},h_{{{\varvec{x}}}_{t}}) \ne \hat{f}_{t}({{\varvec{x}}})$, then $A_t + B_t > 2C_t$.

  • Case 2-1: if $B_t < C_t$, then $A_t > C_t$, $\lambda _t = 0$;

  • Case 2-2: if $B_t \ge C_t$ and $A_t < C_t$, then $\lambda _t = 1$;

  • Case 2-3: if $B_t \ge C_t$ and $A_t \ge C_t$, then $\lambda _t = \frac{B_{t}-C_{t}}{A_{t}+B_{t}-2C_{t}}$, $1 - \lambda _t = \frac{A_{t}-C_{t}}{A_{t}+B_{t}-2C_{t}}$.

Therefore, we have a closed-form solution of Eq. (7) for Case 2 (i.e., $A_t + B_t > 2C_t$):

$$\begin{aligned} \lambda _{t} = \max {\left\{ 0, \min {\left\{ 1,\frac{B_{t}-C_{t}}{A_{t}+B_{t}-2C_{t}}\right\} }\right\} }. \end{aligned}$$

(14)

This concludes the proof of Theorem 1.

1.2 Appendix 2: Proof of Theorem 2

Since $\lambda _1=\cdots =\lambda _{t-1}=\lambda _{t}=\lambda$, then the following equation holds:

$$\begin{aligned} E(\hat{f}_{t+1}(x)) = \lambda E(\hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}})) + (1-\lambda )E(\hat{f}_{t}(x)), \end{aligned}$$

(15)

using the above equation repeatedly, we have:

$$\begin{aligned} E(\hat{f}_{t+1}(x))= & {} \lambda \sum _{j=0}^{t-1} (1-\lambda )^{j} E(\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}}))\nonumber \\&+ (1-\lambda )^{t} E(\hat{f}_{1}(x)). \end{aligned}$$

(16)

Lemma 1 implies that: $\lim \limits _{m_{t-j}\rightarrow \infty }E(\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}}))=f(x)$, where $j=0,1,\ldots ,t-1$.

On the other hand, since $m_1=\cdots =m_{t}$, $h_{\mathcal {D}_{1}}=\cdots =h_{\mathcal {D}_{t}}$, we have:

$$\begin{aligned} \begin{aligned}&\lim \limits _{m_{t}\rightarrow \infty }E(\hat{f}_{t+1}(x)) = \lambda \sum _{j=0}^{t-1} (1-\lambda )^{j} f(x) + (1-\lambda )^{t} E(\hat{f}_{1}(x))\\&\quad = (1-(1-\lambda )^{t})f(x)+(1-\lambda )^{t} E(\hat{f}_{1}(x)). \end{aligned} \end{aligned}$$

(17)

With $t \rightarrow \infty$, for arbitrary $\lambda \in (0,1]$, we have: $\lim \nolimits _{m_{t}\rightarrow \infty }E(\hat{f}_{t+1}(x))=f(x)$.

We then consider the variance. Since $\mathcal {D}_{j}$ and $\mathcal {D}_{k}$ ($j \ne k$) are two independent sets, $Cov\{\hat{f}_{j}^{kde}(x;h_{\mathcal {D}_{j}},h_{x_{j}}), \hat{f}_{k}^{kde}(x;h_{\mathcal {D}_{k}},h_{x_{k}}) \} = 0$. Since $\hat{f}_{1}(x)$ is initialized by two given constants, then for arbitrary t, we have $Cov\{\hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}}),\hat{f}_{1}(x)\} = 0$. Hence, the following equation holds:

$$\begin{aligned} \begin{aligned}&Var\{\hat{f}_{t+1}(x)\} = Var\{ \lambda \hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}}) + (1-\lambda )\hat{f}_{t}(x) \} \\&\quad = \lambda ^2 Var\{ \hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}}) \} + (1-\lambda )^2 Var\{ \hat{f}_{t}(x) \}\\&\qquad + 2\lambda (1-\lambda )Cov\{ \hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}}),\hat{f}_{t}(x) \} \\&\quad = \lambda ^2 Var\{ \hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}}) \} + \lambda ^2 (1-\lambda )^2\\&\quad Var\{ \hat{f}_{t-1}^{kde}(x;h_{\mathcal {D}_{t-1}},h_{x_{t-1}}) \} + (1-\lambda )^4 Var\{ \hat{f}_{t-1}(x) \} \\&\qquad + 2 \lambda (1-\lambda ) Cov \{ \hat{f}_{t}^{kde}(x;h_{\mathcal {D}_{t}},h_{x_{t}})+(1-\lambda )^2\hat{f}_{t-1}^{kde}\\&\qquad \times (x;h_{\mathcal {D}_{t-1}},h_{x_{t-1}}),\hat{f}_{t}(x) \} = \cdots = \\&\quad = \lambda ^2 \sum _{j=0}^{t-1} (1-\lambda )^{2j} Var\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\}\\&\qquad +(1-\lambda )^{2t} Var\{\hat{f}_{1}(x)\} \\&\qquad + 2 \lambda (1-\lambda ) \sum _{j=0}^{t-1} (1-\lambda )^{2j} Cov\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}}),\hat{f}_{1}(x) \} \\&\quad = \lambda ^2 \sum _{j=0}^{t-1} (1-\lambda )^{2j} Var\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\}\\&\qquad +(1-\lambda )^{2t} Var\{\hat{f}_{1}(x)\}. \end{aligned} \end{aligned}$$

(18)

Lemma 1 implies that: $\lim \nolimits _{m_{t-j}\rightarrow \infty }m_{t-j}h_{\mathcal {D}_{t-j}}Var\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\}=f(x)\int _{-\infty }^{\infty }K^2(u)du$, where $j=0,1,\ldots ,t-1$. Since $m_1=\cdots =m_{t}$, $h_{\mathcal {D}_{1}}=\cdots =h_{\mathcal {D}_{t}}$, we have:

$$\begin{aligned} \begin{aligned}&\lim \limits _{m_{t}\rightarrow \infty }m_{t}h_{\mathcal {D}_{t}}Var\{\hat{f}_{t+1}(x)\}\\&\quad = \lambda ^2 \sum _{j=0}^{t-1} (1-\lambda )^{2j} f(x)\int _{-\infty }^{\infty }K^2(u)du \\&\qquad +(1-\lambda )^{2t} Var\{\hat{f}_{1}(x)\}\\&\quad = \frac{\lambda ^2(1-(1-\lambda )^{2t})}{1-(1-\lambda )^2}f(x)\int _{-\infty }^{\infty }K^2(u)du + (1-\lambda )^{2t} Var\{ \hat{f}_{1}(x) \}. \end{aligned} \end{aligned}$$

(19)

Now with $t \rightarrow \infty$, for arbitrary $\lambda \in (0, 1]$, we have: $\lim \nolimits _{m_{t}\rightarrow \infty }m_{t}h_{\mathcal {D}_{t}}Var\{\hat{f}_{t+1}(x)\}= \frac{\lambda f(x)}{2-\lambda }\int _{-\infty }^{\infty }K^2(u)du$. This concludes the proof of Theorem 2.

1.3 Appendix 3: Proof of Theorem 3

We consider the mean squared error of $\hat{f}_{t+1}(x)$, $MSE (\hat{f}_{t+1}(x))=Var\{\hat{f}_{t+1}(x)\}+(Bias\{\hat{f}_{t+1}(x)\})^2$. According to Theorem 2, the estimator $\hat{f}_{t+1}(x)$ is asymptotically unbiased and therefore, $\lim \nolimits _{m_{t}\rightarrow \infty }Bias\{\hat{f}_{t+1}(x)\}=0$. Since $\lim \nolimits _{m_{t}\rightarrow \infty }m_{t}h_{\mathcal {D}_{t}} = \infty$ and $f(x)\int _{-\infty }^{\infty }K^2(u)du<\infty$, then for arbitrary $\lambda \in (0,1]$, we have, $\lim \nolimits _{m_{t}\rightarrow \infty ,t\rightarrow \infty }Var\{\hat{f}_{t+1}(x)\}=\lim \limits _{m_{t}\rightarrow \infty ,t\rightarrow \infty }\frac{\lambda f(x)\int _{-\infty }^{\infty }K^2(u)du}{(2-\lambda )m_{t}h_{\mathcal {D}_{t}}}=0$. Therefore, $\hat{f}_{t+1}(x)$ converges to f(x) in $L^2$ and hence is weakly consistent. This concludes the proof of Theorem 3.

1.4 Appendix 4: Proof of Theorem 4

According to the proof of Theorem 2 ($\lambda _1=\cdots =\lambda _{t-1}=\lambda _{t}=\lambda$), the following equation holds:

$$\begin{aligned} \begin{aligned}&Bias\{\hat{f}_{t+1}(x)\} = E(\hat{f}_{t+1}(x)) -f(x)=\lambda \sum _{j=0}^{t-1} (1-\lambda )^{j} \cdot \\&E\left( \hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\right) + (1-\lambda )^{t} E(\hat{f}_{1}(x)) - f(x). \end{aligned} \end{aligned}$$

(20)

According to the properties of KDE by Taylor expansion [29], we have, $Bias\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\}= \frac{1}{2}f''(x)h_{\mathcal {D}_{t-j}}^2\int _{-\infty }^{\infty }u^2K(u)du+o(h_{\mathcal {D}_{t-j}}^2)$, where $j=0,1,\ldots ,t-1$.

Thus, with $m_1=\cdots =m_{t}$, $h_{\mathcal {D}_{1}}=\cdots =h_{\mathcal {D}_{t}}$, we have:

$$\begin{aligned} \begin{aligned} Bias\{\hat{f}_{t+1}(x)\}&= (1-(1-\lambda )^{t}) \left\{ \frac{1}{2}f''(x)h_{\mathcal {D}_{t}}^2 \int _{-\infty }^{\infty }u^2K(u)du\right\} \\&\quad + o(h_{\mathcal {D}_{t}}^2)+(1-\lambda )^{t} E(\hat{f}_{1}(x)- f(x)). \end{aligned} \end{aligned}$$

(21)

For arbitrary $\lambda \in (0,1]$, let t be large enough ($(1-\lambda )^t \approx 0$), we can immediately derive Eq. (9) from the above equation.

For the variance, since Eq. (18) holds and according to the properties of KDE by Taylor expansion [29], we have, $Var\{\hat{f}_{t-j}^{kde}(x;h_{\mathcal {D}_{t-j}},h_{x_{t-j}})\} = \frac{f(x)}{m_{t-j}h_{\mathcal {D}_{t-j}}}\int _{-\infty }^{\infty }K^2(u)du + o(\frac{1}{m_{t-j}})$, where $j=0,1,\ldots ,t-1$. Since $m_1=\cdots =m_{t}$ and $h_{\mathcal {D}_{1}}=\cdots =h_{\mathcal {D}_{t}}$, we have:

$$\begin{aligned} \begin{aligned} Var\{\hat{f}_{t+1}(x)\}&= \frac{\lambda ^2(1-(1-\lambda )^{2t})}{1-(1-\lambda )^2} \left\{ \frac{f(x)}{m_{t}h_{\mathcal {D}_{t}}}\int _{-\infty }^{\infty }K^2(u)du \right\} \\&\quad + o\left( \frac{1}{m_{t}}\right) +(1-\lambda )^{2t} Var\{\hat{f}_{1}(x) \}. \end{aligned} \end{aligned}$$

(22)

For arbitrary $\lambda \in (0,1]$, let t be large enough ($(1-\lambda )^t \approx 0$), we can immediately derive Eq. (10) from the above equation. This concludes the proof of Theorem 4.