Functional outlier detection by a local depth with application to NO _x levels

Abstract

This paper proposes methods to detect outliers in functional data sets and the task of identifying atypical curves is carried out using the recently proposed kernelized functional spatial depth (KFSD). KFSD is a local depth that can be used to order the curves of a sample from the most to the least central, and since outliers are usually among the least central curves, we present a probabilistic result which allows to select a threshold value for KFSD such that curves with depth values lower than the threshold are detected as outliers. Based on this result, we propose three new outlier detection procedures. The results of a simulation study show that our proposals generally outperform a battery of competitors. We apply our procedures to a real data set consisting in daily curves of emission levels of nitrogen oxides (NO$_{x}$) since it is of interest to identify abnormal NO$_{x}$ levels to take necessary environmental political actions.

Appendix

1.1 From $FSD(x, Y_{n})$ to $KFSD(x, Y_{n})$

To show how to pass from $FSD(x, Y_{n})$ in (1) to $KFSD(x, Y_{n})$ in (4), we first show that $FSD(x, Y_{n})$ can be expressed in terms of inner products. We present this result for $n=2$. The norm in (1) can be written as

$$\begin{aligned} \left\| \sum _{i}^{2}\frac{x-y_{i}}{\Vert x-y_{i}\Vert }\right\| ^{2} &= \left\| \frac{x-y_{1}}{\Vert x-y_{1}\Vert }+\frac{x-y_{2}}{\Vert x-y_{2}\Vert }\right\| ^{2}\\ &= \left\| \frac{x-y_{1}}{\sqrt{\langle x,x\rangle +\langle y_{1},y_{1}\rangle -2\langle x,y_{1}\rangle }}+\frac{x-y_{2}}{\sqrt{\langle x,x\rangle +\langle y_{2},y_{2}\rangle -2\langle x,y_{2}\rangle }}\right\| ^{2} \end{aligned}$$

Let $\delta _{1}=\sqrt{\langle x,x\rangle +\langle y_{1},y_{1}\rangle -2\langle x,y_{1}\rangle }$ and $\delta _{2}=\sqrt{\langle x,x\rangle +\langle y_{2},y_{2}\rangle -2\langle x,y_{2}\rangle }$. Then,

$$\begin{aligned} \left\| \sum _{i}^{2}\frac{x-y_{i}}{\Vert x-y_{i}\Vert }\right\| ^{2} &= \left\| \frac{x-y_{1}}{\delta _{1}}+\frac{x-y_{2}}{\delta _{2}}\right\| ^{2} \\ &= \left\| \frac{x-y_{1}}{\delta _{1}}\right\| +\left\| \frac{x-y_{2}}{\delta _{2}}\right\| + \frac{2}{\delta _{1}\delta _{2}}\langle x-y_{1},x-y_{2}\rangle \\ &= 2 + \frac{2}{\delta _{1}\delta _{2}}(\langle x,x\rangle +\langle y_{1},y_{2}\rangle -\langle x,y_{1}\rangle -\langle x,y_{2}\rangle )\\ &= \sum _{i,j=1}^{2}\frac{\langle x,x\rangle +\langle y_{i},y_{j}\rangle -\langle x,y_{i}\rangle -\langle x,y_{j}\rangle }{\delta _{i}\delta _{j}},\end{aligned}$$

and apply the embedding map $\phi$ to all the observations of the last expression. According to (2), this is equivalent to substitute the inner product function with a positive definite and stationary kernel function $\kappa$, which explains the definition of $KFSD(x, Y_{n})$ in (4) for $n=2$. The generalization of this result to $n>2$ is straightforward.

1.2 Proof of theorem 1

As explained in Sect. 3, Theorem 1 is a functional extension of a result derived by Chen et al. (2009) for KSD, and since they are closely related, next we report a sketch of the proof of Theorem 1. The proof for KSD is mostly based on an inequality known as McDiarmid ’s inequality (McDiarmid 1989), which also applies to general probability spaces, and therefore to functional Hilbert spaces. We report this inequality in the next lemma:

Lemma 1

(McDiarmid [1.2]) Let $\Omega _{1}, \ldots , \Omega _{n}$ be probability spaces. Let ${\mathbf {\Omega }} = \prod _{j=1}^{n} \Omega _{j}$ and let $X: {\mathbf {\Omega} } \rightarrow {\mathbb {R}}$ be a random variable. For any $j \in \left\{ 1, \ldots , n\right\}$, let $(\omega _{1}, \ldots , \omega _{j}, \ldots ,$ $\omega _{n})$ and $\left( \omega _{1}, \ldots , \hat{\omega }_{j}, \ldots , \omega _{n}\right)$ be two elements of ${\mathbf {\Omega }}$ that differ only in their jth coordinates. Assume that X is uniformly difference-bounded by $\{c_j\}$, that is, for any $j \in \left\{ 1, \ldots , n\right\}$,

$$\left| X\left( \omega _{1}, \ldots , \omega _{j}, \ldots , \omega _{n}\right) -X\left( \omega _{1}, \ldots , \hat{\omega }_{j}, \ldots , \omega _{n}\right) \right| \le c_{j}. $$

(12)

Then, if ${\mathbb {E}}[X]$ exists, for any $\tau > 0$

$${\mathrm {Pr}}\left( X-{\mathbb {E}}[X] \ge \tau \right) \le \exp \left( \frac{-2\tau ^2}{\sum _{j=1}^{n} c_{j}^{2}}\right) .$$

In order to apply Lemma 1 to our problem, define

$$X(z_{1},\ldots ,z_{n_{Z}}) = - \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}|Y_{n_{Y}}),$$

(13)

whose expected value is given by

$${\mathbb {E}}[X] = {\mathbb {E}}_{z_{i}|Y_{n_{Y}}}\left[ - \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}|Y_{n_{Y}})\right] = - {\mathbb {E}}_{z_{1}|Y_{n_{Y}}}\left[ g(z_{1},Y_{n_{Y}}|Y_{n_{Y}})\right] . $$

(14)

Now, for any $j \in \left\{ 1, \ldots , n_{Z}\right\}$ and $\hat{z}_{j} \in {\mathbb {H}}$, the following inequality holds

$$\left| X(z_{1},\ldots ,z_{j},\ldots ,z_{n_{Z}}) - X(z_{1},\ldots ,\hat{z}_{j},\ldots ,z_{n_{Z}})\right| \le \frac{1}{n_{Z}}, $$

and it provides assumption (12) of Lemma 1. Therefore, for any $\tau > 0$

$${\mathrm {Pr}}\left( {\mathbb {E}}_{z_{1}|Y_{n_{Y}}}\left[ g(z_{1},Y_{n_{Y}}|Y_{n_{Y}})\right] - \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}|Y_{n_{Y}}) \ge \tau \right) \le \exp \left( -2n_{Z}\tau ^{2}\right) , $$

and by the law of total probability

$$\begin{aligned} \begin{array}{l} {\mathbb {E}}\left[ {\mathrm {Pr}}\left( {\mathbb {E}}_{z_{1}|Y_{n_{Y}}}\left[ g(z_{1},Y_{n_{Y}}|Y_{n_{Y}})\right] - \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}|Y_{n_{Y}}) \ge \tau \right) \right] \\ \quad = {\mathrm {Pr}}\left( {\mathbb {E}}_{z_{1}|Y_{n_{Y}}}\left[ g(z_{1},Y_{n_{Y}})\right] - \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}) \ge \tau \right) \le \exp \left( -2n_{Z}\tau ^{2}\right) \\ \end{array} \end{aligned}$$

Next, setting $\delta = \exp \left( -2n_{Z}\tau ^{2}\right)$, and solving for $\tau$, the following result is obtained:

$$\tau = \sqrt{\frac{\ln 1/\delta }{2n_{Z}}}.$$

Therefore,

$${\mathrm {Pr}}\left( {\mathbb {E}}_{z_{1}|Y_{n_{Y}}}\left[ g(z_{1},Y_{n_{Y}})\right] \le \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}) + \sqrt{\frac{\ln 1/\delta }{2n_{Z}}}\right) \ge 1-\delta . $$

(15)

However, Theorem 1 provides a probabilistic upper bound for ${\mathbb {E}}_{x|Y_{n_{Y}}}\left[ g(x,Y_{n_{Y}})\right]$. First, recall that $z_{1} \sim Y_{mix}$ and note that

$${\mathbb {E}}_{(z_{1} \sim Y_{mix})|Y_{n_{Y}}}\left[ g\left( z_{1},Y_{n_{Y}}\right) \right] = (1-\alpha ){\mathbb {E}}_{(z_{1} \sim Y_{nor})|Y_{n_{Y}}}\left[ g\left( z_{1},Y_{n_{Y}}\right) \right] + \alpha {\mathbb {E}}_{(z_{1} \sim Y_{out})|Y_{n_{Y}}}\left[ g\left( z_{1},Y_{n_{Y}}\right) \right] . $$

Then, since ${\mathbb {E}}_{(z_{1} \sim Y_{nor})|Y_{n_{Y}}}\left[ g\left( z_{1},Y_{n_{Y}}\right) \right] = {\mathbb {E}}_{x|Y_{n_{Y}}}\left[ g\left( x,Y_{n_{Y}}\right) \right]$, for $\alpha >0$,

$${\mathbb {E}}_{x|Y_{n_{Y}}}\left[ g\left( x,Y_{n_{Y}}\right) \right] \le \frac{1}{1-\alpha }{\mathbb {E}}_{(z_{1} \sim Y_{mix})|Y_{n_{Y}}}\left[ g\left( z_{1},Y_{n_{Y}}\right) \right] . $$

(16)

Consequently, combining (15) and (16), and for $r \ge \alpha$, we obtain

$${\mathrm {Pr}}\left( {\mathbb {E}}_{x|Y_{n_{Y}}}\left[ g(x,Y_{n_{Y}})\right] \le \frac{1}{1-r}\left[ \frac{1}{n_{Z}}\sum _{i=1}^{n_{Z}}g(z_{i},Y_{n_{Y}}) + \sqrt{\frac{\ln 1/\delta }{2n_{Z}}}\right] \right) \ge 1-\delta ,$$

which completes the proof. $\square$