A meta-level analysis of online anomaly detectors

Abstract

Real-time detection of anomalies in streaming data is receiving increasing attention as it allows us to raise alerts, predict faults, and detect intrusions or threats across industries. Yet, little attention has been given to compare the effectiveness and efficiency of anomaly detectors for streaming data (i.e., of online algorithms). In this paper, we present a qualitative, synthetic overview of major online detectors from different algorithmic families (i.e., distance, density, tree or projection based) and highlight their main ideas for constructing, updating and testing detection models. Then, we provide a thorough analysis of the results of a quantitative experimental evaluation of online detection algorithms along with their offline counterparts. The behavior of the detectors is correlated with the characteristics of different datasets (i.e., meta-features), thereby providing a meta-level analysis of their performance. Our study addresses several missing insights from the literature such as (a) how reliable are detectors against a random classifier and what dataset characteristics make them perform randomly; (b) to what extent online detectors approximate the performance of offline counterparts; (c) which sketch strategy and update primitives of detectors are best to detect anomalies visible only within a feature subspace of a dataset; (d) what are the trade-offs between the effectiveness and the efficiency of detectors belonging to different algorithmic families; (e) which specific characteristics of datasets yield an online algorithm to outperform all others.

Appendices

Appendix

Appendix 1: Offline detectors

Based on the findings of previous benchmarking efforts [15, 24, 25] we considered representative offline detectors from three families, namely, distance-based like the weighed $\hbox {KNN}_W$ [51], density-based like LOF [14] and tree-based like IF [43] and OCRF [29].

1.1 Weighted K nearest neighbor ($\hbox {KNN}_{W}$)

$\hbox {KNN}_{W}$ is a score-based variation [27] of the distance-based K nearest neighbors (KNN) [51] classifier. An anomaly is a sample that is in substantially higher distance than the rest samples (i.e., they have few neighbors in close distance).

To assign a real-valued score for each sample, it computes a matrix of distances, called MD: each row represents a sample and each column represents the distance from another sample. Then, using MD it computes the score of a sample p as the maximum distance dist over its K (columns) nearest neighbors as follows:

$$\begin{aligned} score(p) = max(dist(p, q)) : 1 \le q \le K \end{aligned}$$

(A1)

Higher score indicates more abnormality. The number of nearest neighbors K and the distance metric dist($\cdot$) are the two hyper-parameters of $\hbox {KNN}_W$. Its effectiveness depends on how carefully K is chosen w.r.t. the data characteristics. For a dataset of size N, $\hbox {KNN}_{W}$ needs quadratic time $O(N^2)$ to construct MD and linear time O(NK) to score each sample.

1.2 Local outlier factor (LOF)

LOF [14] is a density-based detector which computes the local density deviation of a sample w.r.t. its neighbors. Essentially, LOF considers an anomaly any sample that lies on a sparse area while its nearest neighbors lie on dense areas.

LOF first computes the reachability distance rd between a given sample p and another sample o according to Eq. A2, where dist(p, o) is the direct distance between the two samples and k-dist(p,o) is the distance of o to its k-th nearest neighbor.

$$\begin{aligned} rd(p, o) = max(dist(p, o), \text {k-}dist(o)) \end{aligned}$$

(A2)

Then, LOF computes the local reachability density of a sample p as the inverse average reachability distance of p from its k-nearest neighbors KNN(p):

$$\begin{aligned} lrd(p) = \frac{1}{\frac{1}{\vert KNN(p) \vert } \sum _{o \in KNN(p) } rd(p, o) }. \end{aligned}$$

(A3)

The final score of a sample p is computed by comparing to the average local reachability density of its neighbors:

$$\begin{aligned} LOF(p) = \frac{1}{\vert KNN(p) \vert } \sum _{o \in KNN(o) } \frac{ lrd(o) }{ lrd(p) } \end{aligned}$$

(A4)

The number of nearest neighbors K and the distance metric $dist (\cdot )$ are the two hyper-parameters of LOF. The effectiveness of the algorithm strongly depends on the careful selection of K. LOF needs quadratic time $O(N^2)$ to compute the distances between each pair of samples and linear time O(N) to score samples, where N is the dataset size.

1.3 Isolation forest (IF)

IF is a tree-based ensemble detector [43], which relies on the number of partitions needed to isolate a sample from the rest in a dataset. The less partitions needed to isolate, the more likely a sample is to be an anomaly.

The algorithm uses a forest of random Trees built on samples of the dataset using bootstrapping of size Max Samples. For each sub-sample, it constructs an isolation tree by uniformly selecting features and their split values as internal nodes. Samples are stored in the leafs, and the actual height of the tree could be limited up to a Max Height. The length of the path from the tree root to a leaf node, measures how well a sample in that node is isolated from the others: short paths provide evidence of anomalies. The outlyingness score of a sample p is then computed by averaging its path length over all Isolation Trees in the forest as follows:

$$\begin{aligned} score(p, n) = 2^{- E(!t(p))/c(n)} \end{aligned}$$

(A5)

where n is the max samples size, E(!t(p)) is the average of the actual height !t(p)²⁵ and c(n) is used for normalization:

$$\begin{aligned} c(n) = 2 (ln(n - 1) + 0.5772156649) - 2 (n - 1) / n \end{aligned}$$

(A6)

The closer to 1 the score is, the more probable the sample to be an anomaly. A score much smaller than 0.5 indicates an inlier.

The number of Trees t, the number of samples a tree may contain Max Samples m, and its Max Height !t are the three hyper-parameters of IF. Max Height is typically set to $\lceil log(n) \rceil )$ while the variance of scores is usually minimized with  100 random Trees. By avoiding costly distance or density calculation, IF requires linear time O(tmh) to build a forest and linear time O(tnh) to score n samples.

1.4 One-class random forest (OCRF)

OCRF is a tree-based ensemble detector [29], extension of the Random Forest. It relies on the adaption of two-class splitting criterion to one-class setting, by assuming an adaptive distribution of anomalies in each node, i.e., assuming the number of anomalies are equal to the number of normal samples in each node. After splitting, one child node captures the maximum amount of samples in a minimal space, while the other the exact opposite. The closer the leaf containing a sample is to the root of the tree, the more likely this sample is to be an anomaly.

Similarly to IF, it builds a forest of Trees built on samples and features of the dataset using bootstrapping of size Max Samples and Max Features, respectively. For each sub-sample, it constructs a tree by selecting the feature and split value that minimizes the one-class Gini improvement proxy. One-class Gini improvement proxy is given by the following formula:

$$\begin{aligned} I_G^{OC-ad}(t_L, t_R) = \frac{n_{t_L} \gamma n_t \lambda _L}{n_t{_L} + \gamma n_t \lambda _L} + \frac{n_{t_R} \gamma n_t \lambda _R}{n_t{_R} + \gamma n_t \lambda _R} \end{aligned}$$

(A7)

where $n_t, n_t{_L}$ and $n_t{_R}$ are the number of samples on the node t and its children nodes (L, R), respectively, $\lambda _L = Leb(X_{t_L}/ Leb(X_t))$ and $\lambda _R = Leb(X_{t_R}/ Leb(X_t))$ are the volume proportions of the children nodes and $\gamma$ is a parameter which influences the optimal splits.

Table 9 The values of the varying and fixed hyper-parameters

The samples are stored in the leafs, and the actual height of the tree could be limited up to a Max Height. The length of the path from the tree root to a leaf node, measures how well a sample in that node is isolated from the others: short paths provide evidence of anomalies. The anomalousness score of a sample p is then computed by averaging its path length over all trees in the forest as follows:

$$\begin{aligned} log_2 score(x) = - \left( \sum _{t leaves}{\mathbbm {1}_{\{x\epsilon t\}}d_t + c(n_t)}\right) / c(n) \end{aligned}$$

(A8)

where $d_t$ is the depth of node t, and $c(n) = 2H(n-1)-2(n-1)/n$, !t(i) being the harmonic number.

The closer to 1 the score is, the more probable the sample to be an anomaly. A score much smaller than 0.5 indicates an inlier.

The number of Trees t, the number of samples a tree may contain Max Samples m, the number of features for those samples Max Features f and its Max Height !t are the three hyper-parameters of OCRF. Similarly to IF, Max Height is typically set to $\lceil log (n) \rceil$ while the variance scores is usually minimized with  100 random Trees. OCRF requires linear time O(tmfh) to build a forest and linear time O(tnh) to score n samples.

Appendix 2: Hyper-parameters

We are interested in assessing the effectiveness of a detector under optimal conditions. In this respect, we distinguish between hyper-parameters whose values can be set independently of the datasets and those for which we need to estimate their values per dataset.²⁶

Table 9 presents the fixed values of various hyper-parameters we set in our experiments (e.g., training/testing splitting of datasets, window size and slide) as well as the hyper-parameters of detectors along with the range of values we tested per dataset using random search²⁷ (RS) [52]. The employed ranges empirically ensure the existence of at least an optimal value per detector and dataset. To respect the isolation requirement [34, 39], the optimization phase should take place on an additional validation partition, with samples of the training partition that remain unseen by testing. According to Tables 10–14, the optimal hyper-parameters of each detector per dataset are different in most cases to the default values originally proposed.

Table 10 Optimal hyper-parameter values of each online detector per dataset after tuning

Table 11 Optimal hyper-parameter values of online detectors HST/F, RRCF and RS-Hash per Exathlon dataset after tuning (only if the AUC ROC score of the detectors is greater than 0.5)

We are additionally interested in assessing how sensitive detectors are to the tuning of their hyper-parameters. In this respect, we are computing the average coefficient of variation²⁸ of MAP scores of the models tuned with different hyper-parameters across all datasets. As we can see in Fig. 17, for most online detectors (besides MCOD, LEAP and RS-HASH) the average coefficient is below or close to 0.10. Recall that the tinny performance differences between models are mostly due to the non-deterministic nature of the detectors. On the other hand, offline detectors’ average coefficient of variation lies on average around 0.2 and above, revealing that no matter their anomalousness criteria, they are a lot more sensitive to their hyper-parameters tuning. As we can see from Table 10, distance-based detectors’ hyper-parameters, differ in every dataset, which reveals the need for good tuning according to the characteristics of each dataset (value range, average distance, etc.). An interesting observation is that MCOD and CPOD succeeds to have a smaller average coefficient of variation w.r.t. offline detectors (while LEAP is also close). This is attributed to the fact that they exhibit a similar performance to the Random Classifier in most configurations where the two hyper-parameter values are not close to the optimal ones that to a lower coefficient of variation than expected. On the contrary, tuning does not significantly improve the effectiveness of tree-based and projection-based online detectors.

Regarding hyper-parameters with fixed values across all datasets, we are finally interested in investigating the impact of the window size on detectors’ effectiveness. To this end, we tested our detectors on four representative datasets w.r.t. the number of samples/features: Isolet (many samples/features), Wilt (many samples/few features), InternetAds (few samples/many features) and Diabetes (few samples/features). Figure 16 depicts the median ratio of the MAP scores with windows size (ws) = 128 as baseline that we use in our experiments. In terms of the window type, we used sliding windows on RRCF, MCOD, LEAP, CPOD, STARE and RS-HASH and tumbling windows on the remaining detectors. According to our experiments, we found that the previously mentioned detectors are performing better on sliding windows compared to tumbling. Note that the window slide indicates the number of past points to forget from the models built. HST exhibits a slightly better performance on ws = 128 especially on the datasets with few samples. L-S has a major performance boost using ws = 128 on datasets with low samples, while its performance remains stable on the rest. RRCF performs better on low sample datasets with a window size of 64, while HSTF has a performance boost on datasets with many features using ws = 256. RS-HASH is performing better using ws = 128 on all cases besides on datasets with many samples and few features. CPOD, LEAP and STARE exhibit better effectiveness using ws = 128 in all cases. X-S is stable throughout all window sizes.

Table 12 Offline detectors’ number of wins (using AP Scores) and average AP difference from the winner per dataset

Table 13 Offline detectors’ number of wins (using AUC Scores) and average AUC ROC difference from the winner per datase

Table 14 Optimal hyper-parameter values of each offline detector per dataset after tuning

To ensure a fair comparison of detectors in the remaining of our work, we avoid varying the window size per dataset. In this respect, we set the windows size to 128 as it proves to be optimal in most pairs of datasets and detectors. Moreover, we did not choose 256 (or greater) window size, because we need to ensure at least one full window for testing after the training phase; note that the dataset with the lowest number of samples in our benchmark is Ionosphere with 351 samples.

Fig. 16

Median Ratio of MAP scores on different window sizes (64,256) with 128 as baseline using (Diabetes, Isolet, InternetAds, Wilt)

Fig. 17

Average Coefficient of variation across all datasets of MAP scores of the models built with different hyper-parameters: Online detectors appear to be less sensitive

Appendix 3: AUC/MAP of detectors

For the shake of completeness, we are also investigating the performance ranking of online detectors. Table 12 depicts the total number of wins of offline detectors using MAP. X-B and LOF outperform the remaining detectors in 7 datasets, respectively. This places LOF the best performing proximity-based detector and X-B the best ensemble-based detector. KNN is best performing in 3 datasets along with OCRF and IF. The worst performing detector is L-B, managing to lead only in 1 dataset. It is worth mentioning that KNN has the lowest average difference from the leader along with LOF at 9.8%. There are no major changes in Table 13, which illustrates the number of wins using AUC ROC. LOF achieves to get more wins in AUC ROC, being the best detector, leading in 9 out of 24 datasets compared to X-B getting 2 less wins than before. OCRF dropped to just one win, as well as having the highest average difference from leader. It is worth mentioning that OCRF does not manage to make any split in some datasets, due to the high range of features, which leads to an infinite volume ($> 10^{308}$).

Table 15 (M)AP Scores of online and offline detectors per dataset

Table 16 AUC Scores of online and offline detectors per dataset

As we did on the previous section (Sect. 4.2) we use the nonparametric Friedman test [23], in order determine if there are any significant difference between the average ranks of the detectors. With a $p value < 0.000$ we reject the null hypothesis with a confidence level of 5% that all detectors’ performances are the same. Subsequently, we use the post hoc Nemenyi test, in order to compare the detectors in pairs. There is a significant difference when the difference between the average ranks of two detectors is higher than a critical distance (CD) of 1.539 (for 6 detectors on 25 datasets at a significance level a = 0.05) Fig. 18a illustrates the ranking of detectors w.r.t. MAP scores. KNN and X-B are both ranked first (rank tie) followed closely by LOF. KNN is ranked higher than LOF, while having almost half of the wins, due to its low average difference from leader. We observe that there is a statistically significant difference between the first three detectors (LOF,KNN,X-B) and both OCRF and L-B which come last. As we can see in Fig. 18b, there are no drastic changes on ranking when using AUC ROC. LOF is first as expected from the previous results (9/24 wins) followed by KNN and X-B. There is a statistically significant difference between the two proximity-based detectors (LOF, KNN) and both L-B and OCRF, while IF and X-B have a statistical significant difference only with OCRF.

Fig. 18

Offline detectors’ ranking

In overall, we observe that the two proximity-based detectors (LOF, KNN) ,as well as X-B, outperform all other detectors, with L-B and OCRF exhibiting the worst performance. LOF is performing better when using AUC ROC, while the rest of the detectors remain stable w.r.t both metrics.

Appendix 4: Meta-features

Table 17 Statistically significant correlations between the ratio of X-S divided by the respective online detector and meta-features. We report only pairs that had a statistically significant correlation at a significance level of 0.05