Density-Based Mining Algorithms for Dynamic Data: An Incremental Approach

Abstract

Traditionally, data mining algorithms deal with a static set of input. The major bottlenecks with this class of algorithms may include redundant computation, higher latency along with increased consumption of available resources. Given the essence of dealing with dynamic data, this work focused on designing incremental mining algorithms specifically in the field of density-based clustering and outlier detection. The primary reason being density-based algorithms shows robustness in finding clusters of varying granularity or extracting outliers from variable density regions. Through these works, we proposed incremental extensions to two density-based clustering algorithms: MBSCAN, SNN-DBSCAN and an outlier detection algorithm KNNOD. The incremental extensions to MBSCAN (iMass) and KNNOD (KAGO) are approximate in nature supporting single point insertions. While for SNN-DBSCAN, we proposed exact incremental algorithms ($\text {BISDB}_\mathrm{add}$ and $\text {BISDB}_\mathrm{del}$) supporting both insertion and removal of data in batch mode. iMass obtained a maximum efficiency upto an order of 2.28 ($\approx$191 times) maintaining a mean clustering accuracy of around 60.375%. $\text {BISDB}_\mathrm{add}$ and $\text {BISDB}_\mathrm{del}$ achieved high efficiency upto an order of 3, 4, respectively. The clusters obtained through both these algorithms were identical to SNN-DBSCAN. KAGO outperformed KNNOD by achieving a maximum efficiency upto an order of 3.9 ($\approx$8304 times) across two intrusion detection datasets and a bidding data pertaining to a search engine. On evaluating outliers using these datasets, the Rand-Index and F1-score pertaining to KAGO showed an average improved accuracy of around 3.3%.