Nearest Neighbors via a Hybrid Approach in Large Datasets: A Speed up

Abstract

A Spatial data structure such as kd-tree is a proven data structure in searching Nearest Neighbors of a query point. However, constructing a kd-tree for determining the nearest neighbors becomes a computationally difficult task as the size of the data increases both in dimensions and the number of data points. So, we need a method that overcomes this shortcoming. This paper proposes a hybrid algorithm to speed up the process of identifying k-nearest neighbors for a given query point q. The proposed algorithm uses lightweight coreset algorithm to sample K points. These points are then used as a seed to the K-Means clustering algorithm to cluster the data points. The algorithm finally determines the nearest neighbors of a query point by searching the clusters that are closest to the query point. While analyzing the performance of the proposed algorithm, the time consumed for constructing the coreset and K-Means algorithms is not taken in to account. This is because these algorithms are used only once. The proposed method is compared with two existing algorithms in the literature. We called these two methods as “general or normal method” and “without using coresets”. The comparative results prove that the proposed algorithm reduces the time consumed to generate kd-tree and also K-Means clustering.