Nearest Neighbor-Based Clustering Algorithm for Large Data Sets

Abstract

Clustering is an unsupervised learning technique in which data or objects are grouped into sets based on some similarity measure. Most of the clustering algorithms assume that the main memory is infinite and can accommodate the complete set of patterns. In reality, many applications give rise to a large set of patterns which does not fit in the main memory. When the data set is too large, much of the data are stored in the secondary memory. Input/outputs (I/O) from the disk are the major bottlenecks in designing efficient clustering algorithms for large data sets. Different designing techniques have been used to design clustering algorithms for large data sets. External memory algorithm is one class of algorithms which can be used for large data sets. These algorithms exploit the hierarchical memory structure of the computers by incorporating locality of reference directly in the algorithm. This paper contributes towards designing clustering algorithms in the external memory model (proposed by Aggarwal and Vitter) to make the algorithms scalable. In this paper, it is shown that the Shared near neighbors algorithm is not I/O efficient since the computational complexity and the I/O complexity both are same and high. The algorithm is redesigned in the external memory model reducing its I/O complexity without any change in its computational complexity. We substantiate the theoretical analysis by showing the performance of the algorithms with their traditional counterpart by implementing in STXXL library.