Evaluating Hard and Soft Flat-Clustering Algorithms for Text Documents

Abstract

Document clustering refers to unsupervised classification (categorization) of documents into groups (clusters) in such a way that the documents in a cluster are similar, whereas dissimilar documents are assigned in different clusters. The documents may be web pages, blog posts, news articles, or other text files. A popular and computationally efficient clustering technique is flat clustering. Unlike hierarchical techniques, flat clustering algorithms aim to partition the document space into groups of similar documents. The cluster assignments however may be hard or soft. This paper presents our experimental work on evaluating some hard and soft flat-clustering algorithms, namely K-means, heuristic k-means and fuzzy C-means, for categorizing text documents. We experimented with different representations (tf, tf.idf, Boolean) and feature selection schemes (with or without stop word removal and with or without stemming) on some standard datasets. The results indicate that tf.idf representation and the use of stemming obtains better clustering. Moreover, fuzzy clustering obtains better results than K-means on almost all datasets, and is also a more stable method.