When analyzing a document collection, a key piece of information is the number of distinct topics it contains. Document clustering has been used as a tool to facilitate the extraction of such information. However, existing clustering methods do not take into account the sequences of the words in the documents, and usually do not have the means to describe the contents within each topic cluster. In this paper, we record our investigation and results using Maximal Frequent word Sequences (MFSs) as building blocks in identifying distinct topics. The supporting documents of MFSs are grouped into an equivalence class and then linked to a topic cluster, and the MFSs serve as the document cluster identifier. We describe the original method in extracting the set of MFSs, and how it can be adapted to identify topics in a textual dataset. We also demonstrate how the MFSs themselves can act as topic descriptors for the clusters. Finally, the benchmarking study with other existing clustering methods, i.e. k-Means and EM algorithm, shows the effectiveness of our approach for topic detection.


Equivalence Class Association Rule Expectation Maximization Expectation Maximization Algorithm Document Cluster 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agrawal, R., Imieliński, T., Swami, A.: Mining association rules between sets of items in large databases. In: Proceedings of the 1993 ACM SIGMOD international conference on Management of data (1993)Google Scholar
  2. 2.
    Ahonen-Myka, H.: Finding All Frequent Maximal Sequences in Text. In: Proceedings of the 16th International Conference on Machine Learning ICML 1999 Workshop on Machine Learning in Text Data Analysis (1999)Google Scholar
  3. 3.
    Ahonen-Myka, H., Heinonen, O., Klemettinen, M., Verkamo, A.I.: Finding Cooccurring Text Phrases by Combining Sequence and Frequent Set Discovery. In: Proceedings of 16th International Joint Conference on Artificial Intelligence IJCAI 1999 Workshop on Text Mining: Foundations, Techniques and Applications (1999)Google Scholar
  4. 4.
    Allan, J., Carbonell, J., Doddington, G., Yamron, J., Yang, Y.: Topic Detection and Tracking Pilot Study Final Report. In: Proceedings of the DARPA Broadcast News Transcription and Understanding Workshop (1998)Google Scholar
  5. 5.
    Baeza-Yates, R., Ribeiro-Neto, B.: Modern information retrieval. Addison-Wesley Longman Publishing Co., Inc., Boston (1999)Google Scholar
  6. 6.
    Maarek, Y.S., Wecker, A.J.: The Librarian’s Assistant: automatically organizing on-line books into dynamic bookshelves. In: Proceedings of RIAO 1994, Intelligent Multimedia, Information Retrieval Systems and Management (1994)Google Scholar
  7. 7.
    Mannila, H., Toivonen, H., Verkamo, I.: Efficient algorithms for discovering association rules. In: AAAI Workshop on Knowledge Discovery in Databases (KDD 1994) (1994)Google Scholar
  8. 8.
    Salton, G., Buckley, C.: Term Weighting Approaches in Automatic Text Retrieval. Information Processing and Management 24, 513–523 (1988)CrossRefGoogle Scholar
  9. 9.
    Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1983)MATHGoogle Scholar
  10. 10.
    Seo, Y.-W., Sycara, K.: Text Clustering for Topic Detection. CMU-RI-TR-04-03, Robotics Institute, Carnegie Mellon University (2004)Google Scholar
  11. 11.
    Willett, P.: Recent trends in hierarchic document clustering: a critical review. Information Processing and Management 24, 577–597 (1988)CrossRefGoogle Scholar
  12. 12.
    Zamir, O., Etzioni, O.: Web document clustering: a feasibility demonstration. In: Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Ivan Yap
    • 1
  • Han Tong Loh
    • 1
  • Lixiang Shen
    • 2
  • Ying Liu
    • 3
  1. 1.Department of Mechanical Engineering, Blk EA 07-08National University of SingaporeSingapore
  2. 2.Design Technology Institute Ltd, Faculty of Engineering, Blk E4 01-07National University of SingaporeSingapore
  3. 3.Singapore MIT Alliance, E4-04-10National University of SingaporeSingapore

Personalised recommendations