Filtering Very Similar Text Documents: A Case Study

  • Jiří Hroza
  • Jan Žižka
  • Aleš Bourek
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2945)


This paper describes problems with classification and filtration of similar relevant and irrelevant real medical documents from one very specific domain, obtained from the Internet resources. Besides the similarity, the documents are often unbalanced—a lack of irrelevant documents for the training. A definition of similarity is suggested. For the classification, six algorithms are tested from the document similarity point of view. The best results are provided by the back propagation-based neural network and by the radial basis function-based support vector machine.


Textual Document Common Word Internet Resource Medical Document Training Document 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
    Alexandrov, M., Gelbukh, A., Lozovoi, G.: Chi-square Classifier for Document Categorization. In: Gelbukh, A. (ed.) CICLing 2001. LNCS, vol. 2004, pp. 455–457. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  5. 5.
    Gelbukh, A., Sidorov, G., Guzmán-Arenas, A.: Use of a weighted topic hierarchy for text retrieval and classification. In: Matoušek, V., Mautner, P., Ocelíková, J., Sojka, P. (eds.) TSD 1999. LNCS (LNAI), vol. 1692, pp. 130–135. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  6. 6.
    Mitchell, T.M.: Machine Learning. McGraw Hill, New York (1997)zbMATHGoogle Scholar
  7. 7.
    Porter, M.F.: An Algorithm For Suffix Stripping. Program 14(3), 130–137 (1980)Google Scholar
  8. 8.
    Quinlan, J.R.: Bagging, Boosting, and C4.5. In: Proc. of the 8th Annual Conference on Innovative Applications of Artificial Intelligence, AAAI 1996, Portland, Oregon, August 4-8, pp. 725–730 (1996)Google Scholar
  9. 9.
    Sebastiani, F.: Machine Learning in Automated Text Categorization. ACM Computing Surveys 34(1), 1–47 (2002)CrossRefGoogle Scholar
  10. 10.
    Tong, S., Koller, D.: Support Vector Machine Active Learning with Applications to Text Classification. Journal of Machine Learning Research 2, 45–66 (2001)CrossRefGoogle Scholar
  11. 11.
    Van Rijsbergen, C.J.: Information Retrieval, 2nd edn., Department of Computer Science, University of Glasgow (1979)Google Scholar
  12. 12.
    Žižka, J., Bourek, A., Frey, L.: TEA: A Text Analysis Tool for the Intelligent Text Document Filtering. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2000. LNCS (LNAI), vol. 1902, pp. 151–156. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  13. 13.
    Žižka, J., Bourek, A.: Automated Selection of Interesting Medical Text Documents by the TEA Text Analyzer. In: Gelbukh, A. (ed.) CICLing 2002. LNCS, vol. 2276, pp. 402–404. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Jiří Hroza
    • 1
  • Jan Žižka
    • 1
  • Aleš Bourek
    • 2
  1. 1.Faculty of Informatics, Department of Information TechnologiesMasaryk UniversityBrnoCzech Republic
  2. 2.Faculty of Medicine, Department of BiophysicsMasaryk UniversityBrnoCzech Republic

Personalised recommendations