A Technique to Find Out Low Frequency Rare Words in Medical Cancer Text Document Classification

Abstract

A vast amount of digital medical documents are increasing day by day, and there is need of automatic text document classification. Medical research persons, doctors, and medical community search or classify their relevant documents. The documents can be medical research papers, articles, reports, surveys, etc. In this paper, we have investigated that tradition classification method applied on medical data and removed rare low frequency words that degrade performance of classifiers. We find that rare words are important in medical domain and study existing methods to find rare words. The available methods are fixed statistical calculation-based threshold value for all dataset or sample collection. So, we proposed a method for rare word finding using dynamic threshold calculation based on term frequency as well as inverse documents frequency and medical dictionary words matching concept. We have taken two real medical text dataset and applied three text classifiers kNN, NB, and SVM. The results shown that our method finds right rare words. Considering only rare words gives same or nearer accuracy of all features in classification. It also shows that removing rare words degrades performance of classifiers in most of the cases specific in medical domain.