Comparison of the Novel Classification Methods on the Reuters-21578 Corpus

Abstract

The paper describes an evaluation of novel boosting methods of the commonly used Multinomial Naïve Bayes classifier. Evaluation is made upon the Reuters corpus, which consists of 10788 documents and 90 categories. All experiments use the tf-idf weighting model and the one versus the rest strategy. AdaBoost, XGBoost and Gradient Boost algorithms are tested. Additionally the impact of feature selection is tested. The evaluation is carried out with use of commonly used metrics – precision, recall, F1 and Precision-Recall breakeven points. The novel aspect of this work is that all considered boosted methods are compared to each other and several classical methods (Support Vector Machine methods and a Random Forests classifier). The results are much better than in the classic Joachims paper and slightly better than obtained with maximum discrimination method for feature selection. This is important because for the past 20 years most works were concerned with a change of results upon modification of parameters. Surprisingly, the result obtained with the use of feed-forward neural network is comparable to the Bayesian optimization over boosted Naïve Bayes (despite the medium size of the corpus). We plan to extend these results by using word embedding methods.