Scalable Text Classification with Sparse Generative Modeling

  • Antti Puurula
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7458)


Machine learning technology faces challenges in handling “Big Data”: vast volumes of online data such as web pages, news stories and articles. A dominant solution has been parallelization, but this does not make the tasks less challenging. An alternative solution is using sparse computation methods to fundamentally change the complexity of the processing tasks themselves. This can be done by using both the sparsity found in natural data and sparsified models. In this paper we show that sparse representations can be used to reduce the time complexity of generative classifiers to build fundamentally more scalable classifiers. We reduce the time complexity of Multinomial Naive Bayes classification with sparsity and show how to extend these findings into three multi-label extensions: Binary Relevance, Label Powerset and Multi-label Mixture Models. To provide competitive performance we provide the methods with smoothing and pruning modifications and optimize model meta-parameters using direct search optimization. We report on classification experiments on 5 publicly available datasets for large-scale multi-label classification. All three methods scale easily to the largest available tasks, with training times measured in seconds and classification times in milliseconds, even with millions of training documents, features and classes. The presented sparse modeling techniques should be applicable to many other classifiers, providing the same types of fundamental complexity reductions when applied to large scale tasks.


sparse modeling multi-label mixture model generative classifiers Multinomial Naive Bayes sparse representation scalable computing big data 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Maron, M.E.: Automatic indexing: An experimental inquiry. J. ACM 8, 404–417 (1961)zbMATHCrossRefGoogle Scholar
  2. 2.
    McCallum, A., Nigam, K.: A comparison of event models for Naive Bayes text classification. In: AAAI 1998 Workshop on Learning for Text Categorization, pp. 41–48. AAAI Press (1998)Google Scholar
  3. 3.
    Rennie, J.D., Shih, L., Teevan, J., Karger, D.R.: Tackling the poor assumptions of naive bayes text classifiers. In: ICML 2003, pp. 616–623 (2003)Google Scholar
  4. 4.
    Jones, K.S.: A Statistical Interpretation of Term Specificity and its Application in Retrieval. Journal of Documentation 28(1), 11–21 (1972)CrossRefGoogle Scholar
  5. 5.
    Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 1996, pp. 21–29. ACM, New York (1996)CrossRefGoogle Scholar
  6. 6.
    Shanks, V.R., Williams, H.E., Cannane, A.: Indexing for fast categorisation. In: Proceedings of the 26th Australasian Computer Science Conference, ACSC 2003, vol. 16, pp. 119–127. Australian Computer Society, Inc., Darlinghurst (2003)Google Scholar
  7. 7.
    Tsoumakas, G., Katakis, I., Vlahavas, I.P.: Mining multi-label data. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 667–685. Springer (2010)Google Scholar
  8. 8.
    Godbole, S., Sarawagi, S.: Discriminative methods for multi-labeled classification, pp. 22–30 (2004)Google Scholar
  9. 9.
    Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recognition 37(9), 1757 (2004)CrossRefGoogle Scholar
  10. 10.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: A Review of Multi-Label Classification Methods. In: Proceedings of the 2nd ADBIS Workshop on Data Mining and Knowledge Discovery, ADMKD 2006, pp. 99–109 (2006)Google Scholar
  11. 11.
    Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier Chains for Multi-label Classification. In: Buntine, W., Grobelnik, M., Mladenić, D., Shawe-Taylor, J. (eds.) ECML PKDD 2009, Part II. LNCS, vol. 5782, pp. 254–269. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  12. 12.
    McCallum, A.: Multi-label text classification with a mixture model trained by EM. In: Proceedings of the AAAI 1999 Workshop on Text Learning (1999)Google Scholar
  13. 13.
    Ueda, N., Saito, K.: Parametric mixture models for multi-labeled text. In: Advances in Neural Information Processing Systems, vol. 15, pp. 721–728. MIT Press (2002)Google Scholar
  14. 14.
    Wang, H., Huang, M., Zhu, X.: A generative probabilistic model for multi-label classification. In: Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, pp. 628–637. IEEE Computer Society, Washington, DC (2008)CrossRefGoogle Scholar
  15. 15.
    Powell, M.J.D.: Direct search algorithms for optimization calculations. Acta Numerica 7, 287–336 (1998)CrossRefGoogle Scholar
  16. 16.
    Favreau, R.R., Franks, R.G.: Statistical optimization. In: Proceedings Second International Analog Computer Conference (1958)Google Scholar
  17. 17.
    Brunato, M., Battiti, R.: Rash: A self-adaptive random search method. In: Cotta, C., Sevaux, M., Sörensen, K. (eds.) Adaptive and Multilevel Metaheuristics. SCI, vol. 136, pp. 95–117. Springer (2008)Google Scholar
  18. 18.
    Loza Mencía, E., Fürnkranz, J.: Efficient Multilabel Classification Algorithms for Large-Scale Problems in the Legal Domain. In: Francesconi, E., Montemagni, S., Peters, W., Tiscornia, D. (eds.) Semantic Processing of Legal Texts. LNCS, vol. 6036, pp. 192–215. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  19. 19.
    Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: A New Benchmark Collection for Text Categorization Research. J. Mach. Learn. Res. 5, 361–397 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Antti Puurula
    • 1
  1. 1.Department of Computer ScienceThe University of WaikatoHamiltonNew Zealand

Personalised recommendations