Machine Learning

, Volume 39, Issue 2, pp 103–134

Text Classification from Labeled and Unlabeled Documents using EM

  • Kamal Nigam
  • Andrew Kachites Mccallum
  • Sebastian Thrun
  • Tom Mitchell

DOI: 10.1023/A:1007692713085

Cite this article as:
Nigam, K., Mccallum, A.K., Thrun, S. et al. Machine Learning (2000) 39: 103. doi:10.1023/A:1007692713085


This paper shows that the accuracy of learned text classifiers can be improved by augmenting a small number of labeled training documents with a large pool of unlabeled documents. This is important because in many text classification problems obtaining training labels is expensive, while large quantities of unlabeled documents are readily available.

We introduce an algorithm for learning from labeled and unlabeled documents based on the combination of Expectation-Maximization (EM) and a naive Bayes classifier. The algorithm first trains a classifier using the available labeled documents, and probabilistically labels the unlabeled documents. It then trains a new classifier using the labels for all the documents, and iterates to convergence. This basic EM procedure works well when the data conform to the generative assumptions of the model. However these assumptions are often violated in practice, and poor performance can result. We present two extensions to the algorithm that improve classification accuracy under these conditions: (1) a weighting factor to modulate the contribution of the unlabeled data, and (2) the use of multiple mixture components per class. Experimental results, obtained using text from three different real-world tasks, show that the use of unlabeled data reduces classification error by up to 30%.

text classificationExpectation-Maximizationintegrating supervised and unsupervised learningcombining labeled and unlabeled dataBayesian learning
Download to read the full article text

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Kamal Nigam
    • 1
  • Andrew Kachites Mccallum
    • 2
    • 3
  • Sebastian Thrun
    • 4
  • Tom Mitchell
    • 5
  1. 1.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA
  2. 2.Just ResearchPittsburghUSA
  3. 3.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA
  4. 4.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA
  5. 5.School of Computer ScienceCarnegie Mellon UniversityPittsburghUSA