Document Difficulty Framework for Semi-automatic Text Classification

  • Miguel Martinez-Alvarez
  • Alejandro Bellogin
  • Thomas Roelleke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8057)


Text Classification systems are able to deal with large datasets, spending less time and human cost compared with manual classification. This is achieved, however, in expense of loss in quality. Semi-Automatic Text Classification (SATC) aims to achieve high quality with minimum human effort by ranking the documents according to their estimated certainty of being correctly classified. This paper introduces the Document Difficulty Framework (DDF), a unification of different strategies to estimate the document certainty, and its application to SATC. DDF exploits the scores and thresholds computed by any given classifier. Different metrics are obtained by changing the parameters of the three levels the framework is lied upon: how to measure the confidence for each document-class (evidence), which classes to observe (class) and how to aggregate this knowledge (aggregation). Experiments show that DDF metrics consistently achieve high error reduction with large portions of the collection being automatically classified. Furthermore, DDF outperforms all the reported SATC methods in the literature.


Human Expert Evidence Level Correct Class Positive Label Thresholding Strategy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berardi, G., Esuli, A., Sebastiani, F.: A utility-theoretic ranking method for semi-automated text classification. In: SIGIR (2012)Google Scholar
  2. 2.
    Buckley, C., Salton, G., Allan, J.: The effect of adding relevance information in a relevance feedback environment. In: SIGIR (1999)Google Scholar
  3. 3.
    Chang, C.-C., Lin, C.-J.: LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (2011)Google Scholar
  4. 4.
    Esuli, A., Sebastiani, F.: Active learning strategies for multi-label text classification. In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 102–113. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  5. 5.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. SIGKDD Explor. Newsl (2009)Google Scholar
  6. 6.
    Larkey, L.S., Croft, W.B.: Combining classifiers in text categorization. In: SIGIR (1996)Google Scholar
  7. 7.
    Lewis, D.D., Gale, W.A.: A sequential algorithm for training text classifiers. In: SIGIR (1994)Google Scholar
  8. 8.
    Martinez-Alvarez, M., Yahyaei, S., Roelleke, T.: Semi-automatic document classification: Exploiting document difficulty. In: Baeza-Yates, R., de Vries, A.P., Zaragoza, H., Cambazoglu, B.B., Murdock, V., Lempel, R., Silvestri, F. (eds.) ECIR 2012. LNCS, vol. 7224, pp. 468–471. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. (2002)Google Scholar
  10. 10.
    Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. (2002)Google Scholar
  11. 11.
    Yang, B., Sun, J.-T., Wang, T., Chen, Z.: Effective multi-label active learning for text classification. In: SIGKDD (2009)Google Scholar
  12. 12.
    Yang, Y.: A study on thresholding strategies for text categorization. In: SIGIR (2001)Google Scholar
  13. 13.
    Yang, Y., Liu, X.: A re-examination of text categorization methods. In: SIGIR (1999)Google Scholar

Copyright information

© Springer-Verlag GmbH Berlin Heidelberg 2013

Authors and Affiliations

  • Miguel Martinez-Alvarez
    • 1
  • Alejandro Bellogin
    • 2
  • Thomas Roelleke
    • 1
  1. 1.Queen Mary, University of LondonUK
  2. 2.Centrum Wiskunde & Informatica (CWI)The Netherlands

Personalised recommendations