F-Measure Maximization in Topical Classification

  • Weiwei Cheng
  • Krzysztof Dembczyński
  • Eyke Hüllermeier
  • Adrian Jaroszewicz
  • Willem Waegeman
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7413)

Abstract

The F-measure, originally introduced in information retrieval, is nowadays routinely used as a performance metric for problems such as binary classification, multi-label classification, and structured output prediction. In this paper, we describe our methods applied in the JRS 2012 Data Mining Competition for topical classification, where the instance-based F-measure is used as the evaluation metric. Optimizing such a measure is a statistically and computationally challenging problem, since no closed-form maximizer exists. However, it has been shown recently that the F-measure maximizer can be efficiently computed if some properties of the label distribution are known. For independent labels, it is enough to know marginal probabilities. An algorithm based on dynamic programming is then able to compute the F-measure maximizer in cubic time with respect to the number of labels. For dependent labels, one needs a quadratic number (with respect to the number of labels) of parameters for the joint distribution to compute (also in cubic time) the F-measure maximizer. These results suggest a two step procedure. First, an algorithm estimating the required parameters of the distribution has to be run. Then, the inference algorithm computing the F-measure maximizer is used over these estimates. Such a procedure achieved a very satisfactory result in the JRS 2012 Data Mining Competition.

Keywords

Marginal Probability Conditional Random Field Binary Relevance Label Distribution Expect Utility Framework 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    van Rijsbergen, C.J.: Foundation of evaluation. Journal of Documentation 30(4), 365–373 (1974)CrossRefGoogle Scholar
  2. 2.
    Jansche, M.: A maximum expected utility framework for binary sequence labeling. In: ACL 2007, pp. 736–743 (2007)Google Scholar
  3. 3.
    Lewis, D.: Evaluating and optimizing autonomous text classification systems. In: SIGIR 1995, pp. 246–254 (1995)Google Scholar
  4. 4.
    Chai, A.: Expectation of F-measures: Tractable exact computation and some empirical observations of its properties. In: SIGIR 2005, pp. 593–594 (2005)Google Scholar
  5. 5.
    Dembczyński, K., Waegeman, W., Cheng, W., Hüllermeier, E.: An exact algorithm for F-measure maximization. In: NIPS 2011, 223–230 (2011)Google Scholar
  6. 6.
    Dembczyński, K., Cheng, W., Hüllermeier, E.: Bayes optimal multilabel classification via probabilistic classifier chains. In: ICML 2010, pp. 279–286 (2010)Google Scholar
  7. 7.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: ICML 2001, pp. 282–289 (2001)Google Scholar
  8. 8.
    Ghamrawi, N., McCallum, A.: Collective multi-label classification. In: CIKM 2005, pp. 195–200 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Weiwei Cheng
    • 1
  • Krzysztof Dembczyński
    • 2
  • Eyke Hüllermeier
    • 1
  • Adrian Jaroszewicz
    • 2
  • Willem Waegeman
    • 3
  1. 1.Department of Mathematics and Computer ScienceMarburg UniversityGermany
  2. 2.Institute of Computing SciencePoznań University of TechnologyPoland
  3. 3.Department of Mathematical Modelling, Statistics and BioinformaticsGhent UniversityBelgium

Personalised recommendations