F-Measure Maximization in Topical Classification
The F-measure, originally introduced in information retrieval, is nowadays routinely used as a performance metric for problems such as binary classification, multi-label classification, and structured output prediction. In this paper, we describe our methods applied in the JRS 2012 Data Mining Competition for topical classification, where the instance-based F-measure is used as the evaluation metric. Optimizing such a measure is a statistically and computationally challenging problem, since no closed-form maximizer exists. However, it has been shown recently that the F-measure maximizer can be efficiently computed if some properties of the label distribution are known. For independent labels, it is enough to know marginal probabilities. An algorithm based on dynamic programming is then able to compute the F-measure maximizer in cubic time with respect to the number of labels. For dependent labels, one needs a quadratic number (with respect to the number of labels) of parameters for the joint distribution to compute (also in cubic time) the F-measure maximizer. These results suggest a two step procedure. First, an algorithm estimating the required parameters of the distribution has to be run. Then, the inference algorithm computing the F-measure maximizer is used over these estimates. Such a procedure achieved a very satisfactory result in the JRS 2012 Data Mining Competition.
KeywordsMarginal Probability Conditional Random Field Binary Relevance Label Distribution Expect Utility Framework
Unable to display preview. Download preview PDF.
- 2.Jansche, M.: A maximum expected utility framework for binary sequence labeling. In: ACL 2007, pp. 736–743 (2007)Google Scholar
- 3.Lewis, D.: Evaluating and optimizing autonomous text classification systems. In: SIGIR 1995, pp. 246–254 (1995)Google Scholar
- 4.Chai, A.: Expectation of F-measures: Tractable exact computation and some empirical observations of its properties. In: SIGIR 2005, pp. 593–594 (2005)Google Scholar
- 5.Dembczyński, K., Waegeman, W., Cheng, W., Hüllermeier, E.: An exact algorithm for F-measure maximization. In: NIPS 2011, 223–230 (2011)Google Scholar
- 6.Dembczyński, K., Cheng, W., Hüllermeier, E.: Bayes optimal multilabel classification via probabilistic classifier chains. In: ICML 2010, pp. 279–286 (2010)Google Scholar
- 7.Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: ICML 2001, pp. 282–289 (2001)Google Scholar
- 8.Ghamrawi, N., McCallum, A.: Collective multi-label classification. In: CIKM 2005, pp. 195–200 (2005)Google Scholar