Optimizing different loss functions in multilabel classifications

Abstract

Multilabel classification (ML) aims to assign a set of labels to an instance. This generalization of multiclass classification yields to the redefinition of loss functions and the learning tasks become harder. The objective of this paper is to gain insights into the relations of optimization aims and some of the most popular performance measures: subset (or 0/1), Hamming, and the example-based F-measure. To make a fair comparison, we implemented three ML learners for optimizing explicitly each one of these measures in a common framework. This can be done considering a subset of labels as a structured output. Then, we use structured output support vector machines tailored to optimize a given loss function. The paper includes an exhaustive experimental comparison. The conclusion is that in most cases, the optimization of the Hamming loss produces the best or competitive scores. This is a practical result since the Hamming loss can be minimized using a bunch of binary classifiers, one for each label separately, and therefore, it is a scalable and fast method to learn ML tasks. Additionally, we observe that in noise-free learning tasks optimizing the subset loss is the best option, but the differences are very small. We have also noticed that the biggest room for improvement can be found when the goal is to optimize an F-measure in noisy learning tasks.

Appendix

In this section, we report the results obtained in the whole collection of datasets. Since we have two different ways to introduce noise in a learning task (see Sect. 6.1), to represent all datasets at once, we define the similarity of noise-free and noisy versions for each dataset and loss or score function. For a loss function $\Delta$, the similarity is the complementary of the loss of the noisy release with respect to the noise-free output in the test set,

$$\begin{aligned} \mathrm{Sim }(\Delta ,\mathbf {Y}, \mathrm{noise}(\mathbf {Y}))\! =\! 1\!-\! \Delta (\mathbf {Y}, \mathrm{noise}(\mathbf {Y})) \end{aligned}$$

(9)

where $\mathbf {Y}$ represents the matrix of actual labels. On the other hand, the similarity does not use the complementary in $F_1$,

$$\begin{aligned} \mathrm{Sim }(F_1,\mathbf {Y}, \mathrm{noise}(\mathbf {Y})) = F_1(\mathbf {Y}, \mathrm{noise}(\mathbf {Y})). \end{aligned}$$

(10)

In Fig. 3, each dataset is represented in a 2-dimension space. The horizontal axis represents the $F_1$ score achieved by Struct($F_1$) in the noise-free version of the dataset. The vertical axis represents the similarity of the dataset (measured with $F_1$). Thus, points near the top of the picture stand for noise-free datasets or datasets with low noise. On the other hand, points near the left side represent harder datasets, in the sense that the noise-free releases achieves lower $F_1$ scores. Finally, the points in this space are labeled by the name of the learner that achieved the best $F_1$ score.

Fig. 3

Learners winning in $F_1$ score in the 528 datasets. The horizontal axis represents the $F_1$ score of datasets in the noise-free releases. Thus points at the right hand side stand for easier datasets, typically those with less number of labels. The vertical axis represent the similarity of label sets, using $F_1$ similarity (10), with the noise-free version. The higher the points in the figure, the lower the noise in the datasets

Here, we observe that Struct($F_1$) outperforms the other learners in terms of $F_1$ when tackling harder learning tasks (left bottom corner of Fig. 3). In easier tasks, mainly those with $10$ or $25$ labels, the procedure (Algorithm 2) seems to require more evidences to estimate the optimal expected $F_1$. In any case, the differences in the easiest learning tasks are small.

To complete the discussion of the results, we made figures analogous to Fig. 3 using subset $0/1$ loss (Fig. 4), and Hamming loss (Fig. 5).

Fig. 4

Learners winning in $0/1$ loss in the 528 datasets. The horizontal axis represents the $0/1$ loss of datasets in the noise-free releases. Thus points at the left-hand side stand for easier datasets, typically those with less number of labels. The vertical axis represent the similarity of label sets, using $0/1$ similarity (9), with the noise-free version. The higher the points in the figure, the lower the noise in the datasets

Fig. 5

Learners winning in Hamming loss in the 528 datasets. The horizontal axis represents the Hamming loss of datasets in the noise-free releases. Thus points at the left-hand side stand for easier datasets, typically those with less number of labels. The vertical axis represent the similarity of label sets, using Hamming loss similarity (9), with the noise-free version. The higher the points in the figure, the lower the noise in the datasets