A Blended Metric for Multi-label Optimisation and Evaluation
In multi-label classification, a large number of evaluation metrics exist, for example Hamming loss, exact match, and Jaccard similarity – but there are many more. In fact, there remains an apparent uncertainty in the multi-label literature about which metrics should be considered and when and how to optimise them. This has given rise to a proliferation of metrics, with some papers carrying out empirical evaluations under 10 or more different metrics in order to analyse method performance. We argue that further understanding of underlying mechanisms is necessary. In this paper we tackle the challenge of having a clearer view of evaluation strategies. We present a blended loss function. This function allows us to evaluate under the properties of several major loss functions with a single parameterisation. Furthermore we demonstrate the successful use of this metric as a surrogate loss for other metrics. We offer experimental investigation and theoretical backing to demonstrate that optimising this surrogate loss offers best results for several different metrics than optimising the metrics directly. It simplifies and provides insight to the task of evaluating multi-label prediction methodologies. Data related to this paper are available at: http://mulan.sourceforge.net/datasets-mlc.html, https://sourceforge.net/projects/meka/files/Datasets/, http://www.ces.clemson.edu/~ahoover/stare/.
- 1.Dembczyński, K., Cheng, W., Hüllermeier, E.: Bayes optimal multilabel classification via probabilistic classifier chains. In: 27th International Conference on Machine Learning, ICML 2010, Haifa, Israel, pp, 279–286. Omnipress, June 2010Google Scholar
- 10.Read, J., Puurula, A., Bifet, A.: Multi-label classification with meta labels, In: IEEE International Conference on Data Mining (ICDM 2014), pp. 941–946. IEEE, December 2014Google Scholar
- 14.Wu, X., Zhou, Z.: A unified view of multi-label performance measures. In: ICML, vol. 70, pp. 3780–3788. PMLR (2017)Google Scholar