Statistical Tests for Joint Analysis of Performance Measures
Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.
- 4.Hsiao, K.J., Xu, K., Calder, J., Hero, A.O.: Multi-criteria anomaly detection using pareto depth analysis. In: Advances in Neural Information Processing Systems, vol. 25, pp. 845–853. Curran Associates Inc (2012)Google Scholar
- 7.Rice, J.: Mathematical Statistics and Data Analysis. Cengage Learning, Belmont (2006)Google Scholar
- 14.de Campos, C.P., Ji, Q.: Roperties of Bayesian Dirichlet scores to learn Bayesian network structures. In: AAAI Conference on Artificial Intelligence, pp. 431–436. AAAI Press, 2010Google Scholar
- 15.Silander, T., Myllymaki, P.: A simple approach for finding the globally optimal bayesian network structure. In: Conference on Uncertainty in Artificial Intelligence, pp. 445–452. AUAI (2006)Google Scholar
- 16.Barlett, M., Cussens, J.: Advances in Bayesian network learning using integer programming. In: Conference on Uncertainty in Artificial Intelligence, pp. 182–191. AUAI (2013)Google Scholar
Open Access This chapter is distributed under the terms of the Creative Commons Attribution Noncommercial License, which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.