Statistical Tests for Joint Analysis of Performance Measures
Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.
- 4.Hsiao, K.J., Xu, K., Calder, J., Hero, A.O.: Multi-criteria anomaly detection using pareto depth analysis. In: Advances in Neural Information Processing Systems, vol. 25, pp. 845–853. Curran Associates Inc (2012)Google Scholar
- 7.Rice, J.: Mathematical Statistics and Data Analysis. Cengage Learning, Belmont (2006)Google Scholar
- 11.Buntine, W.: Theory refinement on Bayesian networks. In: Conference on Uncertainty in Artificial Intelligence, pp. 52–60. Morgan Kaufmann (1991)Google Scholar
- 14.de Campos, C.P., Ji, Q.: Roperties of Bayesian Dirichlet scores to learn Bayesian network structures. In: AAAI Conference on Artificial Intelligence, pp. 431–436. AAAI Press, 2010Google Scholar
- 15.Silander, T., Myllymaki, P.: A simple approach for finding the globally optimal bayesian network structure. In: Conference on Uncertainty in Artificial Intelligence, pp. 445–452. AUAI (2006)Google Scholar
- 16.Barlett, M., Cussens, J.: Advances in Bayesian network learning using integer programming. In: Conference on Uncertainty in Artificial Intelligence, pp. 182–191. AUAI (2013)Google Scholar
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 2.5 International License (http://creativecommons.org/licenses/by-nc/2.5/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.