Abstract
In the past decades Machine Learning algorithms have been successfully used in numerous classification problems. While they usually significantly outperform domain experts (in terms of classification accuracy or otherwise), they are mostly not being used in practice. A plausible reason for this is that it is difficult to obtain an unbiased estimation of a single classification’s reliability. In the paper we propose a general transductive method for estimation of classification’s reliability on single examples that is independent of the applied Machine Learning algorithm. We compare our method with existing approaches and discuss its advantages. We perform extensive testing on 14 domains and 6 Machine Learning algorithms and show that our approach can frequently yield more than 100% improvement in reliability estimation performance.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
S. D. Bay and M. J. Pazzani. Characterizing model errors and differences. In Proc. 17th International Conf. on Machine Learning, pages 49–56. Morgan Kaufmann, San Francisco, CA, 2000.
A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the 11th Annual Conference on Computational Learning Theory, pages 92–100, 1998.
G. A. Diamond and J. S. Forester. Analysis of probability as an aid in the clinical diagnosis of coronary artery disease. New England Journal of Medicine, 300:1350, 1979.
J. Dougherty, R. Kohavi, and M. Sahami. Supervised and unsupervised discretization of continuous features. In Proc. ICML’95, pages 194–202. Morgan Kaufmann, 1995.
A. Gammerman, V. Vovk, and V. Vapnik. Learning by transduction. In Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pages 148–155, Madison, Wisconsin, 1998.
I. Kononenko, E. Šimec, and M. Robnik-Šikonja. Overcoming the myopia of inductive learning algorithms with ReliefF. Applied Intelligence, 7:39–55, 1997.
I. Kononenko. Semi-naive Bayesian classifier. In Y. Kodratoff, editor, Proc. European Working Session on Learning-91, pages 206–219, Porto, Potrugal, 1991. Springer-Verlag.
M. Kukar. Estimating classifications’ reliability. PhD thesis, University of Ljubljana, Faculty of Computer and Information Science, Ljubljana, Slovenia, 2001. In Slovene.
M. Kukar. Making reliable diagnoses with machine learning: A case study. In Silvana Quaglini, Pedro Barahona, and Steen Andreassen, editors, Proceedings of Artificial Intelligence in Medicine Europe, AIME 2001, pages 88–96, Cascais, Portugal, 2001. Springer.
M. Kukar, I. Kononenko, C. Grošelj, K. Kralj, and J. Fettich. Analysing and improving the diagnosis of ischaemic heart disease with machine learning. Artificial Intelligence in Medicine, 16 (1):25–50, 1999.
M. Li and P. Vitányi. An introduction to Kolmogorov complexity and its applications. Springer-Verlag, New York, 2nd edition, 1997.
K. Nigam, A. K. McCallum, S. Thrun, and T. Mitchell. Text classification from labeled and unlabeled documents using EM. Machine Learning, 39 (2/3):103–134, 2000.
M. Olona-Cabases. The probability of a correct diagnosis. In J. Candell-Riera and D. Ortega-Alcalde, editors, Nuclear Cardiology in Everyday Practice, pages 348–357. Kluwer, 1994.
J. Ortega, M. Koppel, and S. Argamon. Arbitrating among competing classifiers using learned referees. Knowledge and Information Systems Journal, 3:470–490, 2001.
D. E. Rumelhart and J. L. McClelland. Parallel Distributed Processing, volume 1: Foundations. MIT Press, Cambridge, 1986.
C. Saunders, A. Gammerman, and V. Vovk. Transduction with confidence and credibility. In Proceedings of the International Joint Conference on Artificial Intelligence, Stockholm, Sweden, 1999.
A. Seewald and J. Furnkranz. An evaluation of grading classifiers. In Proc. 4th International Symposium on Advances in Intelligent Data Analysis, pages 115–124, 2001.
I. J. Taneja. On generalized information measures and their applications. Adv. Electron. and Elect. Physics, 76:327–416, 1995.
K. M. Ting. Decision combination based on the characterisation of predictive accuracy. Intelligent Data Analysis, 1:181–206, 1997.
V. Vapnik. Statistical Learning Theory. John Wiley, 1998.
V. Vovk, A. Gammerman, and C. Saunders. Machine learning application of algorithmic randomness. In Proceedings of the 16th International Conference on Machine Learning (ICML’99), Bled, Slovenija, 1999.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kukar, M., Kononenko, I. (2002). Reliable Classifications with Machine Learning. In: Elomaa, T., Mannila, H., Toivonen, H. (eds) Machine Learning: ECML 2002. ECML 2002. Lecture Notes in Computer Science(), vol 2430. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36755-1_19
Download citation
DOI: https://doi.org/10.1007/3-540-36755-1_19
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44036-9
Online ISBN: 978-3-540-36755-0
eBook Packages: Springer Book Archive