Skip to main content
SpringerLink
Log in

Transductive Reliability Estimation for Individual Classifications in Machine Learning and Data Mining

  • Conference paper
  • First Online:
Reliable Knowledge Discovery

  • 551 Accesses

Abstract

Machine learning and data mining approaches are nowadays being used in many fields as valuable data analysis tools. However, their serious practical use is affected by the fact, that more often than not, they cannot produce reliable and unbiased assessments of their predictions’ quality. In last years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon the algorithmic theory of randomness, such as (historically ordered) transduction-based confidence estimation, typicalness-based confidence estimation, and transductive reliability estimation. In the chapter we describe typicalness and transductive reliability estimation frameworks and propose a joint approach that compensates their weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation into a joint confidence machine. The resulting confidence machine produces confidence values in the statistical sense (e.g., a confidence level of 95% means that in 95% the predicted class is also a true class), as well as provides us with a general principle that is independent of to the particular underlying classifier.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bay, S.D., Pazzani, M.J.: Characterizing model errors and differences. In: Proc. 17th International Conf. on Machine Learning, pp. 49–56. Morgan Kaufmann, San Francisco, CA (2000)

    Google Scholar 

  2. Birattari, M., Bontempi, H., Bersini, H.: Local learning for data analysis. In: Proceedings of the 8th Belgian-Dutch Conference on Machine Learning, pp. 55–61. Wageningen, The Netherlands (1998)

    Google Scholar 

  3. Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: P. Bartlett, Y. Mansour (eds.) Proceedings of the 11th Annual Conference on Computational Learning Theory, pp. 92–100. ACM Press, New York, USA, Madison, Wisconsin (1998)

    Google Scholar 

  4. Bosni’c, Z., Kononenko, I.: Estimation of individual prediction reliability using the local sensitivity analysis. Appl. intell. 29(3), 187–203 (2008)

    Google Scholar 

  5. Bousquet, O., Elisseeff, A.: Stability and generalization. Journal of Machine Learning Research 2, 499–526 (2002)

    MathSciNet  MATH  Google Scholar 

  6. Breierova, L., Choudhari, M.: An introduction to sensitivity analysis. MIT System Dynamics in Education Project (1996)

    Google Scholar 

  7. Carney, J., Cunningham, P.: Confidence and prediction intervals for neural network ensembles. In: Proceedings of the International Joint Conference on Neural Networks, pp. 1215–1218. Washington, USA (1999)

    Google Scholar 

  8. Diamond, G.A., Forester, J.S.: Analysis of probability as an aid in the clinical diagnosis of coronary artery disease. New England Journal of Medicine 300, 13–50 (1979)

    Article  Google Scholar 

  9. Elidan, G., Ninio, M., Friedman, N., Schuurmans, D.: Data perturbation for escaping local maxima in learning. In: Proceedings of the Eighteenth National Conference on Artificial Intelligence and Fourteenth Conference on Innovative Applications of Artificial Intelligence, pp. 132–139. AAAI Press, Edmonton, Alberta, Canada (2002)

    Google Scholar 

  10. Gammerman, A., Vovk, V., Vapnik, V.: Learning by transduction. In: G.F. Cooper, S. Moral (eds.) Proceedings of the 14th Conference on Uncertainty in Artificial Intelligence, pp. 148–155. Morgan Kaufmann, San Francisco, USA, Madison, Wisconsin (1998)

    Google Scholar 

  11. Giacinto, G., Roli, F.: Dynamic classifier selection based on multiple classifier behaviour. Pattern Recognition 34, 1879–1881 (2001)

    Article  MATH  Google Scholar 

  12. Gibbs, A.L., Su, F.E.: On choosing and bounding probability metrics. International Statistical Review 70(3), 419–435 (2002)

    Article  MATH  Google Scholar 

  13. Halck, O.M.: Using hard classifiers to estimate conditional class probabilities. In: T. Elomaa, H. Mannila, H. Toivonen (eds.) Proceedings of the Thirteenth European Conference on Machine Learning, pp. 124–134. Springer-Verlag, Berlin (2002)

    Google Scholar 

  14. Hastie, T., Tibisharani, R., Friedman, J.: The Elements of Statistical Learning. Springer-Verlag (2001)

    Google Scholar 

  15. Heskes, T.: Practical confidence and prediction intervals. Advances in Neural Information Processing Systems 9, 176–182 (1997)

    Google Scholar 

  16. Ho, S.S., Wechsler, H.: Transductive confidence machine for active learning. In: Proc. Int. Joint Conf. on Neural Networks’03. Portland, OR. (2003)

    Google Scholar 

  17. John, G.H., Langley, P.: Estimating continuous distributions in Bayesian classifiers. In: P. Besnard, S. Hanks (eds.) Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann, San Francisco, USA (1995)

    Google Scholar 

  18. Kearns, M.J., Ron, D.: Algorithmic stability and sanity-check bounds for leave-one-out crossvalidation. In: Y. Freund, R. Shapire (eds.) Computational Learning Theory, pp. 152–162. Morgan Kaufmann (1997)

    Google Scholar 

  19. Kleijnen, J.: Experimental designs for sensitivity analysis of simulation models, tutorial at the Eurosim 2001 Conference (2001)

    Google Scholar 

  20. Kononenko, I., ˇSimec, E., Robnik-ˇSikonja, M.: Overcoming the myopia of inductive learning algorithms with ReliefF. Applied Intelligence 7, 39–55 (1997)

    Google Scholar 

  21. Kononenko, I.: Semi-naive Bayesian classifier. In: Y. Kodratoff (ed.) Proc. EuropeanWorking Session on Learning-91, pp. 206–219. Springer-Verlag, Berlin-Heidelberg-New York, Porto, Potrugal (1991)

    Google Scholar 

  22. Kukar, M.: Transductive reliability estimation for medical diagnosis. Artif. intell. med. pp. 81–106 (2003)

    Google Scholar 

  23. Kukar, M.: Quality assessment of individual classifications in machine learning and data mining. Knowledge and information systems 9(3), 364–384 (2006)

    Article  Google Scholar 

  24. Kukar, M., Kononenko, I.: Reliable classifications with Machine Learning. In: T. Elomaa, H. Mannila, H. Toivonen (eds.) Proceedings of 13th European Conference on Machine Learning, ECML 2002, pp. 219–231. Springer-Verlag, Berlin (2002)

    Chapter  Google Scholar 

  25. Li, M., Vit’anyi, P.: An introduction to Kolmogorov complexity and its applications, 2nd edn. Springer-Verlag, New York (1997)

    Google Scholar 

  26. Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and typicalness frameworks. In: Proc. ECML 2001, vol. 2167, pp. 350–357 (2001)

    Google Scholar 

  27. Nouretdinov, I., Melluish, T., Vovk, V.: Predictions with confidence intervals (local error bars). In: Proceedings of the International Conference on Neural Information Processing, pp. 847–852. Seoul, Korea (1994)

    Google Scholar 

  28. Nouretdinov, I., Melluish, T., Vovk, V.: Ridge regression confidence machine. In: Proc. 18th International Conf. on Machine Learning, pp. 385–392. Morgan Kaufmann, San Francisco (2001)

    Google Scholar 

  29. Olona-Cabases, M.: The probability of a correct diagnosis. In: J. Candell-Riera, D. Ortega-Alcalde (eds.) Nuclear Cardiology in Everyday Practice, pp. 348–357. Kluwer, Dordrecht, NL (1994)

    Google Scholar 

  30. Pfahringer, B., Bensuasan, H., Giraud-Carrier, C.: Meta-learning by landmarking various learning algorithms. In: Proc. 17th International Conf. on Machine Learning. Morgan Kaufmann, San Francisco, CA (2000)

    Google Scholar 

  31. Proedrou, K., Nouretdinov, I., Vovk, V., Gammerman, A.: Transductive confidence machines for pattern recognition. In: Proc. ECML 2002, pp. 381–390. Springer, Berlin (2002)

    Google Scholar 

  32. Rumelhart, D., McClelland, J.L.: Parallel Distributed Processing, vol. 1: Foundations. MIT Press, Cambridge (1986)

    Google Scholar 

  33. Saunders, C., Gammerman, A., Vovk., V.: Transduction with confidence and credibility. In: T. Dean (ed.) Proceedings of the International Joint Conference on Artificial Intelligence. Morgan Kaufmann, San Francisco, USA, Stockholm, Sweden (1999)

    Google Scholar 

  34. Saunders, C., Gammerman, A., Vovk, V.: Computationally efficient transductive machines. In: Algorithmic Learning Theory, 11th International Conference, ALT 2000, Sydney, Australia, December 2000, Proceedings, vol. 1968, pp. 325–333. Springer, Berlin (2000)

    Google Scholar 

  35. Seewald, A., Furnkranz, J.: An evaluation of grading classifiers. In: Proc. 4th International Symposium on Advances in Intelligent Data Analysis, pp. 115–124 (2001)

    Google Scholar 

  36. Smyth, P., Gray, A., Fayyad, U.: Retrofitting decision tree classifiers using kernel density estimation. In: A. Prieditis, S.J. Russell (eds.) Proceedings of the Twelvth International Conference on Machine Learning, pp. 506–514. Morgan Kaufmann, San Francisco, USA, Tahoe City, California, USA (1995)

    Google Scholar 

  37. Specht, D.F., Romsdahl, H.: Experience with adaptive pobabilistic neural networks and adaptive general regression neural networks. In: S.K. Rogers (ed.) Proceedings of IEEE International Conference on Neural Networks. IEEE Press, Piscataway, USA, Orlando, USA (1994)

    Google Scholar 

  38. Taneja, I.J.: On generalized information measures and their applications. Adv. Electron. and Elect. Physics 76, 327–416 (1995)

    Google Scholar 

  39. Tsuda, K., Raetsch, M., Mika, S., Mueller, K.: Learning to predict the leave-one-out error of kernel based classifiers. In: Lecture Notes in Computer Science, pp. 227–331. Springer, Berlin/Heidelberg (2001)

    Google Scholar 

  40. Vapnik, V.: Statistical Learning Theory. John Wiley, New York, USA (1998)

    MATH  Google Scholar 

  41. Venables, W.N., Ripley, B.D.: Modern Applied Statistics with S-PLUS. Fourth edition. Springer-Verlag (2002)

    Google Scholar 

  42. Vovk, V., Gammerman, A., Saunders, C.: Machine learning application of algorithmic randomness. In: I. Bratko, S. Dzeroski (eds.) Proceedings of the 16th International Conference on Machine Learning (ICML’99). Morgan Kaufmann, San Francisco, USA, Bled, Slovenija (1999)

    Google Scholar 

  43. Wand, M.P., Jones, M.C.: Kernel Smoothing. Chapman and Hall, London (1995)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer and Information Science, University of Ljubljana, Tržaška 25, SI-1001, Ljubljana, Slovenia

    Matjaž Kukar

Authors
  1. Matjaž Kukar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matjaž Kukar .

Editor information

Editors and Affiliations

  1. , School of Information Technology, Deakin University, 221 Burwood Highway, Burwood, 3125, Victoria, Australia

    Honghua Dai

  2. , Computing, Hong Kong Polytechnic University, Man Wai Building, Hunghom, PQ806, Hong Kong SAR

    James N. K. Liu

  3. , Department of Knowledge Engineering, Maastricht University, Maastricht, 6200MD, Netherlands

    Evgueni Smirnov

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer Science+Business Media, LLC

About this paper

Cite this paper

Kukar, M. (2012). Transductive Reliability Estimation for Individual Classifications in Machine Learning and Data Mining. In: Dai, H., Liu, J., Smirnov, E. (eds) Reliable Knowledge Discovery. Springer, Boston, MA. https://doi.org/10.1007/978-1-4614-1903-7_1

Download citation

  • DOI: https://doi.org/10.1007/978-1-4614-1903-7_1

  • Published:

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4614-1902-0

  • Online ISBN: 978-1-4614-1903-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation