Off-Line Learning with Transductive Confidence Machines: An Empirical Evaluation

  • Stijn Vanderlooy
  • Laurens van der Maaten
  • Ida Sprinkhuizen-Kuyper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4571)

Abstract

The recently introduced transductive confidence machines (TCMs) framework allows to extend classifiers such that they satisfy the calibration property. This means that the error rate can be set by the user prior to classification. An analytical proof of the calibration property was given for TCMs applied in the on-line learning setting. However, the nature of this learning setting restricts the applicability of TCMs. In this paper we provide strong empirical evidence that the calibration property also holds in the off-line learning setting. Our results extend the range of applications in which TCMs can be applied. We may conclude that TCMs are appropriate in virtually any application domain.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bellotti, T.: Confidence Machines for Microarray Classification and Feature Selection. PhD thesis, Royal Holloway University of London, London, UK (February 2006)Google Scholar
  2. 2.
    Bellotti, T., Luo, Z., Gammerman, A., Van Delft, F., Saha, V.: Qualified predictions for microarray and proteomics pattern diagnostics with confidence machines. International Journal of Neural Systems 15(4), 247–258 (2005)CrossRefGoogle Scholar
  3. 3.
    Blanzieri, E., Ricci, F.: Probability based metrics for nearest neighbor classification and case-based reasoning. In: Althoff, K.-D., Bergmann, R., Branting, L.K. (eds.) ICCBR 1999. LNCS (LNAI), vol. 1650, pp. 14–28. Springer, Heidelberg (1999)Google Scholar
  4. 4.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13(1), 21–27 (1967)MATHCrossRefGoogle Scholar
  5. 5.
    Domingos, P., Pazzani, M.: On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning 29(2), 103–130 (1997)MATHCrossRefGoogle Scholar
  6. 6.
    Fisher, R.: The use of multiple measurements in taxonomics problems. Annals of Eugenics 7, 178–188 (1936)Google Scholar
  7. 7.
    Gammerman, A., Vovk, V.: Prediction algorithms and confidence measures based on algorithmic randomness theory. Theoretical Computer Science 287(1), 209–217 (2002)MATHCrossRefMathSciNetGoogle Scholar
  8. 8.
    Gammerman, A., Vovk, V., Vapnik, V.: Learning by transduction. In: Cooper, G., Moral, S. (eds.) UAI 1998. 14th Conference on Uncertainty in Artificial Intelligence, Madison, WI, July 24-26 1998, pp. 148–155. Morgan Kaufmann, San Francisco (1998)Google Scholar
  9. 9.
    Khardon, R., Roth, D., Servedio, R.: Efficiency versus convergence of boolean kernels for on-line learning algorithms. Journal of Artificial Intelligence Research 24, 341–356 (2005)MATHMathSciNetGoogle Scholar
  10. 10.
    Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and typicalness frameworks. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 360–371. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  11. 11.
    Newman, D., Hettich, S., Blake, C., Merz, C.: UCI repository of machine learning databases (1998)Google Scholar
  12. 12.
    Proedrou, K., Nouretdinov, I., Vovk, V., Gammerman, A.: Transductive confidence machines for pattern recognition. Technical Report 01-02, Royal Holloway University of London, London, UK (2001)Google Scholar
  13. 13.
    Saunders, C., Gammerman, A., Vovk, V.: Transduction with confidence and credibility. In: Dean, T. (ed.) IJCAI 1999. 16th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, July 31 - August 6, 1999, pp. 722–726. Morgan Kaufmann, San Francisco (1999)Google Scholar
  14. 14.
    Saunders, C., Gammerman, A., Vovk, V.: Computationally efficient transductive machines. In: Okamoto, T., Hartley, R., Kinshuk, Klus, J. (eds.) ICALT 2000. 11th International Conference on Algorithmic Learning Theory, Madison, WI, August 6-8, 2000, pp. 325–333. IEEE Computer Society Press, Los Alamitos (2000)Google Scholar
  15. 15.
    Shawe-Taylor, J., Cristianini, N.: Kernel Methods for Pattern Analysis. Cambridge University Press, Cambridge (2004)Google Scholar
  16. 16.
    Vanderlooy, S., van der Maaten, L., Sprinkhuizen-Kuyper, I.: Off-line learning with transductive confidence machines: an empirical evaluation. Technical Report MICC-IKAT 07-03, Universiteit Maastricht, Maastricht, The Netherlands (2007)Google Scholar
  17. 17.
    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. LNCS. Springer, Heidelberg (2005)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Stijn Vanderlooy
    • 1
  • Laurens van der Maaten
    • 1
  • Ida Sprinkhuizen-Kuyper
    • 2
  1. 1.MICC-IKAT, Universiteit Maastricht, P.O. Box 616, 6200 MD MaastrichtThe Netherlands
  2. 2.NICI, Radboud University Nijmegen, P.O. Box 9104, 6500 HE NijmegenThe Netherlands

Personalised recommendations