Advertisement

Importance-Weighted Cross-Validation for Covariate Shift

  • Masashi Sugiyama
  • Benjamin Blankertz
  • Matthias Krauledat
  • Guido Dornhege
  • Klaus-Robert Müller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4174)

Abstract

A common assumption in supervised learning is that the input points in the training set follow the same probability distribution as the input points used for testing. However, this assumption is not satisfied, for example, when the outside of training region is extrapolated. The situation where the training input points and test input points follow different distributions is called the covariate shift. Under the covariate shift, standard machine learning techniques such as empirical risk minimization or cross-validation do not work well since their unbiasedness is no longer maintained. In this paper, we propose a new method called importance-weighted cross-validation, which is still unbiased even under the covariate shift. The usefulness of our proposed method is successfully tested on toy data and furthermore demonstrated in the brain-computer interface, where strong non-stationarity effects can be seen between calibration and feedback sessions.

Keywords

Supervise Learning Unbiased Estimator Importance Sampling Test Error Input Point 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Agostinelli, C.: Robust model selection by cross-validation via weighted likelihood methodology. Technical Report 1999.37, Dipartimento di Scienze Statistiche, Università di Padova (1999)Google Scholar
  2. 2.
    Blankertz, B., Dornhege, G., Krauledat, M., Müller, K.-R., Curio, G.: The Berlin brain-computer interface: Report from the feedback sessions. Technical Report 1, Fraunhofer FIRST (2005)Google Scholar
  3. 3.
    Duda, R.O., Hart, P.E., Stor, D.G.: Pattern Classification. Wiley, New York (2001)MATHGoogle Scholar
  4. 4.
    Heckman, J.J.: Sample selection bias as a specification error. Econometrica 47(1), 153–162 (1979)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Schölkopf, B., Smola, A.J.: Learning with Kernels. MIT Press, Cambridge (2002)Google Scholar
  6. 6.
    Shimodaira, H.: Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference 90(2), 227–244 (2000)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Sugiyama, M.: Active learning in approximately linear regression based on conditional expectation of generalization error. Journal of Machine Learning Research 7, 141–166 (2006)MathSciNetGoogle Scholar
  8. 8.
    Sugiyama, M., Blankertz, B., Krauledat, M., Dornhege, G., Müller, K.-R.: Importance-weighted cross-validation for covariate shift. Technical Report TR06-0002, Department of Computer Science, Tokyo Institute of Technology (February 2006)Google Scholar
  9. 9.
    Vapnik, V.N.: Statistical Learning Theory. Wiley, New York (1998)MATHGoogle Scholar
  10. 10.
    Vidaurre, C., Schlögl, A., Cabeza, R., Pfurtscheller, G.: About adaptive classifiers for brain computer interfaces. Biomedizinische Technik 49(1), 85–86 (2004)Google Scholar
  11. 11.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clinical Neurophysiology 113(6), 767–791 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Masashi Sugiyama
    • 1
  • Benjamin Blankertz
    • 2
  • Matthias Krauledat
    • 2
    • 3
  • Guido Dornhege
    • 2
  • Klaus-Robert Müller
    • 2
    • 3
  1. 1.Department of Computer ScienceTokyo Institute of TechnologyTokyoJapan
  2. 2.Fraunhofer FIRST.IDABerlinGermany
  3. 3.Department of Computer ScienceUniversity of PotsdamPotsdamGermany

Personalised recommendations