Importance Weighted Inductive Transfer Learning for Regression

  • Jochen Garcke
  • Thomas Vanck
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8724)


We consider inductive transfer learning for dataset shift, a situation in which the distributions of two sampled, but closely related, datasets differ. When the target data to be predicted is scarce, one would like to improve its prediction by employing data from the other, secondary, dataset. Transfer learning tries to address this task by suitably compensating such a dataset shift. In this work we assume that the distributions of the covariates and the dependent variables can differ arbitrarily between the datasets. We propose two methods for regression based on importance weighting. Here to each instance of the secondary data a weight is assigned such that the data contributes positively to the prediction of the target data. Experiments show that our method yields good results on benchmark and real world datasets.


inductive transfer learning importance weighting dataset shift 


  1. 1.
    Al-Stouhi, S., Reddy, C.K.: Adaptive boosting for transfer learning using dynamic updates. In: Gunopulos, D., Hofmann, T., Malerba, D., Vazirgiannis, M. (eds.) ECML PKDD 2011, Part I. LNCS (LNAI), vol. 6911, pp. 60–75. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  2. 2.
    Allen, T., Wald, D.: Evaluation of ground-motion modeling techniques for use in global shakemap–a critique of instrumental ground-motion prediction equations, peak ground motion to macroseismic intensity conversions, and macroseismic intensity predictions in different tectonic settings. U.S. Geological Survey Open-File Report 2009-1047 (2009)Google Scholar
  3. 3.
    Argyriou, A., Evgeniou, T., Pontil, M.: Convex multi-task feature learning. Machine Learning 73(3), 243–272 (2008)CrossRefGoogle Scholar
  4. 4.
    Bishop, C.M.: Pattern recognition and machine learning. Springer (2006)Google Scholar
  5. 5.
    Quiñonero Candela, J.N., Sugiyama, M., Schwaighofer, A., Lawrence, N.D. (eds.): Dataset Shift in Machine Learning. MIT Press (2009)Google Scholar
  6. 6.
    Cao, B., Pan, S.J., Zhang, Y., Yeung, D.Y., Yang, Q.: Adaptive transfer learning. In: AAAI. AAAI Press (2010)Google Scholar
  7. 7.
    Cortes, C., Mohri, M.: Domain adaptation and sample bias correction theory and algorithm for regression. Theoretical Computer Science 519, 103–126 (2014)CrossRefzbMATHMathSciNetGoogle Scholar
  8. 8.
    Dai, W., Yang, Q., Rong Xue, G., Yu, Y.: Boosting for transfer learning. In: International Conference on Machine Learning (ICML) (2007)Google Scholar
  9. 9.
    Daumé III, H.: Frustratingly easy domain adaptation. In: ACL, pp. 256–263 (2007)Google Scholar
  10. 10.
    Gretton, A., Smola, A., Huang, J., Schmittfull, M., Borgwardt, K., Schölkopf, B.: Covariate Shift by Kernel Mean Matching, pp. 131–160. MIT Press (2009)Google Scholar
  11. 11.
    Kuzborskij, I., Orabona, F.: Stability and hypothesis transfer learning. In: International Conference on Machine Learning (ICML) (2013)Google Scholar
  12. 12.
    Pan, S.J., Yang, Q.: A Survey on Transfer Learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  13. 13.
    Pardoe, D., Stone, P.: Boosting for regression transfer. In: International Conference on Machine Learning (ICML) (2010)Google Scholar
  14. 14.
    Raina, R., Ng, A.Y., Koller, D.: Constructing informative priors using transfer learning. In: International Conference on Machine Learning (ICML), pp. 713–720 (2006)Google Scholar
  15. 15.
    Rückert, U., Kramer, S.: Kernel-based inductive transfer. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 220–233. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  16. 16.
    Sugiyama, M., Kawanabe, M.: Machine learning in non-stationary environments: Introduction to covariate shift adaptation. MIT Press, Cambridge (2012)CrossRefGoogle Scholar
  17. 17.
    Sugiyama, M., Nakajima, S., Kashima, H., Bünau, P., Kawanabe, M.: Direct importance estimation with model selection and its application to covariate shift adaptation. In: NIPS 20, pp. 1433–1440 (2008)Google Scholar
  18. 18.
    Tan, B., Zhong, E., Wei, E., Yang, X.Q.: Multi-transfer: Transfer learning with multiple views and multiple sources. In: SDM (2013)Google Scholar
  19. 19.
    Tommasi, T., Orabona, F., Caputo, B.: Safety in numbers: Learning categories from few examples with multi model knowledge transfer. In: CVPR, pp. 3081–3088. IEEE (2010)Google Scholar
  20. 20.
    Yang, P., Gao, W.: Multi-view discriminant transfer learning. In: Rossi, F. (ed.) IJCAI. IJCAI/AAAI (2013)Google Scholar
  21. 21.
    Yang, Q., Pan, S.J., Zheng, V.W.: Estimating location using wi-fi. IEEE Intelligent Systems 23(1), 8–13 (2008)CrossRefGoogle Scholar
  22. 22.
    Zhang, D., He, J., Liu, Y., Si, L., Lawrence, R.D.: Multi-view transfer learning with a large margin approach. In: Apté, C., Ghosh, J., Smyth, P. (eds.) KDD, pp. 1208–1216. ACM (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Jochen Garcke
    • 1
  • Thomas Vanck
    • 1
  1. 1.Institut für Numerische SimulationBonnGermany

Personalised recommendations