Proactive Transfer Learning for Heterogeneous Feature and Label Spaces

  • Seungwhan Moon
  • Jaime Carbonell
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9852)


We propose a framework for learning new target tasks by leveraging existing heterogeneous knowledge sources. Unlike the traditional transfer learning, we do not require explicit relations between source and target tasks, and instead let the learner actively mine transferable knowledge from a source dataset. To this end, we develop (1) a transfer learning method for source datasets with heterogeneous feature and label spaces, and (2) a proactive learning framework which progressively builds bridges between target and source domains in order to improve transfer accuracy. Experiments on a challenging transfer learning scenario (learning from hetero-lingual datasets with non-overlapping label spaces) show the efficacy of the proposed approach.


Near Neighbor Transfer Learning Source Domain Target Task Target Instance 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Amini, M., Usunier, N., Goutte, C.: Learning from multiple partially observed views-an application to multilingual text categorization. In: NIPS, pp. 28–36 (2009)Google Scholar
  2. 2.
    Chattopadhyay, R., Fan, W., Davidson, I., Panchanathan, S., Ye, J.: Joint transfer and batch-mode active learning. In: ICML, pp. 253–261 (2013)Google Scholar
  3. 3.
    Dai, W., Chen, Y., Xue, G.R., Yang, Q., Yu, Y.: Translated learning: transfer learning across different feature spaces. In: NIPS, pp. 353–360 (2008)Google Scholar
  4. 4.
    Dhillon, P., Foster, D.P., Ungar, L.H.: Multi-view learning of word embeddings via CCA. In: NIPS, pp. 199–207 (2011)Google Scholar
  5. 5.
    Duan, L., Xu, D., Tsang, I.: Learning with augmented features for heterogeneous domain adaptation. In: ICML (2012)Google Scholar
  6. 6.
    Fang, M., Yin, J., Tao, D.: Active learning for crowdsourcing using knowledge transfer. In: AAAI (2014)Google Scholar
  7. 7.
    Frome, A., Corrado, G., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: Devise: a deep visual-semantic embedding model. In: NIPS (2013)Google Scholar
  8. 8.
    Fu, Z., Xiang, T., Kodirov, E., Gong, S.: Zero-shot object recognition by semantic manifold distance. In: CVPR, pp. 2635–2644 (2015)Google Scholar
  9. 9.
    Gavves, E., Mensink, T.E.J., Tommasi, T., Snoek, C.G.M., Tuytelaars, T.: Active transfer learning with zero-shot priors: reusing past datasets for future tasks. In: ICCV (2015)Google Scholar
  10. 10.
    Halko, N., Martinsson, P.G., Tropp, J.A.: Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Huang, J., Gretton, A., Borgwardt, K.M., Schölkopf, B., Smola, A.J.: Correcting sample selection bias by unlabeled data. In: NIPS (2007)Google Scholar
  12. 12.
    Kale, D., Liu, Y.: Accelerating active learning with transfer learning. In: ICDM, pp. 1085–1090 (2013)Google Scholar
  13. 13.
    Kodirov, E., Xiang, T., Fu, Z., Gong, S.: Unsupervised domain adaptation for zero-shot learning. In: ICCV (2015)Google Scholar
  14. 14.
    Kshirsagar, M., Carbonell, J., Klein-Seetharaman, J.: Multisource transfer learning for host-pathogen protein interaction prediction in unlabeled tasks (2013)Google Scholar
  15. 15.
    Li, X., Guo, Y., Schuurmans, D.: Semi-supervised zero-shot classification with label representation learning. In: ICCV, pp. 4211–4219 (2015)Google Scholar
  16. 16.
    Long, M., Wang, J.: Learning transferable features with deep adaptation networks. In: ICML (2015)Google Scholar
  17. 17.
    Loy, C., Hospedales, T., Xiang, T., Gong, S.: Stream-based joint exploration-exploitation active learning. In: CVPR, pp. 1560–1567, June 2012Google Scholar
  18. 18.
    Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(2579–2605), 85 (2008)MATHGoogle Scholar
  19. 19.
    Mensink, T., Gavves, E., Snoek, C.G.: Costa: Co-occurrence statistics for zero-shot classification. In: CVPR, pp. 2441–2448 (2014)Google Scholar
  20. 20.
    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: ICLR (2013)Google Scholar
  21. 21.
    Moon, S., Carbonell, J.: Proactive learning with multiple class-sensitive labelers. In: International Conference on Data Science and Advanced Analytics (DSAA) (2014)Google Scholar
  22. 22.
    Moon, S., Kim, S., Wang, H.: Multimodal transfer deep learning with applications in audio-visual recognition. In: NIPS MMML Workshop (2015)Google Scholar
  23. 23.
    Nguyen, H., Smeulders, A.: Active learning using pre-clustering. In: International Conference on Machine Learning (ICML) (2004)Google Scholar
  24. 24.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  25. 25.
    Rohrbach, M., Ebert, S., Schiele, B.: Transfer learning in a transductive setting. In: NIPS. pp. 46–54 (2013)Google Scholar
  26. 26.
    Settles, B., Craven, M.: Training text classifiers by uncertainty sampling. In: EMNLP, pp. 1069–1078 (2008)Google Scholar
  27. 27.
    Shi, X., Fan, W., Ren, J.: Actively transfer domain knowledge. In: Daelemans, W., Goethals, B., Morik, K. (eds.) ECML PKDD 2008, Part II. LNCS (LNAI), vol. 5212, pp. 342–357. Springer, Heidelberg (2008)Google Scholar
  28. 28.
    Socher, R., Ganjoo, M., Manning, C.D., Ng, A.Y.: Zero shot learning through cross-modal transfer. In: NIPS (2013)Google Scholar
  29. 29.
    Sun, Q., Amin, M., Yan, B., Martell, C., Markman, V., Bhasin, A., Ye, J.: Transfer learning for bilingual content classification. In: KDD, pp. 2147–2156 (2015)Google Scholar
  30. 30.
    Wang, W., Arora, R., Livescu, K., Bilmes, J.: On deep multi-view representation learning. In: ICML (2015)Google Scholar
  31. 31.
    Weston, J., Bengio, S., Usunier, N.: Wsabie: Scaling up to large vocabulary image annotation. In: IJCAI 2011 (2011)Google Scholar
  32. 32.
    Xiao, M., Guo, Y.: Semi-supervised subspace co-projection for multi-class heterogeneous domain adaptation. In: Appice, A., Rodrigues, P.P., Santos Costa, V., Gama, J., Jorge, A., Soares, C. (eds.) ECML PKDD 2015. LNCS (LNAI), vol. 9285, pp. 525–540. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-23525-7_32 CrossRefGoogle Scholar
  33. 33.
    Zhang, Z., Saligrama, V.: Zero-shot learning via semantic similarity embedding. In: ICCV (2015)Google Scholar
  34. 34.
    Zhou, J.T., Pan, S.J., Tsang, I.W., Yan, Y.: Hybrid heterogeneous transfer learning through deep learning. In: AAAI (2014)Google Scholar
  35. 35.
    Zhu, J., Wang, H., Tsou, B., Ma, M.: Active learning with sampling by uncertainty and density for data annotations. IEEE Trans. Audio Speech Lang. Process. 18, 1323–1331 (2010)CrossRefGoogle Scholar
  36. 36.
    Zhu, Z., Zhu, X., Ye, Y., Guo, Y.F., Xue, X.: Transfer active learning. In: CIKM, pp. 2169–2172 (2011)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.School of Computer Science, Language Technologies InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations